Method for improving naturalness of synthetic audio and reducing noise

Abstract

The invention relates to a method for improving the naturalness of a synthetic audio and reducing noise, and the method comprises the following steps: 1, constructing a phoneme encoder, 2, constructing a variance adapter, and 3, constructing a frequency spectrum noise reducer, thereby achieving the improvement of the naturalness of the synthetic audio and the noise reduction. According to the scheme, a two-way encoder based on Transform is added in a phoneme encoder stage, so that training of a speech synthesis model can be completed by using fewer training resources; according to the method, linear time sequence modeling with a selective state space is used for a speech synthesis task for the first time, wavelet decomposition is introduced into a speech synthesis network by means of application of wavelet decomposition in the field of image denoising, and noise of generated audio is successfully reduced.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, artificial intelligence, and speech processing technology, and in particular to a method for reducing training resource usage and improving the naturalness of synthesized audio and noise reduction. Background Art

[0002] In recent years, text-to-speech, which aims to generate audio from text that closely resembles human speech, has attracted widespread interest in the machine learning community. Text-to-speech technology is essential for converting text into machine-understandable instructions, enabling accessibility, communication, and enhanced user experience across a variety of applications and industries.

[0003] Current text-to-speech methods include traditional speech synthesis methods and deep learning speech synthesis methods. Traditional speech synthesis methods include rule-based methods, splicing methods, and hidden Markov models (HMM). Deep learning speech synthesis methods include Tacotron2, FastSpeech, FastSpeech2, FastDiff, ProDiff, and NaturalSpeech. The present invention has studied these methods and techniques and found that the audio synthesized by these methods still has deficiencies in terms of naturalness, less noise, and use of training resources.

[0004] In order to further improve the naturalness of synthesized audio, make it contain less noise, and the method itself uses less training resources, the following three questions need to be answered: 1) How to enhance the naturalness of generated speech; 2) How to minimize the noise in generated speech; 3) How to reduce the use of training resources in speech synthesis models. In this invention, we conduct an in-depth analysis of these issues in text-to-speech. First, to enhance the naturalness of generated audio, we use a Transformer-based bidirectional encoder as part of the phoneme encoder. Then we introduce wavelet decomposition to minimize the noise of generated speech. Finally, we reduce the use of training resources by replacing each Transformer encoder block with a linear time series layer with a selective state space in the speech synthesis model. With the development of linear time series modeling with selective state space, we find that linear time series models with selective state space can take up less training resources and achieve similar performance to the Transformer architecture. Considering that the latest speech synthesis model is based on the Transformer architecture, the present invention proposes to replace each Transformer encoder block with a linear time series layer with a selective state space. By doing so, we can further reduce the occupation of training resources. Compared with previous speech synthesis methods, the present invention has several advantages: 1) Enhance the naturalness of generated speech. In previous cascaded vocoder pipelines, the audio they generate lacks naturalness. Our fully end-to-end text-to-waveform generation can improve naturalness with the help of a Transformer-based bidirectional encoder. 2) Minimize noise in generated audio. Previous work generates audio with some noise. By introducing wavelet decomposition, our network has a strong ability to remove data correlation, which is crucial for reducing noise in generated speech. 3) Reduce training resource usage. Previous models are based on Transformer encoders. When the Transformer model is used for speech synthesis, it has a large number of model parameters, which leads to a large amount of training resources. In this invention, we replace each Transformer encoder block with a linear time series layer with a selective state space, which helps to reduce the use of training resources. We perform an effect evaluation of the generated audio on the widely used LJSpeech dataset. Compared with the latest speech synthesis methods, the present invention achieves better results on subjective and objective evaluation indicators of speech synthesis and occupies less training resources. Summary of the invention

[0005] The purpose of the present invention is to provide a method for reducing the occupation of training resources and improving the naturalness and noise reduction of synthesized audio, so as to solve the deficiencies of synthesized audio in the existing speech synthesis technology introduced in the background technology in terms of naturalness, low noise and utilization of training resources.

[0006] In order to solve the above-mentioned shortcomings, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for reducing training resource usage and improving the naturalness of synthesized audio and noise reduction, and the specific steps are as follows:

[0008] Step 1: Build a phoneme encoder.

[0009] 1.1. Build a bidirectional encoder model based on Transformer and train it on a large-scale English Wikipedia dataset to obtain a trained model, represented by symbol b. The specific training process includes data preprocessing, model architecture, training parameter setting and use of loss function. In the data preprocessing stage, a crawler script is written to fully automatically download the XML dump of Wikipedia to obtain a large-scale English Wikipedia dataset. Subsequently, the text is cleaned and an automated script is written to implement batch processing to remove tags, links, and special characters at one time to ensure that the data is clean. The WordPiece word segmenter is used to divide the text into sub-word units. A masked language model task is used to generate training samples. In terms of model architecture, multiple stacked encoder layers are used, each encoder layer includes a self-attention mechanism and a feedforward neural network. In terms of training parameters, the initial learning rate is set to 3e-5 and the batch size is set to 64. In terms of loss function, the present invention uses a cross-entropy loss function to optimize the model to maximize the prediction probability of masked words.

[0010] 1.2. Based on 1.1, the present invention integrates the pre-trained model obtained by training with various creative improvement strategies, loads the Transformer-based bidirectional encoder model, and tokenizes the text, that is, loads the trained Transformer-based bidirectional encoder model and tokenizer, uses symbol A to represent the text, uses symbol b to represent the model, and uses symbol ∞ to represent the output after this operation. The above process is expressed by the following formula: ∞=b(A);

[0011] 1.3, split the input long sentence, use μ to represent the input long sentence, use ∪ to represent the sentence segmentation, and use θ to represent the final token format that can be understood by the Transformer's bidirectional encoder. Then the final formula can be expressed as follows: θ = ∪ (μ);

[0012] 1.4, through the forward propagation of the Transformer-based bidirectional encoder model, the process can be denoted by the symbol: ∝, the input word is denoted by: ω, and the forward propagation is completed by the following formula: ρ = ∝ (ω). Finally, from the last hidden layer of the model: H, the embedding is extracted by the following formula: ∈ = H (ρ);

[0013] 1.5, here we summarize the general expressions of 1.1-1.4, using α to represent the original input and Enc b To represent the encoding of the Transformer-based bidirectional encoder model, the output after 1.1-1.4 is: β = Enc b (α);

[0014] 1.6, Load the linear time series model with selective state space. Define the linear time series model with selective state space, which uses the selective state space model instead of the attention mechanism to effectively process sequence data;

[0015] 1.7, using a linear time series layer with a selective state space, using the notation Represents, to replace each Transformer encoder block, here, the present invention proposes to remove the multi-head attention and feed-forward sublayers in the Transformer, retain the layer normalization and residual connection, and use the rest This effectively ensures the stability of the model and the expected gradient flow during training.

[0016] 1.8, adjust the forward propagation. During the forward pass of the encoder, the input sequence is passed to each linear time series layer with a selective state space in order: For each linear time series layer with a selective state space, the output is calculated based on the selective state space transformation of the relevant sequence information;

[0017] 1.9, use To represent the linear time series modeling with selective state space, encode the output β obtained in 1.5. After 1.6-1.8, we can finally get:

[0018] Step 2, build the variance adapter,

[0019] The core task of the variance adapter of the present invention is to predict the duration, pitch and energy of the audio, so it is necessary to construct duration prediction, pitch prediction and energy prediction.

[0020] 2.1, duration prediction:

[0021] 2.1.1, Model structure: To capture the dependencies between sequences, C represents a 1D convolution operation;

[0022] 2.1.2, Loss function: Use the following formula to optimize the loss function and optimize the duration prediction:

[0023] Where D predis the duration predicted by the model, where D true is the actual duration of the audio;

[0024] 2.2, Pitch Prediction:

[0025] 2.2.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 4 layers of convolution;

[0026] 2.2.2, Loss function: Use the following formula as the loss function to optimize pitch prediction:

[0027] Where P pred is the pitch predicted by the model, where P true is the actual pitch of the audio;

[0028] 2.3, Energy prediction:

[0029] 2.3.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 8 layers of convolution;

[0030] 2.3.2, Loss function: Use the following formula as the loss function to optimize energy prediction:

[0031] Where E pred is the energy predicted by the model, where E true is the actual energy of the model audio;

[0032] 2.4, after the above pitch, energy and duration adjustment, the embedding is recorded as It will undergo further convolution C through the following formula: Description, and finally modeling to obtain the global characteristics of speech.

[0033] Step 3: Build a spectral denoiser.

[0034] 3.1, after the variance adapter, δ is obtained, which is generally sent to a vocoder or a Mel spectrogram decoder to generate the final speech. In order to further reduce the noise, the present invention refers to the application of wavelet decomposition in image denoising, and creatively adds wavelet decomposition to the spectrum denoiser for the first time for speech synthesis, using Wavelet spec Represents wavelet transform, then the audio obtained after wavelet decomposition, final spectral denoiser and vocoder: ε = Wavelet spec (δ).

[0035] A system for improving the naturalness of synthesized audio and reducing noise, the system includes three parts: a phoneme encoder, a variance adapter, and a spectral denoiser. The first part is to construct a phoneme encoder, which encodes and embeds input text. The second part is to construct a variance adapter, which is mainly to further process the high-dimensional vector obtained by the phoneme encoder, pass the input feature vector to the variance adapter, and further adjust and map it to a time dimension aligned with the target feature. The pitch, duration and energy of the audio are predicted by convolution. The embeddedness after pitch, energy and duration adjustment will be further convolved to model the contextual dependency and global features of speech. The third part is to construct a spectral denoiser.

[0036] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, a method for improving the naturalness of synthesized audio and reducing noise is implemented.

[0037] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement a method for improving the naturalness of synthesized audio and reducing noise.

[0038] As a further optimization scheme of the present invention, the step 1 creatively proposes for the first time to introduce a Transformer-based bidirectional encoder model into the speech synthesis phoneme encoder. The present invention proposes for the first time to conduct training on a large-scale English Wikipedia dataset to obtain a trained pre-trained model and incorporate it into the phoneme encoder for speech synthesis.

[0039] As a further optimization scheme of the present invention, the step 1 creatively proposes for the first time to replace each Transformer encoder block with a linear time series layer with a selective state space. The present invention creatively proposes for the first time to replace each Transformer encoder block with a linear time series layer with a selective state space, and effectively reduces the occupation of training resources.

[0040] As a further optimization scheme of the present invention, wavelet decomposition is integrated into the traditional spectrum denoiser in step 3. The present invention creatively introduces it into the speech synthesis task for the first time and constructs a new spectrum denoiser for speech synthesis.

[0041] As a further optimization scheme of the present invention, the present invention trains and tests the neural network constructed by steps 1-3 on the general speech synthesis test data set LJSpeech data set, and compares the speech synthesis method proposed by the present invention with the existing speech synthesis method on the same test set. The results are as follows:

[0042] Method MOS PESQ NB_PESQ STOI Tacotron2 2.5345 2.4159 2.4134 0.6423 FastSpeech 2.7944 2.6189 2.6495 0.6779 FastSpeech2 2.8256 2.7439 2.7660 0.7750 FastDiff 2.8483 2.8164 2.8246 0.7993 ProDiff 2.9343 2.8305 2.8722 0.8123 NaturalSpeech 3.1241 3.0134 3.1326 0.8312 Method of the present invention 3.4906 3.5018 3.5342 0.8713

[0043] It can be seen from the above table that the method proposed in the present invention has achieved the best results in both subjective and objective evaluation indicators of speech synthesis compared with the existing speech synthesis methods. It can be seen that the present invention has outstanding substantive characteristics and significant progress.

[0044] As a further optimization solution of the present invention, the present invention compares the video memory resources used by various existing speech synthesis models in the training phase, and the results are as follows:

[0045]

[0046]

[0047] It can be seen from the above table that the method proposed in the present invention has achieved the best results in terms of training resource occupation of speech synthesis compared with the existing speech synthesis methods. It can be seen that the present invention has made significant progress compared with the existing speech synthesis methods.

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] ① Occupies fewer training resources: The present invention can complete the training of the speech synthesis model using fewer training resources;

[0050] ② The generated audio has less noise: The present invention uses the application of wavelet decomposition in the field of image denoising and introduces it into the speech synthesis network, successfully reducing the noise of the generated audio;

[0051] ③ The generated audio has high naturalness: The present invention compares the existing speech synthesis models on the same test set. From the subjective and objective evaluation indicators of speech synthesis, it can be seen that the audio quality generated by the present invention is high.

[0052] In general, the present invention solves the shortcomings of the existing speech synthesis methods introduced in the background technology in terms of naturalness, low noise and use of training resources. The method can use the public LJSpeech to build the training set required for speech synthesis, and verify the effect of the speech synthesis model on the same test set, improve the naturalness of the synthesized audio, reduce the noise of the speech synthesis audio, and occupy fewer training resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0055] Example: See Figure 1 , a method for improving the naturalness of synthesized audio and reducing noise, the specific steps are as follows:

[0056] Step 1: Build a phoneme encoder.

[0057] 1.1, build a Transformer-based bidirectional encoder model and train it on a large-scale English Wikipedia dataset, so that you can get a trained model, represented by symbol b;

[0058] 1.2, load the Transformer-based bidirectional encoder model and tokenize the text, that is, load the trained Transformer-based bidirectional encoder model and tokenizer, use symbol A to represent the text, use symbol b to represent the model, and use symbol ∞ to represent the output after this operation. Then the above process can be expressed by the following formula: ∞=b(A);

[0059] 1.3, split the input long sentence, use μ to represent the input long sentence, use ∪ to represent the sentence segmentation, and use θ to represent the final token format that can be understood by the Transformer's bidirectional encoder. Then the final formula can be expressed as follows: θ = ∪ (μ);

[0060] 1.4, through the forward propagation of the Transformer-based bidirectional encoder model, the process can be denoted by the symbol: ∝, the input word is denoted by: ω, and the forward propagation is completed by the following formula: ρ = ∝ (ω). Finally, from the last hidden layer of the model: H, the embedding is extracted by the following formula: ∈ = H (ρ);

[0061] 1.5, here we summarize the general expressions of 1.1-1.4, using α to represent the original input and Enc b To represent the encoding of the Transformer-based bidirectional encoder model, the output after 1.1-1.4 is: β = Enc b (α);

[0062] 1.6, Load the linear time series model with selective state space. Define the linear time series model with selective state space, which uses the selective state space model instead of the attention mechanism to effectively process sequence data;

[0063] 1.7, using a linear time series layer with a selective state space, using the notation Represents, to replace each Transformer encoder block, here, the present invention proposes to remove the multi-head attention and feed-forward sublayers in the Transformer, retain the layer normalization and residual connection, and use the rest This effectively ensures the stability of the model and the expected gradient flow during training.

[0064] 1.8, adjust the forward propagation. During the forward pass of the encoder, the input sequence is passed to each linear time series layer with a selective state space in order: For each linear time series layer with a selective state space, the output is calculated based on the selective state space transformation of the relevant sequence information;

[0065] 1.9, use To represent the linear time series modeling with selective state space, encode the output β obtained in 1.5. After 1.6-1.8, we can finally get:

[0066] Step 2, build the variance adapter,

[0067] The core task of the variance adapter of the present invention is to predict the duration, pitch and energy of the audio, so it is necessary to construct duration prediction, pitch prediction and energy prediction.

[0068] 2.1, duration prediction:

[0069] 2.11, Model structure: To capture the dependencies between sequences, C represents a 1D convolution operation;

[0070] 2.1.2, Loss function: Use the following formula to optimize the loss function and optimize the duration prediction:

[0071] Where D pred,i is the duration of the audio of the i-th sample predicted by the model, where P true,i is the actual duration of the audio of the i-th sample. In the training of the deep learning model, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios;

[0072] 2.2, Pitch Prediction:

[0073] 2.2.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 4 layers of convolution;

[0074] 2.2.2, Loss function: Use the following formula as the loss function to optimize pitch prediction:

[0075] Where P pred,i is the pitch of the audio of the i-th sample predicted by the model, where P true,i is the actual pitch of the audio of the i-th sample. In the training of the deep learning model, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios;

[0076] 2.3, Energy prediction:

[0077] 2.3.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 8 layers of convolution;

[0078] 2.3.2, Loss function: Use the following formula as the loss function to optimize energy prediction:

[0079] Where E pred,i is the energy of the audio of the i-th sample predicted by the model, where E true,i is the actual energy of the audio of the i-th sample. In the training of the deep learning model, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios;

[0080] 2.4, after the above pitch, energy and duration adjustment, the embedding is recorded as It will undergo further convolution C, thus passing the following formula: Modeling obtains the global characteristics of speech;

[0081] Step 3: Build a spectral denoiser.

[0082] 3.1, after the variance adapter, δ is obtained, which is generally sent to a vocoder or a Mel spectrogram decoder to generate the final speech. In order to further reduce the noise, the present invention refers to the application of wavelet decomposition in image denoising, and creatively adds wavelet decomposition to the spectrum denoiser for the first time for speech synthesis, using Wavelet spec Represents wavelet transform, then the audio obtained after wavelet decomposition, final spectral denoiser and vocoder: ε = Wavelet spec (δ).

[0083] A schematic diagram of a method proposed by the present invention for reducing training resource usage and improving the naturalness of synthesized audio and noise reduction is shown in FIG. Figure 1As shown in the figure. The figure mainly consists of three parts: phoneme encoder, variance adapter, and spectral denoiser. The first part is to construct the phoneme encoder. This part encodes and embeds the input text. Compared with the existing technology, the bidirectional encoder model based on Transformer and the linear time series modeling with selective state space are creatively used here. The second part is the construction of the variance adapter. This part mainly processes the high-dimensional vector obtained by the phoneme encoder and passes the input feature vector to the variance adapter, so as to further adjust and map it to the time dimension aligned with the target feature. The pitch, duration and energy of the audio are predicted by convolution. The embedding after pitch, energy and duration adjustment will be further convolved to model the contextual dependency and global features of speech. The third part is the construction of the spectral denoiser. Compared with the existing technology, wavelet decomposition is creatively introduced here to further improve the effect of synthesized audio, which is a significant improvement.

[0084] The above is an implementation method of this patent. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that he or she can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for improving the naturalness of synthesized audio and reducing noise, characterized in that: The following steps are involved: Step 1: Build a phoneme encoder. Step 2, build the variance adapter, Step 3: Build a spectral denoiser to improve the naturalness of the synthesized audio and reduce noise.

2. A method for improving the naturalness of synthesized audio and reducing noise according to claim 1, characterized in that: Step 1: Construct a phoneme encoder as follows: 1.

1. Build a bidirectional encoder model based on Transformer and train it on a large-scale English Wikipedia dataset to obtain a trained model, represented by symbol b. The specific training process includes data preprocessing, model architecture, training parameter setting and the use of loss function. In the data preprocessing stage, a crawler script was written to fully automatically download the XML dump of Wikipedia to obtain a large-scale English Wikipedia dataset. Subsequently, the text was cleaned and an automated script was written to implement batch processing to remove tags, links, and special characters at one time to ensure that the data is clean. The WordPiece tokenizer was used to segment the text into sub-word units, and the masked language model task was used to generate training samples. In terms of model architecture, multiple stacked encoder layers were used, each of which included a self-attention mechanism and a feedforward neural network. In terms of training parameters, the initial learning rate was set to 3e-5 and the batch size was set to 64. In terms of loss function, the cross entropy loss function was used to optimize the model to maximize the prediction probability of masked words. 1.2, load this Transformer-based bidirectional encoder model and tokenize the text, that is, load the trained Transformer-based bidirectional encoder model and tokenizer, use symbol A to represent the text, use symbol b to represent the model, and use symbol ∞ to represent the output after this operation. The above process is expressed by the following formula: ∞=b(A); 1.3, split the input long sentence, use μ to represent the input long sentence, use ∪ to represent the sentence segmentation, and use θ to represent the final token format that can be understood by the Transformer's bidirectional encoder. The final formula is as follows: θ = ∪ (μ); 1.4, through the forward propagation of the Transformer-based bidirectional encoder model, the process is denoted by the symbol: ∝, the input word is denoted by: ω, and the forward propagation is completed by the following formula: ρ = ∝(ω). Finally, from the last hidden layer of the model: H, the embedding is extracted by the following formula: ε = H(ρ); 1.5, summarize the general expressions of 1.1-1.4, use α to represent the original input, and use Enc b To represent the encoding of the Transformer-based bidirectional encoder model, the output after 1.1-1.4 is: β = Enc b (α); 1.6, Load the linear time series model with selective state space, define the linear time series model with selective state space, which uses the selective state space model to effectively process sequence data; 1.7, using a linear time series layer with a selective state space, using the notation Represents, to replace each Transformer encoder block, proposes to remove the multi-head attention and feed-forward sublayers in the Transformer, retaining the layer normalization and residual connection, and the rest is used The stacking effectively ensures the stability of the model and the expected gradient flow during training; 1.8, adjust the forward propagation. During the forward pass of the encoder, the input sequence is passed to each linear time series layer with a selective state space in order: For each linear time series layer with a selective state space, the output is calculated based on the selective state space transformation of the relevant sequence information; 1.9, use To represent the linear time series modeling with selective state space, encode the output β obtained in 1.

5. After 1.6-1.8, we can finally get:

3. A method for improving the naturalness of synthesized audio and reducing noise according to claim 2, characterized in that: Step 2, build the variance adapter, as follows, Predict the duration, pitch and energy of the audio, and construct duration prediction, pitch prediction and energy prediction.

4. A method for improving the naturalness of synthesized audio and reducing noise according to claim 3, characterized in that: In step 2, construct the variance adapter. 2.1 Duration prediction, as follows: 2.1.1, Model structure: To capture the dependencies between sequences, C represents a 1D convolution operation; 2.1.2, Loss function: Use the following formula to optimize the loss function and optimize the duration prediction: Where D pred,i is the duration of the audio of the i-th sample predicted by the model, where P true,i is the actual duration of the audio of the i-th sample. In the training of the deep learning model, each time the loss function is calculated, there will be a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios.

5. A method for improving the naturalness of synthesized audio and reducing noise according to claim 4, characterized in that: In step 2, construct the variance adapter. 2.2, pitch prediction, as follows, 2.2.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 4 layers of convolution; 2.2.2, Loss function: Use the following formula as the loss function to optimize pitch prediction: Where P pred,i is the pitch of the audio of the i-th sample predicted by the model, where P true,i is the actual pitch of the audio of the i-th sample. In the training of the deep learning model, each time the loss function is calculated, there will be a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios.

6. A method for improving the naturalness of synthesized audio and reducing noise according to claim 5, characterized in that in step 2, in constructing a variance adapter, 2.3, Energy prediction, as follows, 2.3.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 8 layers of convolution; 2.3.2, Loss function: Use the following formula as the loss function to optimize energy prediction: Where E pred,i is the energy of the audio of the i-th sample predicted by the model, where E true,i is the actual energy of the audio of the i-th sample. In the training of the deep learning model, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios; 2.4, after the above pitch, energy and duration adjustment, the embedding is recorded as It will undergo further convolution C through the following formula: Description, and finally modeling to obtain the global characteristics of speech.

7. A method for improving the naturalness of synthesized audio and reducing noise according to claim 6, characterized in that step 3, constructing a spectrum denoiser, is specifically as follows: 3.1, after the variance adapter, we get δ, and add the wavelet decomposition to the spectral denoiser for speech synthesis, using Wavelet spec Represents the wavelet transform, the audio obtained after wavelet decomposition, the final spectral denoiser and the vocoder: ε = Wavelet spec (δ).

8. A system for improving the naturalness of synthesized audio and reducing noise, characterized in that: A method for improving the naturalness of synthesized audio and reducing noise as described in any one of claims 1 to 7, the system includes three parts: a phoneme encoder, a variance adapter, and a spectral denoiser. The first part first constructs a phoneme encoder, which encodes and embeds input text. The second part is the construction of a variance adapter, which mainly processes the high-dimensional vector obtained by the phoneme encoder and passes the input feature vector to the variance adapter, thereby further adjusting and mapping it to a time dimension aligned with the target feature. The pitch, duration and energy of the audio are predicted by convolution. The embeddedness after pitch, energy and duration adjustment will be further convolved to model the contextual dependency and global features of speech. The third part is the construction of a spectral denoiser.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, a method for improving the naturalness of synthesized audio and reducing noise as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instruction is executed by a processor, a method for improving the naturalness of synthesized audio and reducing noise is implemented as described in any one of claims 1-7.

Images