A method for improving the naturalness of synthesized audio and reducing noise

Abstract

The present invention relates to a method for improving the naturalness of synthesized audio and reducing noise, comprising the following steps: step 1, constructing a phoneme encoder, step 2, constructing a variance adapter, and step 3, constructing a spectral denoiser, so as to achieve the improvement of the naturalness of the synthesized audio and reduce noise. The scheme proposes adding a Transformer-based bidirectional encoder to the phoneme encoder stage, which can complete the training of the speech synthesis model with less training resources. The present invention is the first to use linear time series modeling with selective state space for speech synthesis tasks, and with the help of the application of wavelet decomposition in the field of image denoising, it is introduced into the speech synthesis network, successfully reducing the noise of the generated audio.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning, artificial intelligence, and speech processing technology, and in particular to a method for reducing training resource usage, improving the naturalness of synthesized audio, and reducing noise. Background Art

[0002] In recent years, text-to-speech (TTS), which aims to generate audio from text that closely resembles human speech, has garnered significant interest in the machine learning community. Text-to-speech technology is crucial for converting text into machine-understandable instructions, enabling accessibility, communication, and enhanced user experiences across a wide range of applications and industries.

[0003] Current text-to-speech methods include traditional speech synthesis methods and deep learning speech synthesis methods. Traditional speech synthesis methods include rule-based methods, concatenation methods, and hidden Markov models (HMMs). Deep learning speech synthesis methods include Tacotron2, FastSpeech, FastSpeech2, FastDiff, ProDiff, and NaturalSpeech. The present inventors have studied these methods and techniques and found that the audio synthesized by these methods still has shortcomings in terms of naturalness, low noise, and low training resource usage.

[0004] To further improve the naturalness of synthesized audio, making it less noisy and using fewer training resources, three questions need to be answered: 1) How to enhance the naturalness of generated speech; 2) How to minimize noise in generated speech; and 3) How to reduce training resource usage in speech synthesis models. In this paper, we conduct an in-depth analysis of these issues in text-to-speech. First, to enhance the naturalness of generated audio, we use a Transformer-based bidirectional encoder as part of the phoneme encoder. Then, we introduce wavelet decomposition to minimize noise in the generated speech. Finally, we reduce training resource usage by replacing each Transformer encoder block in the speech synthesis model with a linear time series layer with a selective state space. With the advancement of linear time series modeling with a selective state space, we have found that linear time series models with a selective state space can consume fewer training resources and achieve similar performance to the Transformer architecture. Considering that the latest speech synthesis models are based on the Transformer architecture, this paper proposes replacing each Transformer encoder block with a linear time series layer with a selective state space. By doing so, we can further reduce training resource usage. Compared with previous speech synthesis methods, this paper has several advantages: 1) Enhanced naturalness of generated speech. In previous cascaded vocoder pipelines, the audio they generated lacked naturalness. Our fully end-to-end text-to-waveform generation can improve naturalness with the help of a Transformer-based bidirectional encoder. 2) Minimize the noise in the generated audio. Previous work generated audio with some noise. By introducing wavelet decomposition, our network has strong data correlation removal capabilities, which is crucial for reducing noise in the generated speech. 3) Reduce training resource usage. Previous models are based on Transformer encoders. When the Transformer model is used for speech synthesis, it has a large number of model parameters, which leads to a large amount of training resources. In this invention, we replace each Transformer encoder block with a linear time series layer with a selective state space, which helps reduce the use of training resources. We perform an evaluation of the generated audio on the widely used LJSpeech dataset. Compared with the latest speech synthesis methods, this invention achieves better results in subjective and objective evaluation indicators of speech synthesis and takes up less training resources. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for reducing training resource usage and improving the naturalness of synthesized audio and reducing noise, so as to solve the deficiencies of the synthesized audio in the existing speech synthesis technology introduced in the background technology in terms of naturalness, low noise and training resource usage.

[0006] In order to solve the above-mentioned shortcomings, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for reducing training resource usage, improving the naturalness of synthesized audio, and reducing noise. The specific steps are as follows:

[0008] Step 1: Build a phoneme encoder.

[0009] 1.1. Construct a bidirectional encoder model based on Transformer and train it on a large-scale English Wikipedia dataset to obtain a trained model, represented by the symbol b. The specific training process includes data preprocessing, model architecture, training parameter setting, and the use of loss function. In the data preprocessing stage, a crawler script was written to fully automatically download the XML dump of Wikipedia, thereby obtaining a large-scale English Wikipedia dataset. Subsequently, the text was cleaned and an automated script was written to implement batch processing, removing tags, links, and special characters at one time to ensure data cleanliness. The WordPiece word segmenter was used to segment the text into subword units. A masked language model task was used to generate training samples. In terms of model architecture, multiple stacked encoder layers were used, each encoder layer including a self-attention mechanism and a feedforward neural network. In terms of training parameters, the initial learning rate was set to 3e-5 and the batch size was set to 64. In terms of loss function, the present invention uses a cross-entropy loss function to optimize the model to maximize the prediction probability of masked words.

[0010] 1.2. Based on 1.1, the present invention integrates the pre-trained model obtained by training with various creative improvement strategies, loads this Transformer-based bidirectional encoder model, and tokenizes the text. That is, the trained Transformer-based bidirectional encoder model and tokenizer are loaded. The symbol A is used to represent the text, the symbol b is used to represent the model, and the symbol ∞ is used to represent the output after this operation. The above process is expressed as follows: ∞ = b(A);

[0011] 1.3. Split the input long sentence into two parts. Use μ to represent the input long sentence, ∪ to represent the sentence segmentation, and θ to represent the final token format that can be understood by the Transformer's bidirectional encoder. The final formula can be expressed as follows: θ = ∪(μ);

[0012] 1.4, through the forward propagation of the Transformer-based bidirectional encoder model, the process can be denoted by the symbol: ∝, the input word is denoted by: ω, and the forward propagation is completed by the following formula: ρ = ∝(ω). Finally, from the last hidden layer of the model: H, the embedding is extracted by the following formula: ∈ = H(ρ);

[0013] 1.5, here we summarize the general expressions of 1.1-1.4, using α to represent the original input and Enc b To represent the encoding of the Transformer-based bidirectional encoder model, the output after 1.1-1.4 is: β = Enc b (α);

[0014] 1.6, Loading a Linear Time Series Model with Selective State Space. Define a linear time series model with selective state space, which uses a selective state space model instead of an attention mechanism to effectively process sequence data;

[0015] 1.7, using a linear time series layer with a selective state space, using the symbol Represents, to replace each Transformer encoder block, here, the present invention proposes to remove the multi-head attention and feedforward sublayers in Transformer, retain the layer normalization and residual connection, and use the rest This effectively ensures the stability of the model and the expected gradient flow during training.

[0016] 1.8, adjust the forward propagation, during the forward pass of the encoder, pass the input sequence to each linear time series layer with selective state space in sequence: For each linear time series layer with a selective state space, the output is calculated based on the selective state space transformation of the relevant sequence information;

[0017] 1.9, use To represent the linear time series modeling with selective state space, encode the output β obtained in 1.5. After 1.6-1.8, we can finally get:

[0018] Step 2, build the variance adapter,

[0019] The core task of the variance adapter of the present invention is to predict the duration, pitch and energy of the audio, so it is necessary to build duration prediction, pitch prediction and energy prediction.

[0020] 2.1, duration prediction:

[0021] 2.1.1, Model structure: To capture the dependencies between sequences, C represents a 1D convolution operation;

[0022] 2.1.2, Loss function: Use the following formula as the loss function to optimize the duration prediction:

[0023] Among them D predis the duration predicted by the model, where D true The actual duration of the audio;

[0024] 2.2, Pitch Prediction:

[0025] 2.2.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 4 layers of convolution;

[0026] 2.2.2, Loss function: Use the following formula as the loss function to optimize pitch prediction:

[0027] Among them, P pred is the pitch predicted by the model, where P true is the actual pitch of the audio;

[0028] 2.3, Energy Prediction:

[0029] 2.3.1, Model structure: to capture the fluctuating pattern of the pitch, Indicates 8 layers of convolution;

[0030] 2.3.2 Loss function: Use the following formula as the loss function to optimize energy prediction:

[0031] Among them E pred is the energy predicted by the model, where E true is the actual energy of the model audio;

[0032] 2.4, after the above pitch, energy and duration adjustments, the embedding is recorded as It will undergo further convolution C through the following formula: Description, and finally the global characteristics of speech are modeled.

[0033] Step 3: Build a spectral denoiser.

[0034] 3.1, after the variance adapter, δ is obtained, which is generally sent to the vocoder or Mel spectrogram decoder to generate the final speech. In order to further reduce the noise, the present invention refers to the application of wavelet decomposition in image denoising and creatively adds wavelet decomposition to the spectrum denoiser for speech synthesis for the first time. spec Represents wavelet transform, then the audio obtained after wavelet decomposition, final spectral denoiser and vocoder: ε = Wavelet spec (δ).

[0035] A system for improving the naturalness of synthesized audio and reducing noise, the system comprising three parts: a phoneme encoder, a variance adapter, and a spectral denoiser. The first part constructs a phoneme encoder, which encodes and embeds input text. The second part constructs a variance adapter, which further processes the high-dimensional vector obtained by the phoneme encoder and passes the input feature vector to the variance adapter for further adjustment and mapping to a time dimension aligned with the target feature. The pitch, duration, and energy of the audio are predicted using convolution. The embedded image after pitch, energy, and duration adjustment is further convolved for modeling the contextual dependency and global features of speech. The third part constructs a spectral denoiser.

[0036] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for enhancing the naturalness of synthesized audio and reducing noise is implemented.

[0037] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement a method for enhancing the naturalness of synthesized audio and reducing noise.

[0038] As a further optimization scheme of the present invention, step 1 creatively proposes for the first time to introduce a Transformer-based bidirectional encoder model into the speech synthesis phoneme encoder. The present invention proposes for the first time to train on a large-scale English Wikipedia dataset to obtain a trained pre-trained model and incorporate it into the speech synthesis phoneme encoder.

[0039] As a further optimization scheme of the present invention, the step 1 creatively proposes for the first time to replace each Transformer encoder block with a linear time series layer with a selective state space. The present invention creatively proposes for the first time to replace each Transformer encoder block with a linear time series layer with a selective state space, and effectively reduces the occupation of training resources.

[0040] As a further optimization solution of the present invention, the wavelet decomposition is integrated into the traditional spectrum denoiser in step 3. The present invention creatively introduces it into the speech synthesis task for the first time and constructs a new spectrum denoiser for speech synthesis.

[0041] As a further optimization solution of the present invention, the present invention trained and tested the neural network built by steps 1-3 on the general speech synthesis test dataset LJSpeech dataset, and compared the speech synthesis method proposed by the present invention with the existing speech synthesis method on the same test dataset. The results are as follows:

[0042] Method MOS PESQ NB_PESQ STOI Tacotron2 2.5345 2.4159 2.4134 0.6423 FastSpeech 2.7944 2.6189 2.6495 0.6779 FastSpeech2 2.8256 2.7439 2.7660 0.7750 FastDiff 2.8483 2.8164 2.8246 0.7993 ProDiff 2.9343 2.8305 2.8722 0.8123 NaturalSpeech 3.1241 3.0134 3.1326 0.8312 Method of the present invention 3.4906 3.5018 3.5342 0.8713

[0043] It can be seen from the above table that the method proposed in the present invention has achieved the best results in both subjective and objective evaluation indicators of speech synthesis compared with existing speech synthesis methods. It can be seen that the present invention has outstanding substantive features and significant progress.

[0044] As a further optimization solution of the present invention, the present invention compares the video memory resources used by various existing speech synthesis models during the training phase. The results are as follows:

[0045]

[0046]

[0047] From the above table, it can be seen that the method proposed in the present invention has achieved the best results in terms of training resource occupation of speech synthesis compared with existing speech synthesis methods. It can be seen that the present invention has made significant progress compared with existing speech synthesis methods.

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] ① Occupy fewer training resources: The present invention can complete the training of the speech synthesis model using fewer training resources;

[0050] ② The generated audio has less noise: This invention uses the application of wavelet decomposition in the field of image denoising and introduces it into the speech synthesis network, successfully reducing the noise of the generated audio;

[0051] ③ The generated audio is highly natural: The present invention compares existing speech synthesis models on the same test set. From the subjective and objective evaluation indicators of speech synthesis, it can be seen that the audio generated by the present invention is of high quality.

[0052] In general, the present invention solves the shortcomings of the existing speech synthesis methods introduced in the background technology in terms of naturalness, low noise and use of training resources. This method can use the public LJSpeech to construct the training set required for speech synthesis, and verify the effect of the speech synthesis model on the same test set, improve the naturalness of the synthesized audio, reduce the noise of the speech synthesis audio, and occupy fewer training resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0055] Example: See Figure 1 , a method for improving the naturalness of synthesized audio and reducing noise, the specific steps are as follows:

[0056] Step 1: Build a phoneme encoder.

[0057] 1.1, build a bidirectional encoder model based on Transformer and train it on a large-scale English Wikipedia dataset, so that we can obtain a trained model, denoted by symbol b;

[0058] 1.2. Load the Transformer-based bidirectional encoder model and tokenize the text. That is, load the trained Transformer-based bidirectional encoder model and tokenizer. Use symbol A to represent the text, symbol b to represent the model, and symbol ∞ to represent the output after this operation. The above process can be expressed as follows: ∞ = b(A);

[0059] 1.3. Split the input long sentence into two parts. Use μ to represent the input long sentence, ∪ to represent the sentence segmentation, and θ to represent the final token format that can be understood by the Transformer's bidirectional encoder. The final formula can be expressed as follows: θ = ∪(μ);

[0060] 1.4, through the forward propagation of the Transformer-based bidirectional encoder model, the process can be denoted by the symbol: ∝, the input word is denoted by: ω, and the forward propagation is completed by the following formula: ρ = ∝(ω). Finally, from the last hidden layer of the model: H, the embedding is extracted by the following formula: ∈ = H(ρ);

[0061] 1.5, here we summarize the general expressions of 1.1-1.4, using α to represent the original input and Enc b To represent the encoding of the Transformer-based bidirectional encoder model, the output after 1.1-1.4 is: β = Enc b (α);

[0062] 1.6, Loading a Linear Time Series Model with Selective State Space. Define a linear time series model with selective state space, which uses a selective state space model instead of an attention mechanism to effectively process sequence data;

[0063] 1.7, using a linear time series layer with a selective state space, using the symbol Represents, to replace each Transformer encoder block, here, the present invention proposes to remove the multi-head attention and feedforward sublayers in Transformer, retain the layer normalization and residual connection, and use the rest This effectively ensures the stability of the model and the expected gradient flow during training.

[0064] 1.8, adjust the forward propagation, during the forward pass of the encoder, pass the input sequence to each linear time series layer with selective state space in sequence: For each linear time series layer with a selective state space, the output is calculated based on the selective state space transformation of the relevant sequence information;

[0065] 1.9, use To represent the linear time series modeling with selective state space, encode the output β obtained in 1.5. After 1.6-1.8, we can finally get:

[0066] Step 2, build the variance adapter,

[0067] The core task of the variance adapter of the present invention is to predict the duration, pitch and energy of the audio, so it is necessary to build duration prediction, pitch prediction and energy prediction.

[0068] 2.1, duration prediction:

[0069] 2.11, Model structure: Use To capture the dependencies between sequences, C represents a 1D convolution operation;

[0070] 2.1.2, Loss function: Use the following formula as the loss function to optimize the duration prediction:

[0071] Among them D pred,i is the duration of the audio of the i-th sample predicted by the model, where P true,i is the actual duration of the audio of the i-th sample. In the training of deep learning models, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios;

[0072] 2.2, Pitch Prediction:

[0073] 2.2.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 4 layers of convolution;

[0074] 2.2.2, Loss function: Use the following formula as the loss function to optimize pitch prediction:

[0075] Among them, P pred,i is the pitch of the audio of the i-th sample predicted by the model, where P true,i is the actual pitch of the audio of the i-th sample. In the training of deep learning models, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios;

[0076] 2.3, Energy Prediction:

[0077] 2.3.1, Model structure: to capture the fluctuating pattern of the pitch, Indicates 8 layers of convolution;

[0078] 2.3.2 Loss function: Use the following formula as the loss function to optimize energy prediction:

[0079] Among them E pred,i is the energy of the audio of the i-th sample predicted by the model, where E true,i is the actual energy of the audio of the i-th sample. In the training of deep learning models, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios.

[0080] 2.4, after the above pitch, energy and duration adjustments, the embedding is recorded as It will undergo further convolution C, thus passing the following formula: Modeling to obtain the global characteristics of speech;

[0081] Step 3: Build a spectral denoiser.

[0082] 3.1, after the variance adapter, δ is obtained, which is generally sent to the vocoder or Mel spectrogram decoder to generate the final speech. In order to further reduce the noise, the present invention refers to the application of wavelet decomposition in image denoising and creatively adds wavelet decomposition to the spectrum denoiser for speech synthesis for the first time. spec Represents wavelet transform, then the audio obtained after wavelet decomposition, final spectral denoiser and vocoder: ε = Wavelet spec (δ).

[0083] The schematic diagram of the method proposed by the present invention for reducing training resource usage and improving the naturalness of synthesized audio and noise reduction is as follows: Figure 1As shown in the figure, the diagram mainly consists of three parts: the phoneme encoder, the variance adapter, and the spectral denoiser. The first part involves the construction of the phoneme encoder, which encodes and embeds the input text. Compared to existing technologies, this innovatively uses a Transformer-based bidirectional encoder model and linear time series modeling with a selective state space. The second part involves the construction of the variance adapter, which further processes the high-dimensional vector obtained by the phoneme encoder. The input feature vector is passed to the variance adapter for further adjustment and mapping to a time dimension aligned with the target feature. Using convolution, the audio pitch, duration, and energy are predicted. The pitch, energy, and duration-adjusted embeddings are further convolved to model the contextual dependencies and global characteristics of speech. The third part involves the construction of the spectral denoiser. Compared to existing technologies, this innovatively introduces wavelet decomposition to further improve the quality of synthesized audio, representing a significant improvement.

[0084] The above is an implementation method of this patent. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that it is still possible to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein with equivalents; and these modifications or replacements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for improving the naturalness of synthesized audio and reducing noise, comprising the following steps: Step 1: Build a phoneme encoder. Step 2, build the variance adapter, Step 3: Build a spectral denoiser to improve the naturalness of the synthesized audio and reduce noise; Among them, step 1 is to build a phoneme encoder, as follows: 1.

1. Build a Transformer-based bidirectional encoder model and train it on a large-scale English Wikipedia dataset to obtain a trained model, denoted by symbol b. The specific training process includes data preprocessing, model architecture, training parameter setting, and the use of loss function. In the data preprocessing stage, a crawler script was written to fully automate the download of Wikipedia's XML dump, thereby obtaining a large-scale English Wikipedia dataset. Subsequently, the text was cleaned and an automated script was written to implement batch processing, removing tags, links, and special characters at one time to ensure data cleanliness. The WordPiece tokenizer was used to segment the text into subword units, and the masked language model task was used to generate training samples. In terms of model architecture, multiple stacked encoder layers were used, each of which included a self-attention mechanism and a feedforward neural network. In terms of training parameters, the initial learning rate was set to 3e-5 and the batch size was set to 64. In terms of loss function, the cross-entropy loss function was used to optimize the model and maximize the prediction probability of masked words. 1.

2. Load this Transformer-based bidirectional encoder model and tokenize the text. That is, load the trained Transformer-based bidirectional encoder model and tokenizer. Use symbol A to represent the text, symbol b to represent the model, and symbol ∞ to represent the output after this operation. The above process is expressed as follows: ∞ = b(A); 1.

3. Split the input long sentence into two parts. Use μ to represent the input long sentence, ∪ to represent the sentence segmentation, and θ to represent the final token format that can be understood by the Transformer's bidirectional encoder. The final formula is as follows: θ = ∪(μ); 1.4, through the forward propagation of the Transformer-based bidirectional encoder model, the process is denoted by ∝, the input word is denoted by ω, and the forward propagation is completed by the following formula: ρ = ∝(ω). Finally, from the last hidden layer of the model: H, the embedding is extracted by the following formula: ∈ = H(ρ); 1.5, summarize the general expressions of 1.1-1.4, use α to represent the original input, and use Enc b To represent the encoding of the Transformer-based bidirectional encoder model, the output after 1.1-1.4 is: β = Enc b (α); 1.6, Loading the Linear Time Series Model with Selective State Space, defining a linear time series model with selective state space, which uses the selective state space model to effectively process sequence data; 1.7, using a linear time series layer with a selective state space, using the symbol Represents, to replace each Transformer encoder block, it is proposed to remove the multi-head attention and feedforward sublayers in the Transformer, retain the layer normalization and residual connection, and use the rest The stack effectively ensures the stability of the model and the expected gradient flow during training; 1.8, adjust the forward propagation, during the forward pass of the encoder, pass the input sequence to each linear time series layer with selective state space in sequence: For each linear time series layer with a selective state space, the output is calculated based on the selective state space transformation of the relevant sequence information; 1.9, use To represent the linear time series modeling with selective state space, encode the output β obtained in 1.

5. After 1.6-1.8, we can finally get:

2. The method for improving the naturalness of synthesized audio and reducing noise according to claim 1, characterized in that: Step 2: Build the variance adapter as follows: Predict the duration, pitch, and energy of the audio, and build duration prediction, pitch prediction, and energy prediction.

3. The method for improving the naturalness of synthesized audio and reducing noise according to claim 2, characterized in that: In step 2, build the variance adapter. 2.1 Duration prediction, as follows: 2.1.1, Model structure: To capture the dependencies between sequences, C represents a 1D convolution operation; 2.1.2, Loss function: Use the following formula as the loss function to optimize the duration prediction: Among them D pred,i is the duration of the audio of the i-th sample predicted by the model, where P true,i is the actual duration of the audio of the i-th sample. In the training of the deep learning model, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios.

4. The method for improving the naturalness of synthesized audio and reducing noise according to claim 3, characterized in that: In step 2, build the variance adapter. 2.2, pitch prediction, specifically as follows, 2.2.1, Model structure: to capture the fluctuating pattern of the pitch, Represents 4 layers of convolution; 2.2.2, Loss function: Use the following formula as the loss function to optimize pitch prediction: Among them, P pred,i is the pitch of the audio of the i-th sample predicted by the model, where P true,i is the actual pitch of the audio of the i-th sample. In the training of the deep learning model, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios.

5. The method for improving the naturalness of synthesized audio and reducing noise according to claim 4, wherein in step 2, in constructing the variance adapter, 2.3, Energy prediction, specifically as follows, 2.3.1, Model structure: to capture the fluctuating pattern of the pitch, Indicates 8 layers of convolution; 2.3.2 Loss function: Use the following formula as the loss function to optimize energy prediction: Among them E pred,i is the energy of the audio of the i-th sample predicted by the model, where E true,i is the actual energy of the audio of the i-th sample. In the training of deep learning models, each loss function calculation will have a batch of data. Here N represents the total number of this batch of data, which refers to the number of audios. 2.4, after the above pitch, energy and duration adjustments, the embedding is recorded as It will undergo further convolution C through the following formula: Description, and finally the global characteristics of speech are modeled.

6. The method for improving the naturalness of synthesized audio and reducing noise according to claim 5, characterized in that, in step 3, a spectrum noise reducer is constructed, specifically as follows: 3.1, after the variance adapter, we get δ, and add the wavelet decomposition to the spectrum denoiser for speech synthesis, using Wavelet spec Represents the wavelet transform, the audio obtained after wavelet decomposition, the final spectrum denoiser and vocoder: ε = Wavelet spec (δ).

7. A system for improving the naturalness of synthesized audio and reducing noise, characterized in that: A method for improving the naturalness of synthesized audio and reducing noise as described in any one of claims 1 to 6, the system includes three parts: a phoneme encoder, a variance adapter, and a spectral denoiser. The first part first constructs a phoneme encoder, which encodes and embeds the input text. The second part is the construction of the variance adapter, which mainly processes the high-dimensional vector obtained by the phoneme encoder and passes the input feature vector to the variance adapter for further adjustment and mapping to a time dimension aligned with the target feature. The pitch, duration and energy of the audio are predicted by convolution. The embedding after pitch, energy and duration adjustment will be further convolved to model the context dependency and global features of speech. The third part is the construction of the spectral denoiser.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for improving the naturalness of synthesized audio and reducing noise as described in any one of claims 1 to 6 above is implemented.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, a method for improving the naturalness of synthesized audio and reducing noise is implemented as described in any one of claims 1 to 6.

Images