Speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and device, electronic equipment and a storage medium. The speech synthesis method comprises the following steps: acquiring an intermediate acoustic feature generated according to a target text, wherein the intermediate acoustic feature corresponds to a to-be-synthesized target speech; obtaining a prior noise signal, and extracting a time domain envelope feature of the target voice from the intermediate acoustic feature; and inputting the prior noise signal, the intermediate acoustic feature and the time domain envelope feature into a pre-trained neural vocoder, and obtaining a target voice which is output by the neural vocoder and is matched with the target text.

Description

Technical Field

[0001] The present invention relates to the field of AI (Artificial Intelligence), and in particular to a speech synthesis method, device, electronic device and storage medium. Background Art

[0002] As an important branch of AI, TTS (Text-to-Speech) combines speech synthesis, artificial intelligence, natural language generation and other related technologies, aiming to convert plain text into natural speech recognizable by human ears. TTS has been widely used in intelligent voice broadcasting, navigation, smart speakers, voice assistants and other products.

[0003] The TTS model used to implement text-based speech usually consists of a text front-end, an acoustic model, and a vocoder. As the core components of the TTS model, the latter two usually use a neural codec network and a neural vocoder built based on neural network technology. The neural codec network is used to extract the intermediate acoustic features of speech, while the neural vocoder is used to convert the features into speech signals as the output of the TTS model.

[0004] Conventional neural vocoders based on GAN (Generative Adversarial Networks) require complex discriminator configurations, resulting in strong model instability and easy crash during operation. To this end, the relevant technology proposes a DDPM (Denoising Diffusion Probabilistic Model) vocoder. However, although this type of diffusion-based vocoder can quickly reconstruct speech signals from acoustic features, it often suffers from significant loss of speech features after denoising, prolonged denoising process, excessive model parameters leading to increased computational complexity, and slow model convergence. Summary of the invention

[0005] In view of this, the present invention provides a speech synthesis method, device, electronic device and storage medium to solve the deficiencies in the related art.

[0006] Specifically, the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, there is provided a speech synthesis method, comprising:

[0008] Acquiring intermediate acoustic features generated according to the target text, wherein the intermediate acoustic features correspond to the target speech to be synthesized;

[0009] Acquiring a priori noise signal, and extracting the time domain envelope features of the target speech from the intermediate acoustic features;

[0010] The prior noise signal, the intermediate acoustic features and the time domain envelope features are input into a pre-trained neural vocoder, and a target speech output by the neural vocoder that matches the target text is obtained.

[0011] According to a second aspect of the present invention, there is provided a speech synthesis device, comprising:

[0012] A feature acquisition unit, configured to acquire intermediate acoustic features generated according to the target text, wherein the intermediate acoustic features correspond to the target speech to be synthesized;

[0013] A feature extraction unit, used to obtain a priori noise signal and extract the time domain envelope feature of the target speech from the intermediate acoustic feature;

[0014] The speech synthesis unit is used to input the prior noise signal, the intermediate acoustic features and the time domain envelope features into a pre-trained neural vocoder, and obtain a target speech output by the neural vocoder that matches the target text.

[0015] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0016] a processor; a memory for storing instructions executable by the processor;

[0017] The processor implements the steps of the method described in the first aspect by running the executable instructions.

[0018] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0019] In the embodiment of the present invention, a priori noise signal is first generated according to the intermediate acoustic features corresponding to the target speech to be synthesized, and the time domain envelope features of the target speech are extracted from the intermediate acoustic features, and then the priori noise signal, the intermediate acoustic features and the time domain envelope features are input into the pre-trained neural vocoder for inference, and the target speech matched to the target text is obtained, thereby completing the TTS process. The time domain envelope features refer to the amplitude variation pattern of the target speech in the time domain, which are objective acoustic features of the speech signal in the time domain.

[0020] It can be understood that in addition to the intermediate acoustic features corresponding to the target speech, this scheme also obtains the time domain envelope features of the speech and inputs them into the neural vocoder (of course, the neural vocoder needs to be trained in advance using the sample time domain envelope features) for predicting the target speech. Therefore, the vocoder can learn more speech features more accurately during the reasoning process, thereby avoiding the loss of speech features while improving the speech quality of the predicted target speech; and reducing the number of variables, thereby reducing the amount of model calculations, accelerating the model convergence speed, and thus improving the speech generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of the structure of a DDPM vocoder shown in an embodiment of the present invention;

[0023] Figure 2 is a flow chart of a neural vocoder training method shown in an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of a process of extracting time domain envelope features shown in an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of the principle of a DWT and an iDWT shown in an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of the structure of a vocoder to be trained shown in an embodiment of the present invention;

[0027] Figure 6 is a flow chart of a speech synthesis method shown in an embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of the structure of a neural vocoder shown in an embodiment of the present invention;

[0029] Figure 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present invention;

[0030] Fig. 9 The block diagram of a speech synthesis device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention.

[0032] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0034] As an important branch of AI, TTS combines speech synthesis, artificial intelligence, natural language generation and other related technologies, aiming to convert plain text into natural speech recognizable by human ears. TTS has been widely used in intelligent voice broadcasting, navigation, smart speakers, voice assistants and other products.

[0035] The TTS model used to implement Wensheng speech usually consists of a text front-end, an acoustic model, and a vocoder, of which the vocoder is the core component of TTS, which is used to convert the intermediate acoustic features extracted by the acoustic model into speech signals. Among them, the vocoder based on a neural network (i.e., a neural vocoder) is a common type of vocoder.

[0036] Traditional neural vocoders can be divided into two categories: AR (Auto regressive) vocoders and NAR (non-AR) vocoders. The AR vocoder is based on an autoregressive model, and its output at the current moment depends on the output at the previous moment. For example, WaveNet is a representative AR vocoder that generates the entire audio waveform in a step-by-step manner by predicting subsequent sampling points. Although the AR vocoder can generate high-quality audio waveforms and has a high degree of fidelity in capturing detailed information of the audio, its speech generation speed is slow because it needs to generate audio waveform samples sample by sample.

[0037] Unlike AR vocoders, non-AR vocoders usually do not rely on the output of the previous moment, but directly generate the entire audio waveform. WaveGlow and MelGAN are two typical non-AR vocoders, which can avoid the tedious process of generating audio waveforms sample by sample, but the audio waveforms generated by non-AR vocoders may not reach the same quality level as those generated by AR vocoders.

[0038] In view of the above-mentioned defects of traditional vocoders, the industry has proposed a GAN-based neural encoder. For example, HiFi GAN can achieve a better balance between efficiency and performance. In addition, the industry has also proposed a multi-scale subband constant Q transform (MS-SB-CQT) discriminator and a multi-scale time-compressed continuous wavelet transform (MS-TC-CWT) discriminator to improve the speech synthesis quality of GAN-based neural encoders. However, the above-mentioned GAN-based neural vocoders usually require complex discriminator configurations, so the model is highly unstable and the operation process is prone to crash.

[0039] To this end, the related technology further proposes a DDPM vocoder. Figure 1 Figure 2 shows the speech synthesis principle of a typical DDPM vocoder. Figure 1As shown, the speech synthesis of the vocoder includes two parts: the forward diffusion process and the reverse inverse diffusion process. The diffusion process starts from the original, clear speech data (such as the pure speech signal corresponding to a text), which is the starting point of the entire diffusion process. The model now has high-quality, noise-free speech information. Then, noise is gradually added to the speech data according to the Gaussian distribution at a predetermined number of time steps (usually represented by "T"). The above-mentioned process of adding noise has a Markov property, that is, the state of each time step depends only on the state of the previous time step and the currently added noise, and has nothing to do with the previous time step. This makes the entire forward diffusion process easier to model and analyze mathematically. The reverse diffusion process belongs to the learning and training stage of the model. Its goal is to train a deep learning model so that it can gradually remove noise from a noisy speech signal without knowing the original speech signal, and restore a speech signal close to the original. The above goal is achieved by letting the model learn to predict the noise added to the speech signal. During the training process, noise is predicted and denoised. For example, for each time step, the model predicts the average value of the noise added from the current time step to the previous time step based on the current noise signal and the corresponding time step information, and then subtracts the predicted average value from the current noise signal to remove the noise and make the speech signal gradually clear. After multiple time steps of reverse denoising, the model can eventually gradually restore the speech signal that is completely submerged by noise to a close to original, clear speech signal. Through the above inverse diffusion process, the model can learn the mapping relationship from noise to clear speech, and thus has the ability to generate new speech signals.

[0040] However, although this type of diffusion-based vocoder can quickly reconstruct speech signals from acoustic features, it often suffers from problems such as significant loss of speech features after denoising, prolonged denoising process, excessive number of model parameters resulting in increased computational effort, and slow model convergence.

[0041] In view of the above-mentioned technical problems existing in the related technologies, the present invention proposes a new speech synthesis method, which introduces the time domain envelope features that can characterize the objective acoustic features of speech into the training and reasoning process of the neural vocoder, in order to enhance the neural vocoder to learn more speech features more accurately in the process of generating the target speech, thereby effectively improving the speech quality of the predicted target speech, accelerating the model convergence speed and reducing its computational complexity.

[0042] The following is a detailed description of an embodiment of the speech synthesis method of the present invention in conjunction with the accompanying drawings.

[0043] The speech synthesis method described in the embodiment of the present invention needs to be implemented based on a special neural vocoder. The training process of the neural vocoder is first introduced below. Figure 22 is a flowchart of a neural vocoder training method according to an embodiment of the present invention, the method comprising the following steps 202 to 206. Steps 202 to 204 are used to prepare training samples, and step 206 is used to perform model training using the training samples.

[0044] Step 202: Acquire sample speech and sample intermediate acoustic features of the sample speech.

[0045] First of all, it should be noted that the training method of the neural vocoder can be applied to a model training device, which can be a server, for example, a physical server including an independent host, or a virtual server, a cloud server, etc. carried by a host cluster. Of course, in the above training process, relevant technicians often need to interact with the server through the client running in the electronic device they use, such as uploading training samples, setting loss functions, adjusting model parameters, etc. The electronic device can be at least one of a PC (Personal Computer), a mobile phone, a tablet device, a laptop, a PDA (Personal Digital Assistants), a wearable device (such as smart glasses, smart watches, etc.), a VR (Virtual Reality) device, an AR (Augmented Reality) device, etc., and one or more embodiments of this specification do not limit this. The client can be started and run on the electronic device. The client can be a native application (Application, i.e., APP) installed on the electronic device, or it can also be a mini program (Mini Program), a quick application (Quick APP) or other similar forms. Of course, when using web technologies such as HTML5 or similar, relevant functions can be implemented through pages displayed by browsers. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0046] The sample speech described in the embodiment of the present invention is a speech corresponding to a natural language, which can be obtained by speaking and recording, or can also be generated by other TTS technologies or related models, and the embodiment of the present invention is not limited to this. However, it should be noted that the sample speech should be a pure speech without noise (or with noise but the noise ratio is lower than a preset value).

[0047] Before training the neural vocoder, it is necessary to obtain the sample speech and its sample intermediate acoustic features. Among them, the sample intermediate acoustic features described in this specification can be one of Mel spectrum (Mel-spectrogram, that is, Mel spectrogram, or Mel spectrum), logarithmic Mel spectrum (Log-Mel-spectrogram), spectrogram (Spectrogram), linear spectrum (LinearSpectrum), fundamental frequency features (Fundamental Frequency Features, or F0 features). Of course, after using any one of these features as the sample intermediate acoustic feature to train the neural vocoder, this feature also needs to be used as the intermediate acoustic feature in the inference stage to ensure the accuracy of the neural vocoder reasoning result.

[0048] Among them, the Mel spectrum is a spectrum representation obtained by nonlinearly transforming the frequency based on the spectrogram according to the auditory characteristics of the human ear. The Mel frequency scale is a frequency scale based on the auditory perception of the human ear. It converts the linear frequency into the Mel frequency, so that the resolution in the lower frequency area is higher and the resolution in the higher frequency area is relatively low. Since the Mel spectrum is more in line with the auditory characteristics of the human ear, this feature is used as a sample intermediate acoustic feature for training the neural vocoder, so that the trained neural vocoder can extract speech features that are more in line with the laws of human ear perception, thereby helping to improve the quality of the synthesized speech. The following embodiment takes the sample intermediate acoustic features including the Mel spectrum as an example to illustrate the solution.

[0049] In one embodiment, the sample intermediate acoustic features may include a Mel spectrum. In this scenario, when obtaining the sample intermediate acoustic features, the sample speech may be first divided into speech frames located in a plurality of windows, and the speech frames in each window may be subjected to Fourier Transform (FT) to obtain a linear spectrum; then the linear spectrum may be subjected to Mel filtering to obtain the Mel spectrum. This method may smoothly extract the corresponding Mel spectrum from the sample speech through a series of processes such as framing (i.e., dividing speech frames), windowing (i.e., adding time windows), Fourier transform, and Mel filtering.

[0050] Among them, Mel filtering processing can be implemented by Mel filter, and the Mel filter is mainly used to convert the linear spectrum into Mel spectrum, so that the spectrum characteristics are more in line with the auditory perception of the human ear. The human ear's perception of sound frequency is not linear. It is sensitive in the low-frequency part and can distinguish smaller frequency differences; it is relatively less sensitive in the high-frequency part and has a weaker ability to distinguish frequency changes. This conversion of the Mel filter group can better simulate the human ear's perception of sounds of different frequencies, and can provide a feature representation that is more in line with human auditory perception in the field of speech processing. Specifically, the linear spectrum obtained by FT can be input into a Mel filter group (composed of multiple Mel filters). For each filter, the product of the linear spectrum and the filter is calculated, and integrated (summed) over the entire frequency range to obtain the output corresponding to the filter. This output is the value of a frequency point in the Mel spectrum. In this way, through the processing of all Mel filters, a complete Mel spectrum can be obtained, which represents the energy distribution of the sample speech at the Mel frequency scale. The features processed by the Mel filter group can highlight the important information in the sample speech and reduce the interference of irrelevant frequency changes on subsequent processing. For example, in a noisy environment, the Mel spectrum can better focus on the main frequency components of the sample speech, thereby helping to improve the robustness of the trained neural vocoder.

[0051] Step 204, generating a sample priori noise signal according to the sample intermediate acoustic features, and adding the sample priori noise signal to the sample speech to obtain sample noisy speech; and extracting the sample time domain envelope features of the sample speech from the sample intermediate acoustic features.

[0052] After obtaining the sample intermediate acoustic feature, it is necessary to generate a sample priori noise signal based on the feature to add it to the sample speech to obtain the sample noisy speech. It can be understood that because the sample intermediate acoustic feature is a strongly related feature of the sample speech, the sample priori noise signal extracted from the feature is more consistent with the law of the sample speech itself than random noise or preset regular noise, so it can facilitate the vocoder to be trained to learn the relevant features more accurately, thereby improving the prediction accuracy of the model.

[0053] In one embodiment, the prior distribution information of the sample speech can be first extracted from the intermediate acoustic features of the sample, and then the prior distribution information is sampled to obtain the sample prior noise signal. Taking the case where the intermediate acoustic features of the sample include the Mel spectrum as an example, at this time, the mean and variance of the Mel spectrum and other indicators can be calculated as the prior distribution information of the sample speech, and then the information is sampled to obtain the sample prior noise signal. Among them, the sampling method of the prior distribution information can be random sampling, or it can also be stratified sampling, importance sampling, etc., and the embodiment of the present invention does not limit this. It can be understood that the distribution calculated according to the value of the Mel spectrum is easier for the trained vocoder to learn than the general distribution (such as the normal distribution), which helps to improve the training quality of the vocoder. In addition, extracting the prior distribution information from the intermediate acoustic features of the sample can realize adaptive prior, and using the prior distribution information as the forward diffusion target in the training process helps to improve the speech synthesis efficiency of the training object (i.e., the neural vocoder).

[0054] In addition, it is also necessary to extract the sample time-domain envelope features of the sample speech from the intermediate acoustic features of the sample. The time-domain envelope feature (Envelope Feature, referred to as envelope feature) described in the embodiment of this specification is also called envelope tracking response, which is used to describe the neural response of the human brain when listening to speech that changes synchronously with the speech envelope. Studies have shown that the appearance of time-domain envelope features indicates that the auditory cortex encodes information related to the speech envelope, which constitutes an important part of human brain speech processing. The time-domain envelope feature can be visualized as a curve that connects the highest amplitude points of different frequency signals in the speech, reflecting the change of sound pressure over time.

[0055] In one embodiment, when the sample intermediate acoustic feature includes the aforementioned fundamental frequency feature (i.e., F0 feature), the sample time domain envelope feature of the sample speech can be directly generated based on the fundamental frequency feature. When the sample intermediate acoustic feature includes a Mel spectrum, the fundamental frequency feature of each frame in the Mel spectrum can be estimated by using the peak value of the autocorrelation function, and then the sample time domain envelope feature of the sample speech can be generated based on the fundamental frequency feature. The autocorrelation function can be a linear autocorrelation function, a normalized autocorrelation function, or a partial autocorrelation function, etc., and the embodiment of the present invention does not limit this.

[0056] like Figure 3 As shown, the time domain signal of the sample speech corresponds to the corresponding Mel spectrum. Further, the sample time domain envelope feature (i.e., the time domain envelope feature shown in the figure) can be extracted from the Mel spectrum corresponding to the speech. In this way, the sample time domain envelope feature of the sample speech can be accurately extracted to facilitate subsequent model training.

[0057] Step 206, training the neural vocoder using sample data including the sample noisy speech, the sample intermediate acoustic features and the sample time-domain envelope features; wherein the difference between the sample speech and the predicted speech output by the neural vocoder is used as the model loss of the neural vocoder.

[0058] After obtaining the sample noisy speech, the sample intermediate acoustic features, and the intermediate acoustic features time domain envelope features and other information in the aforementioned manner, the sample data including this information can be used to train the neural vocoder. For example, the sample data can be input into the vocoder to be trained to obtain a trained neural vocoder.

[0059] The sample data may include multiple pieces of data, each of which may include information such as the sample noisy speech, the sample intermediate acoustic features, and the time domain envelope features of the intermediate acoustic features. Alternatively, in addition to the information, the sample intermediate acoustic features may also include a sampling time step corresponding to the sample intermediate acoustic features, wherein each time step and a corresponding set of the above information together constitute a piece of sample data.

[0060] In addition, before training, all sample data can be divided into training set (Training Set), test set (TestSet) and validation set (Validation Set). Among them, the test set is a data set used to train the machine learning model. During training, the model uses the samples in the training set to learn features and patterns in order to make predictions or classifications. The test set is a data set used to evaluate the performance of the trained model. After the model training is completed, the test set is used to verify the model's generalization ability for unknown data. The performance of the model on the test set can help evaluate the accuracy and performance of the model. The validation set is a data set used to adjust the model hyperparameters and evaluate the performance of the model. During the training process, the validation set is used to adjust the parameters of the model to prevent the model from overfitting on the training set. In order to balance training efficiency and quality, an appropriate ratio can be set for the sizes of the training set, test set, and validation set. For example, the number of sample data items contained in the three can be reduced successively. For example, 60% of all sample data can be used as the training set, and the remaining 20% ​​can be used as the test set and validation set respectively; or, when the total number of sample data is large, 80% of all sample data can be used as the training set, and the remaining 10% can be used as the test set and validation set respectively. The details will not be repeated here.

[0061] As can be seen from the above embodiments, the sample data used in this solution to train the neural vocoder includes not only the sample noisy speech and the sample intermediate acoustic features of the sample speech, but also the sample time domain envelope features. This method can ensure that the trained neural vocoder can learn more speech features more accurately during the reasoning process, thereby avoiding the loss of speech features, improving the speech quality of the predicted target speech, accelerating the model convergence speed, and reducing its computational complexity.

[0062] The vocoder to be trained is essentially a pre-built neural network, which includes multiple input terminals. During the training process, sample data is input into the vocoder to be trained, that is, each information in the sample data is input into the network through the corresponding input terminals.

[0063] In one embodiment, a piece of sample data can be input into the model separately, or a batch consisting of multiple pieces of sample data can be input into the model in batches to reduce the number of inputs. Regardless of the input method, multiple pieces of information (i.e., a group of information) in an actual piece of sample data will be input in sequence. Exemplarily, a group of the sample noisy speech, the sample intermediate acoustic features, and the sample time-domain envelope features can be respectively input into the corresponding input ends of the vocoder to be trained, and the predicted speech output by the network can be obtained; and, it is also necessary to adjust the network parameters of the vocoder to be trained according to the difference between the predicted speech and the sample speech and continue training until the preset termination condition is met to terminate the training and obtain the neural vocoder. Among them, the termination condition can be that the error of the test result and / or verification result does not exceed the preset value, or the test set has been used up, etc., and the embodiment of the present invention does not limit this.

[0064] In one embodiment, the sampling time step corresponding to the sample intermediate acoustic feature can also be determined, and the sampling time step is used to constitute the sample data. Accordingly, the sample data used to train the neural vocoder at this time may include the aforementioned sample prior noise signal, the sample intermediate acoustic feature, the sample time domain envelope feature and the sampling time step and other information. In this regard, these information can be respectively input into the corresponding input end of the vocoder to be trained for training the neural vocoder.

[0065] In one embodiment, the vocoder to be trained may include a dilated convolution (Dailed Conv, also known as dilated convolution, atrous convolution, also known in English as atrous convolution) block, which may be composed of upsampling convolution, layer normalization and ReLU sequential connection; of course, other structures may also be used to adapt to different processing requirements. The dilated convolution block is used to perform a dilated convolution operation on the sample prior noise signal and the accumulated features of the sampling time step to expand the receptive field of different layers, that is, to allow the convolution kernel to cover a larger area without increasing the number of parameters, thereby obtaining a wider range of contextual information. In addition, the dilated convolution block can also capture long-term dependencies to help the model better understand the long-distance correlation in the speech signal and improve the modeling ability of speech features. It can be seen that the dilated convolution block can reduce the amount of calculation and the number of parameters of the model to a certain extent, while maintaining the ability to capture long-term information.

[0066] In one embodiment, the vocoder to be trained may also include a DWT (Discrete Wavelet Transform) module and an iDWT (Inverse Discrete Wavelet Transform) module, wherein the DWT module is used to perform a discrete wavelet transform operation on the sample prior noise signal to split the sample prior noise signal into a sample high-frequency noise signal and a sample low-frequency noise signal; and the iDWT module is used to perform an inverse discrete wavelet transform operation on the processing results of the sample high-frequency noise signal and the sample low-frequency noise signal to obtain the predicted speech. Relying on the above-mentioned DWT module and iDWT module, the vocoder to be trained can process the sample high-frequency noise signal and the sample low-frequency noise signal split from the sample prior noise signal separately, so that the amount of calculation for each path is small, and ensure that the predicted speech finally generated can simultaneously contain the high-frequency characteristics and low-frequency characteristics of the sample prior noise signal, so as to improve the quality of the predicted speech.

[0067] Among them, DWT is a time-frequency analysis method that decomposes the signal into components of different frequencies along an orthogonal basis, thereby providing local information in time and frequency. After the first-order DWT, the number of channels of the signal changes from 1 dimension to 2 dimensions, while the time dimension is halved. Downsampling is achieved in this way, thereby effectively reducing the number of parameters in the model. This method can provide a relatively more comprehensive understanding of the unique characteristics and potential structure of speech signals.

[0068] The working principles of DWT module and iDWT module can be found in Figure 4 .like Figure 4 The figure shows the classic fast discrete wavelet transform, which is an effective implementation of the one-dimensional DWT. Figure 4As shown, the length of the original signal is 2 j , which can be written as x j =x, Hd(z), Ld(z), Hr(z) and Lr(z) are filter arrays designed according to ideal reconstruction conditions. Hd(z) and Ld(z) are decomposition filters, Hr(z) and Lr(z) are reconstruction filters. ↓2 and ↑2 represent two-way downsampling and two-way upsampling, respectively. After the decomposition filter and downsampling of the signal, the low-frequency component L and the high-frequency component H are obtained.

[0069] Take the scenario where the sample Mel spectrum includes the sample intermediate acoustic features as an example. Figure 5 The internal structure of the model to be trained is drawn. Figure 5 As shown, the vocoder to be trained includes four input terminals, which are respectively used to input the following four parameters: sample Mel spectrum; sampling time step used to characterize the sampling moment, that is, time series t~μ({0, 1, 2, ..., T}); sample noisy speech obtained by adding a sample priori noise signal to the sample language; sample time domain envelope features extracted from the sample Mel spectrum.

[0070] The sampling time step includes T+1 sampling moments, and the vocoder to be trained needs to execute T+1 according to each sampling moment. Figure 5 As shown, the neural network in the vocoder to be trained includes a total of T+1 layers of neural networks (the top layer is shown as a solid line, and the remaining layers are shown as dotted lines), and the internal structure of each layer is the same. Any layer of the neural network contains an expanded convolution block consisting of upsampling convolution, layer normalization, and ReLU sequential connection. Its function is to achieve different perception fields at each layer without changing the size of the feature map, without the need for pooling and striding. The network follows the frequency-aware network of DiffWave, which is the backbone network of the diffusion-based vocoder. Among them, the perception field increases along the time axis without losing information, so each convolution output can contain a wider range of information while maintaining the original duration of the speech signal.

[0071] From the perspective of data flow, the sample speech (i.e., clean speech signal) is first subjected to noise processing (the sample prior noise signal obtained by sampling the sample speech with a prior distribution is added) to obtain the sample noisy speech, and the sample noisy speech is input into a layer of neural network corresponding to a certain time step after DWT processing. After adding the result of one-dimensional convolution (Conv1D) of the time step (the result of the addition is the audio feature containing the sampling step information), it is input into the dilated convolution block. The output of the dilated convolution block is added with the result of one-dimensional convolution of the sample time domain envelope feature and the sample Mel spectrum (i.e., the prior knowledge used as a condition), and the corresponding result is activated by a gated activation unit (including tanh activation function and sigmoid activation function). The outputs of these two nonlinear functions are element-by-element multiplied and then subjected to one-dimensional convolution and iDWT to obtain the output of the neural network layer. The output of each layer of the neural network adopts a residual connection method to avoid information forgetting; and the output of each layer of the neural network is added, and then the final predicted speech is output after one-dimensional convolution, ReLU activation and one-dimensional convolution.

[0072] The difference between the sample speech and the predicted speech output by the neural vocoder is taken as the model loss of the neural vocoder, that is, Loss of the vocoder to be trained = f(|predicted speech - sample speech|).

[0073] It can be seen from the above embodiments that the training process described in this specification is a neural vocoder process that gradually adds noise to a clean sample speech and combines it with prior feature training to obtain a neural vocoder that recovers clean data from the sample noisy speech; in other words, the trained neural vocoder can perform reasoning operations based on intermediate acoustic features and noise (noisy speech can be regarded as a special kind of noise), and finally generate (or synthesize) and output a speech. Given that the noise can be extracted from the intermediate acoustic features, it can also be simply considered that the neural vocoder can generate speech based on the intermediate acoustic features.

[0074] Furthermore, the present invention proposes a method for generating speech using a neural vocoder trained based on the aforementioned embodiment, that is, a method for generating speech using the neural vocoder to perform reasoning. Figure 6 602-606 is a flowchart of the speech synthesis method according to an embodiment of the present invention. The method includes the following steps 602-606.

[0075] Step 602: Acquire intermediate acoustic features generated according to the target text, wherein the intermediate acoustic features correspond to the target speech to be synthesized.

[0076] Similar to the aforementioned training process, this speech generation method (i.e., the inference process) can be executed by the operating device of the neural vocoder (the neural vocoder is deployed and run in this device). This device can be a physical server containing an independent host, or a virtual server, cloud server, etc. hosted by a host cluster. Additionally, the caller of the neural vocoder (such as the user who requests to generate the target speech) needs to interact with the model or related services, so a corresponding client can be run on the electronic device used by itself. This electronic device can also be a PC, mobile phone, tablet computer, etc., and the client can also be in the form of an APP, applet, H5 web page, etc., which will not be elaborated here.

[0077] The target text described in the present invention is text that can be read in natural speech. The embodiments of the present invention do not limit the language to which the text belongs, the number of characters it contains, etc. For example, it can be Chinese, English, Japanese, Korean, Russian, etc. The purpose of this solution is to generate the target speech that is expected to be emitted when reading the target text in natural language. For example, the user can input the Chinese sentence "Where are you going" in the client. After the user submits a speech generation request carrying this text to the server, the server can respond to this request, take this text as the target text, and process it according to steps 602 - 606. Finally, it calls the locally deployed neural vocoder for inference to generate the sound corresponding to this text when speaking this text in Chinese. It can be seen that this solution is expected to generate the sound that should be emitted when reading the target text in the language corresponding to the target text.

[0078] As mentioned above, the neural vocoder used in this solution is trained to generate speech based on intermediate acoustic features. Therefore, it is first necessary to obtain the intermediate acoustic features corresponding to the aforementioned target text. It can be understood that this feature is a feature that can be used to reflect the audio law of the (to-be-generated) target speech, that is, what actually needs to be obtained at this time is the intermediate acoustic features of the target speech - only the intermediate acoustic features have not been generated yet.

[0079] Similar to the sample intermediate acoustic features used in the training stage of the aforementioned neural vocoder, the intermediate acoustic features used in the inference stage of the neural vocoder can also include Mel spectrum, log Mel spectrum, spectrogram, linear spectrum, or fundamental frequency features. Of course, which feature (such as Mel spectrum) was used as the sample intermediate acoustic feature in the training stage, the corresponding feature of the target text should also be obtained as the intermediate acoustic feature in the inference stage to ensure the smooth operation of the neural vocoder in the inference stage, which will not be elaborated here.

[0080] In one embodiment, when obtaining the intermediate acoustic features corresponding to the target text, the phoneme sequence corresponding to the text may be obtained first, and then the intermediate acoustic features corresponding to the target text may be generated according to the phoneme sequence. The phoneme sequence corresponding to the target text may be obtained according to the steps of vocabulary analysis, phoneme splitting and phoneme connection. Exemplarily, each character in the target text (such as Chinese characters, words, English words, English phrases, etc.) may be analyzed first to determine the pronunciation of each character, and then the pronunciation of each character may be split into individual phonemes, and finally, according to the position and front-to-back order of each word in the target text, each phoneme may be connected in sequence to obtain the phoneme sequence corresponding to the target text.

[0081] Alternatively, the target text may be input into a pre-trained G2P (Grapheme to Phoneme) model, and the phoneme sequence corresponding to the target text output by the G2P model may be received. G2P is a key step in speech synthesis, and its purpose is to convert written forms (letters, characters, etc.) into their corresponding pronunciation forms (phonemes). This process usually needs to consider the context, because the pronunciation of a character or letter may be different in different words or sentences. The G2P model can be pre-trained using a large-scale annotated corpus so that the model learns the mapping relationship between characters and phonemes, thereby determining the phonemes corresponding to each character in the target text according to the mapping relationship, and then generating the corresponding phoneme sequence.

[0082] The phoneme sequence may be embedded in the IPA (International Phonetic Alphabet) with a finer granularity to alleviate problems such as missing words and repeated readings.

[0083] In one embodiment, a reference speech may be obtained and conditional features such as acoustic features and / or semantic features may be extracted. Based on this, when generating intermediate acoustic features corresponding to the target text, the intermediate acoustic features corresponding to the target text may be generated according to the phoneme sequence and the above-mentioned conditional features. Among them, the semantic features are used to characterize the contextual semantics between different words in the target text, and the acoustic features may include timbre, pronunciation pattern, high and low audio frequencies, etc. It can be understood that in addition to being based on the phoneme sequence, this solution also generates intermediate acoustic features based on the acoustic features and / or semantic features of the target text, so that intermediate acoustic features of higher quality and carrying more comprehensive information can be generated, thereby helping the neural vocoder to learn the timbre, rhythm and other information of the reference speech, and ultimately generating a relatively natural, fluent and personalized target speech that conforms to the reference speech habits.

[0084] Step 604: Acquire a priori noise signal, and extract the time domain envelope feature of the target speech from the intermediate acoustic feature.

[0085] Furthermore, on the one hand, a priori noise signals can be obtained, and on the other hand, time domain envelope features (of the target speech) can be extracted from the intermediate acoustic features obtained in the aforementioned manner.

[0086] In one embodiment, the sample prior noise signal generated according to the intermediate acoustic features of the sample in the training phase can be directly used in the reasoning process as the prior noise signal to reduce the workload of data preprocessing in the reasoning process. Alternatively, the prior distribution information can be first extracted from the intermediate acoustic features, and then the prior distribution information is sampled to obtain the prior noise signal. Similar to the aforementioned sample prior noise signal, taking the case where the intermediate acoustic features include the Mel spectrum as an example, at this time, the mean and variance of the Mel spectrum and other indicators can be calculated as the prior distribution information of the target speech, and then the information is sampled to obtain the prior noise signal. Among them, the sampling method of the prior distribution information can be random sampling, or it can also be stratified sampling, importance sampling, etc., and the embodiment of the present invention does not limit this. It can be understood that the distribution calculated according to the value of the Mel spectrum is easier for the trained vocoder to learn than the general distribution (such as the normal distribution), which helps to improve the quality of the target speech generated by the neural vocoder. In addition, extracting prior distribution information from intermediate acoustic features can achieve adaptive priors, and using this prior distribution information as the forward diffusion target of inference can help improve the efficiency of the neural vocoder in generating target speech.

[0087] In addition, it is also necessary to extract the time domain envelope features of the target speech from the intermediate acoustic features. In one embodiment, when the intermediate acoustic features include fundamental frequency features, the time domain envelope features of the target speech can be directly generated based on the fundamental frequency features. When the intermediate acoustic features include Mel spectrum, the fundamental frequency features of each frame in the Mel spectrum can be estimated by using the peak value of the autocorrelation function, and then the time domain envelope features of the target speech can be generated based on the fundamental frequency features. Among them, the autocorrelation function can adopt a linear autocorrelation function, a normalized autocorrelation function or a partial autocorrelation function, etc., and the embodiment of the present invention does not limit this.

[0088] Taking the Mel spectrum as an example, the intermediate acoustic features can be extracted by smoothing, peak detection, and envelope fitting. For example, the Mel spectrum can be smoothed first to reduce the impact of noise; then the peak points in the Mel spectrum are found to determine the key positions of the envelope curve; finally, the peak points can be fitted using a suitable function (such as the aforementioned autocorrelation function) to obtain the envelope curve. In addition, the mean, variance and other related features of the target speech can also be extracted from the envelope curve for subsequent processing, which will not be described in detail.

[0089] Step 606: input the prior noise signal, the intermediate acoustic features, and the time domain envelope features into a pre-trained neural vocoder, and obtain a target speech output by the neural vocoder that matches the target text.

[0090] After obtaining the aforementioned prior noise signal, intermediate acoustic features, time domain envelope features and other information, this information can be input into the neural vocoder for inference operations in order to output the target speech corresponding to the target text. In one embodiment, the sampling time step corresponding to the intermediate acoustic feature can also be determined, and the sampling time step is used as an input signal and input into the neural vocoder together with other information. At this time, the information input into the neural vocoder includes four: the Mel spectrum generated according to the target text, the sampling time step, the prior noise signal generated according to the Mel spectrum, and the time domain envelope features extracted from the Mel spectrum.

[0091] In one embodiment, the neural vocoder may include a dilated convolution block, which may be composed of upsampling convolution, layer normalization, and ReLU sequential connections; of course, other structures may also be used to adapt to different processing requirements. The dilated convolution block is used to perform a dilated convolution operation on the accumulated features of the prior noise signal and the sampling time step to expand the receptive field of different layers, that is, to allow the convolution kernel to cover a larger area without increasing the number of parameters, thereby obtaining a wider range of contextual information. In addition, the dilated convolution block can also capture long-term dependencies to help the model better understand the long-distance correlations in the speech signal and improve the ability to model speech features. It can be seen that the dilated convolution block can reduce the amount of computation and the number of parameters of the model to a certain extent, while maintaining the ability to capture long-term information.

[0092] In one embodiment, the neural vocoder may further include a DWT module and an iDWT module, wherein the DWT module is used to perform a discrete wavelet transform operation on the prior noise signal to split the prior noise signal into a high-frequency noise signal and a low-frequency noise signal; and the iDWT module is used to perform an inverse discrete wavelet transform operation on the processing results of the high-frequency noise signal and the low-frequency noise signal to obtain the target speech. Thanks to the above-mentioned DWT module and iDWT module, the neural vocoder can process the high-frequency noise signal and the low-frequency noise signal split from the prior noise signal separately, so that the amount of calculation for each path is small, and ensure that the target speech finally generated can simultaneously contain the high-frequency features and low-frequency features of the prior noise signal to improve the quality of the target speech.

[0093] Corresponding to the above Figure 5 The training process shown, Figure 7 This is the internal structure diagram of the trained neural vocoder. It can be seen that Figure 7 The internal structure of the neural vocoder shown is similar to Figure 5 The internal structure of the vocoder to be trained is the same as shown. From the perspective of data flow, the prior noise signal is processed by DWT and then input into a layer of neural network corresponding to a certain time step. After adding the result of one-dimensional convolution (Conv1D) of the time step (the result of the addition is the audio feature containing the sampling step information), it is input into the dilated convolution block. The output of the dilated convolution block is added with the result of one-dimensional convolution of the time domain envelope feature and the Mel spectrum (i.e., the prior knowledge used as a condition), and the corresponding result is activated by a gated activation unit (including tanh activation function and sigmoid activation function). The outputs of these two nonlinear functions are multiplied element by element and then passed through one-dimensional convolution and iDWT to obtain the output of the neural network layer. The output of each layer of the neural network adopts a residual connection method to avoid information forgetting; and the output of each layer of the neural network is added, and then the final target speech is output after one-dimensional convolution, ReLU activation and one-dimensional convolution.

[0094] In the embodiment of the present invention, a priori noise signal is first generated according to the intermediate acoustic features corresponding to the target speech to be synthesized, and the time domain envelope features of the target speech are extracted from the intermediate acoustic features, and then the priori noise signal, the intermediate acoustic features and the time domain envelope features are input into the pre-trained neural vocoder for inference, and the target speech matched to the target text is obtained, thereby completing the TTS process. The time domain envelope features refer to the amplitude variation pattern of the target speech in the time domain, which are objective acoustic features of the speech signal in the time domain.

[0095] It can be understood that in addition to the intermediate acoustic features corresponding to the target speech, this scheme also obtains the time domain envelope features of the speech and inputs them into the neural vocoder (of course, the neural vocoder needs to be trained in advance using the sample time domain envelope features) for predicting the target speech. Therefore, the vocoder can learn more speech features more accurately during the reasoning process, thereby avoiding the loss of speech features while improving the speech quality of the predicted target speech; and reducing the number of variables, thereby reducing the amount of model calculations, accelerating the model convergence speed, and thus improving the speech generation efficiency.

[0096] Figure 8 is a schematic structural diagram of an electronic device shown in an embodiment of the present invention. Figure 8At the hardware level, the device includes a processor 801, a network interface 802, a memory 803, a non-volatile memory 804, and an internal bus 805, and may also include hardware required for other services. One or more embodiments of the present invention may be implemented based on software, such as the processor 801 reading the corresponding computer program from the non-volatile memory 804 to the memory 803 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present invention do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0097] Fig. 9 A block diagram of a speech synthesis device shown in an embodiment of the present invention. Fig. 9 , the device can be applied to Figure 8 In the device shown, to implement the technical solution described in the present invention, the device includes:

[0098] A feature acquisition unit 901 is used to acquire intermediate acoustic features generated according to the target text, wherein the intermediate acoustic features correspond to the target speech to be synthesized;

[0099] A feature extraction unit 902, configured to obtain a priori noise signal and extract a time domain envelope feature of the target speech from the intermediate acoustic feature;

[0100] The speech synthesis unit 903 is used to input the prior noise signal, the intermediate acoustic features and the time domain envelope features into a pre-trained neural vocoder, and obtain the target speech output by the neural vocoder that matches the target text.

[0101] Optionally, the feature acquisition unit 901 is specifically used for:

[0102] A phoneme sequence corresponding to the target text is obtained, and an intermediate acoustic feature corresponding to the target text is generated according to the phoneme sequence.

[0103] Optionally, it further includes a reference speech acquisition unit 904, which is used to acquire a reference speech and extract conditional features of the reference speech, wherein the conditional features include acoustic features and / or semantic features;

[0104] The feature acquisition unit 901 is specifically used to generate an intermediate acoustic feature corresponding to the target text according to the phoneme sequence and the conditional feature.

[0105] Optionally, the method further includes a first time step determining unit 905, which is used to:

[0106] A sampling time step corresponding to the intermediate acoustic feature is determined, and the sampling time step is input into the neural vocoder.

[0107] Optionally, the intermediate acoustic feature includes a Mel spectrum, and the feature extraction unit 902 is specifically used for:

[0108] The peak value of the autocorrelation function is used to estimate the fundamental frequency feature of each frame in the Mel spectrum, and the time domain envelope feature of the target speech is generated based on the fundamental frequency feature.

[0109] Optionally, a vocoder training unit 906 is further included, configured to:

[0110] Acquiring sample speech and sample intermediate acoustic features of the sample speech;

[0111] Generating a sample priori noise signal according to the sample intermediate acoustic feature, and adding the sample priori noise signal to the sample speech to obtain the sample noisy speech; and extracting the sample time domain envelope feature of the sample speech from the sample intermediate acoustic feature;

[0112] The neural vocoder is trained using sample data including the sample noisy speech, the sample intermediate acoustic features and the sample time-domain envelope features; wherein the difference between the sample speech and the predicted speech output by the neural vocoder is used as the model loss of the neural vocoder.

[0113] Optionally, the sample intermediate acoustic feature includes a Mel spectrum, and the vocoder training unit 906 is specifically used for:

[0114] Dividing the sample speech into speech frames located in a plurality of windows, and performing Fourier transform on the speech frames in each window to obtain a linear spectrum of the sample speech;

[0115] The linear frequency spectrum is subjected to Mel filtering to obtain the Mel spectrum.

[0116] Optionally, the sample intermediate acoustic feature includes a Mel spectrum, and the vocoder training unit 906 is specifically used for:

[0117] The peak value of the autocorrelation function is used to estimate the fundamental frequency feature of each frame in the Mel spectrum, and the sample time domain envelope feature of the sample speech is generated based on the fundamental frequency feature.

[0118] Optionally, the vocoder training unit 906 is specifically used for:

[0119] The prior distribution information of the sample speech is extracted from the intermediate acoustic features of the sample, and the prior distribution information is sampled to obtain a sample prior noise signal.

[0120] Optionally, the vocoder training unit 906 is specifically used for:

[0121] Inputting a group of the sample noisy speech, the sample intermediate acoustic features and the sample time-domain envelope features into corresponding input terminals of the vocoder to be trained, and obtaining the predicted speech output by the network;

[0122] The network parameters of the vocoder to be trained are adjusted according to the difference between the predicted speech and the sample speech, and the training is continued until the training is terminated when a preset termination condition is met, thereby obtaining the neural vocoder.

[0123] Optionally, the method further includes a second time step determining unit 907, which is used to:

[0124] A sampling time step corresponding to the intermediate acoustic feature of the sample is determined, and the sampling time step is used to form the sample data.

[0125] Optionally, the neural vocoder includes a dilated convolution block, which is used to perform a dilated convolution operation on the prior noise signal and the accumulated features of the sampling time step to expand the receptive fields of different layers.

[0126] Optionally, the neural vocoder comprises a DWT module and an iDWT module, wherein:

[0127] The DWT module is used to perform a discrete wavelet transform operation on the prior noise signal to split the prior noise signal into a high-frequency noise signal and a low-frequency noise signal;

[0128] The iDWT module is used to perform an inverse discrete wavelet transform operation on the processing results of the high-frequency noise signal and the low-frequency noise signal to obtain the target speech.

[0129] Optionally, the intermediate acoustic feature or the sample intermediate acoustic feature includes one of the following:

[0130] Mel spectrum, logarithmic Mel spectrum, spectrogram, linear spectrum, fundamental frequency characteristics.

[0131] Accordingly, the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the speech synthesis method as described in any of the above embodiments is implemented.

[0132] Accordingly, the present specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the speech synthesis method described in any of the above embodiments.

[0133] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0134] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0135] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0136] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

Claims

1. A speech synthesis method, characterized in that: include: Acquiring intermediate acoustic features generated according to the target text, wherein the intermediate acoustic features correspond to the target speech to be synthesized; Acquire a priori noise signal, and extract the time domain envelope feature of the target speech from the intermediate acoustic feature; The prior noise signal, the intermediate acoustic features and the time domain envelope features are input into a pre-trained neural vocoder, and a target speech output by the neural vocoder that matches the target text is obtained.

2. The method according to claim 1, characterized in that The obtaining of the intermediate acoustic features generated according to the target text includes: A phoneme sequence corresponding to the target text is obtained, and an intermediate acoustic feature corresponding to the target text is generated according to the phoneme sequence.

3. The method according to claim 2, characterized in that The method further comprises: acquiring a reference speech, and extracting conditional features of the reference speech, wherein the conditional features include acoustic features and / or semantic features; The generating the intermediate acoustic features corresponding to the target text according to the phoneme sequence includes: generating the intermediate acoustic features corresponding to the target text according to the phoneme sequence and the conditional features.

4. The method according to claim 1, characterized in that: Also includes: A sampling time step corresponding to the intermediate acoustic feature is determined, and the sampling time step is input into the neural vocoder.

5. The method according to claim 1, characterized in that: The intermediate acoustic feature includes a Mel spectrum, and the step of extracting a time domain envelope feature of the target speech from the intermediate acoustic feature includes: The peak value of the autocorrelation function is used to estimate the fundamental frequency feature of each frame in the Mel spectrum, and the time domain envelope feature of the target speech is generated based on the fundamental frequency feature.

6. The method according to claim 1, characterized in that The vocoder is trained in the following manner: Acquiring sample speech and sample intermediate acoustic features of the sample speech; Generating a sample priori noise signal according to the sample intermediate acoustic feature, and adding the sample priori noise signal to the sample speech to obtain the sample noisy speech; and extracting the sample time domain envelope feature of the sample speech from the sample intermediate acoustic feature; The neural vocoder is trained using sample data including the sample noisy speech, the sample intermediate acoustic features and the sample time-domain envelope features; wherein the difference between the sample speech and the predicted speech output by the neural vocoder is used as the model loss of the neural vocoder.

7. The method according to claim 6, characterized in that The sample intermediate acoustic feature includes a Mel spectrum, and the step of obtaining the sample intermediate acoustic feature of the sample speech includes: Dividing the sample speech into speech frames located in a plurality of windows, and performing Fourier transform on the speech frames in each window to obtain a linear spectrum of the sample speech; The linear frequency spectrum is subjected to Mel filtering to obtain the Mel spectrum.

8. The method according to claim 6, characterized in that The sample intermediate acoustic feature includes a Mel spectrum, and extracting a sample time domain envelope feature of the sample speech from the sample intermediate acoustic feature includes: The peak value of the autocorrelation function is used to estimate the fundamental frequency feature of each frame in the Mel spectrum, and the sample time domain envelope feature of the sample speech is generated based on the fundamental frequency feature.

9. The method according to claim 6, characterized in that Generating a sample priori noise signal according to the sample intermediate acoustic feature comprises: The prior distribution information of the sample speech is extracted from the intermediate acoustic features of the sample, and the prior distribution information is sampled to obtain a sample prior noise signal.

10. The method according to claim 6, characterized in that The method of training the neural vocoder using the sample data consisting of the sample noisy speech, the sample intermediate acoustic features and the sample time domain envelope features includes: Inputting a group of the sample noisy speech, the sample intermediate acoustic features and the sample time-domain envelope features into corresponding input terminals of the vocoder to be trained, and obtaining the predicted speech output by the network; The network parameters of the vocoder to be trained are adjusted according to the difference between the predicted speech and the sample speech, and the training is continued until the training is terminated when a preset termination condition is met, thereby obtaining the neural vocoder.

11. The method according to claim 6, characterized in that Also includes: A sampling time step corresponding to the intermediate acoustic feature of the sample is determined, and the sampling time step is used to form the sample data.

12. The method according to claim 11, characterized in that The neural vocoder comprises a dilated convolution block, which is used to perform a dilated convolution operation on the prior noise signal and the accumulated features of the sampling time step to expand the receptive fields of different layers.

13. The method according to any one of claims 1 to 11, characterized in that The neural vocoder comprises a DWT module and an iDWT module, wherein: The DWT module is used to perform a discrete wavelet transform operation on the prior noise signal to split the prior noise signal into a high-frequency noise signal and a low-frequency noise signal; The iDWT module is used to perform an inverse discrete wavelet transform operation on the processing results of the high-frequency noise signal and the low-frequency noise signal to obtain the target speech.

14. The method according to any one of claims 1 to 4, 6 and 9 to 12, characterized in that The intermediate acoustic feature or the sample intermediate acoustic feature includes one of the following: Mel spectrum, logarithmic Mel spectrum, spectrogram, linear spectrum, fundamental frequency characteristics.

15. A speech synthesis device, characterized in that: include: A feature acquisition unit, configured to acquire intermediate acoustic features generated according to the target text, wherein the intermediate acoustic features correspond to the target speech to be synthesized; A feature extraction unit, used to obtain a priori noise signal and extract the time domain envelope feature of the target speech from the intermediate acoustic feature; The speech synthesis unit is used to input the prior noise signal, the intermediate acoustic features and the time domain envelope features into a pre-trained neural vocoder, and obtain a target speech output by the neural vocoder that matches the target text.

16. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 14 by running the executable instructions.

17. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

Images