Speech synthesis system and method based on multi-sub-band generation strategy, medium and equipment

Abstract

The embodiment of the invention discloses a speech synthesis system and method based on a multi-subband generation strategy, a medium and equipment, and the system comprises a preprocessing module which is used for converting a text into a phoneme sequence, and converting corresponding speech into a linear frequency spectrum through short-time Fourier transform; the posterior encoder module is used for converting the linear frequency spectrum into posterior distribution, and sampling according to a mean value and a standard deviation in the posterior distribution to obtain a potential variable; the conversion layer module is used for converting the potential variables into sampling data; the priori encoder module is used for extracting the phoneme sequence as a depth feature and mapping the depth feature into priori distribution; the pronunciation duration speculation module is used for aligning the prior distribution with the sampling data to obtain an alignment matrix, and obtaining pronunciation duration distribution through coding according to the depth features and the alignment matrix; and the decoder module is used for sampling the prior distribution into a multi-sub-band signal according to the pronunciation duration distribution and synthesizing the multi-sub-band signal into a voice waveform signal, thereby realizing direct mapping from the text to the voice.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to a speech synthesis system, method, medium and device based on a multi-subband generation strategy. Background Art

[0002] The field of speech synthesis is currently experiencing rapid development, but it still faces a series of critical issues. Among them, the issues of speech quality and generation speed are particularly prominent. Existing speech synthesis systems still have room for improvement in achieving naturalness, fluency and emotional expression. Synthesized speech often appears stiff and unnatural, and there are challenges such as stress problems, unreasonable pauses and abnormal intonation. The expression of emotion and tone also needs to be further improved to meet the higher requirements of speech synthesis quality in different application scenarios. Support for multiple languages ​​and dialects is also a key issue. Different languages ​​and dialects have different pronunciation rules and voice characteristics, which require more sophisticated modeling and adaptation. In addition, achieving personalized synthesis to adapt to the voice characteristics of specific users is the future development direction, which is crucial for personalized assistants and personalized speech synthesis.

[0003] In terms of generation speed, especially in applications that require real-time response, such as voice assistants and video calls, generation speed becomes a critical issue. Some existing speech synthesis technologies may require a lot of computing resources, resulting in a slow generation speed, which is unacceptable for real-time applications. In addition, in embedded systems, mobile devices, and edge computing environments, hardware resources are limited, and more lightweight speech synthesis models and algorithms are needed to meet the needs of these resource-constrained environments. Summary of the invention

[0004] Based on this, it is necessary to propose a speech synthesis system, method, medium and device based on a multi-subband generation strategy to address the above problems.

[0005] A speech synthesis system based on a multi-subband generation strategy, the system comprises a preprocessing module, a conversion layer module, a priori encoder module, a posteriori encoder module, a pronunciation duration estimation module and a decoder module,

[0006] The preprocessing module is used to convert the input text into a phoneme sequence, and convert the real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform.

[0007] The posterior encoder module is used to convert the linear spectrum into a posterior distribution, and obtain latent variables by sampling according to the mean and standard deviation in the posterior distribution.

[0008] The conversion layer module is used to convert the latent variables output by the posterior encoder module into sampled data;

[0009] The prior encoder module is used to extract the deep features of the phoneme sequence output by the preprocessing module and map the deep features into a prior distribution.

[0010] The pronunciation duration inference module is used to align the prior distribution with the sampling data through monotone alignment search, obtain an alignment matrix, sum the alignment matrix to obtain a duration sequence, conditionally encode the deep features and the duration sequence as conditional information, and convert the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization according to the conditional information.

[0011] The decoder module is used to sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the subband signal through linear transformation of the sampled data, generate the subband signal through short-time Fourier inverse transformation according to the amplitude and phase information, and synthesize multiple subband signals into a speech waveform signal through a synthesis filter.

[0012] Wherein, the system further comprises:

[0013] The discriminator module is used to form adversarial training with the decoder module, discriminate between a real speech waveform signal and the speech waveform signal, and optimize the speech waveform signal.

[0014] A speech synthesis method based on a multi-subband generation strategy, the method comprising:

[0015] The input text is converted into a phoneme sequence, and the real speech waveform corresponding to the input text is converted into a linear spectrum through short-time Fourier transform.

[0016] The linear spectrum is converted into a posterior distribution, and latent variables are obtained by sampling according to the mean and standard deviation in the posterior distribution.

[0017] The output latent variables are converted into sample data.

[0018] Extract deep features of the output phoneme sequence, and map the deep features into a prior distribution.

[0019] The prior distribution is aligned with the sampling data through monotone alignment search to obtain an alignment matrix, the alignment matrix is ​​summed to obtain a duration sequence, the deep features and the duration sequence are conditionally encoded as conditional information, and the Gaussian noise sequence is converted into a pronunciation duration distribution through variational dequantization according to the conditional information.

[0020] The prior distribution is sampled according to the pronunciation duration distribution, the sampled data is linearly transformed to generate amplitude and phase information of subband signals, subband signals are generated according to the amplitude and phase information through short-time Fourier inverse transform, and a plurality of subband signals are synthesized into a speech waveform signal through a synthesis filter.

[0021] The converting of the linear spectrum into a posterior distribution and sampling according to the mean and standard deviation in the posterior distribution to obtain a latent variable specifically includes:

[0022] The latent variables of the linear spectrum are extracted by a recurrent neural network.

[0023] The latent variables are output as the mean and standard deviation of the posterior distribution through a linear transformation performed by a fully connected layer.

[0024] Data points are sampled using a sampling function according to the mean and standard deviation of the posterior distribution to obtain latent variables.

[0025] Among them, the prior distribution is aligned with the sampled data through monotone alignment search to obtain an alignment matrix, the alignment matrix is ​​summed to obtain a duration sequence, the deep feature and the duration sequence are conditionally encoded as condition information, and the Gaussian noise sequence is converted into a pronunciation duration distribution through variational dequantization according to the condition information, specifically including:

[0026] The prior distribution is aligned with the sampled data through a monotone alignment search to obtain an alignment matrix.

[0027] Each row of the alignment matrix is ​​summed to obtain a duration sequence corresponding to each phoneme.

[0028] The depth feature and the duration sequence are conditionally encoded as conditional information.

[0029] The Gaussian noise sequence is converted into a first random variable and a second random variable according to the condition information.

[0030] The duration sequence is subtracted from the second random variable to determine a dequantized continuous variable.

[0031] The dequantized continuous variable and the first random variable are converted into pronunciation duration distribution.

[0032] The method further specifically includes:

[0033] A reconstruction loss function is defined according to a real speech waveform signal and a mel-spectrogram of the speech waveform signal.

[0034] A KL divergence loss function is defined according to the posterior distribution and the prior distribution.

[0035] A pronunciation duration inference loss function is defined according to the first random variable and the second random variable.

[0036] An adversarial loss function and a feature matching loss function are defined based on the identification of the real speech waveform signal and the speech waveform signal.

[0037] The total loss function is determined by the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the adversarial loss function and the feature matching loss function.

[0038] The total loss function is determined by using the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the adversarial loss function and the feature matching loss function, and specifically includes:

[0039] According to L total =L recon +L kl +L dur +L adv (DI)+L adv (DE)+L fm (DE) Determine the total loss function of speech synthesis, where L total is the total loss function of speech synthesis, L recon To reconstruct the loss function, x mel is the Mel spectrogram corresponding to the real speech waveform, It is the Mel-spectrogram corresponding to the speech waveform signal.

[0040] L kl is the KL divergence loss function, L kl =log q φ (z|x lin )-log p θ (z|c text ,A),q φ (z|x lin ) is the posterior distribution, x lin is the linear spectrum, z is the latent variable, p θ (z|c text ,A) is the prior distribution under the alignment matrix condition, A is the alignment matrix, c text is a phoneme sequence.

[0041] L dur is the pronunciation duration inference loss function, is the log-likelihood distribution, v is the first random variable, u is the second random variable, d is the duration sequence, is the expectation of the log-likelihood distribution, logp θ (du,v|c text ) is Gaussian noise distribution, is the expectation of the Gaussian noise distribution.

[0042] L adv (DI) and L adv (DE) is the adversarial loss function, L adv (DI)=E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ], L adv (DE)=E z [(DI(DE(z))-1) 2 ], DI(y) is the probability of identifying the real speech waveform as true, DI(DE(z)) is the probability of identifying the speech waveform signal as true, DE(z) is the speech waveform signal, E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] is the expected error caused by the discriminator’s error in distinguishing between the real speech waveform and the generated speech waveform signal, E z [(DI(DE(z))-1) 2 ] is the expected error between the probability that the discriminator determines that the generated speech waveform is the real speech waveform and 1.

[0043] L fm (DE) is the feature matching loss function, DI l (y) is the probability of identifying the feature generated by the real speech waveform after each layer of convolution as true, DI l (DE(z)) is the probability that the feature generated by the speech waveform signal after each layer of convolution is true, T is the number of speech samples, Nl is the number of convolution layers in the identification process, is the expectation of the weighted sum of the errors between the generated speech waveform and the real speech waveform produced by each convolution layer in the discriminator, and ||·||1 is the absolute value norm.

[0044] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method described above.

[0045] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0046] The embodiments of the present invention have the following beneficial effects:

[0047] The present invention first obtains the phoneme sequence and linear spectrum of the input text through a preprocessing module, and uses the inverse operation of the short-time Fourier transform to effectively convert the frequency domain to the time domain. Further, the potential variables and deep features are output by the prior encoder module and the posterior encoder module to obtain the relationship between the text and the speech waveform. Furthermore, the pronunciation duration distribution is obtained according to the deep features, and the pronunciation duration rhythm distribution of different texts is learned through the pronunciation duration distribution, which helps the speech synthesis system based on the multi-subband generation strategy to generate a more natural and high-quality speech waveform. Finally, in the decoder module, based on the multi-subband generation strategy, the prior distribution is sampled as a multi-subband signal according to the pronunciation duration distribution and synthesized into a speech waveform signal, which greatly improves the calculation efficiency and reasoning speed. The speech synthesis system based on the multi-subband generation strategy provided by the present invention adopts an end-to-end training method, directly learns from the text to the speech waveform, without explicitly learning the intermediate representation or features, and realizes the direct mapping of text to speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] in:

[0050] Figure 1 A flowchart of an embodiment of a speech synthesis system based on a multi-subband generation strategy provided by the present invention;

[0051] Figure 2 A schematic diagram of the structure of the a posteriori encoder module provided by the present invention;

[0052] Figure 3 A schematic diagram of the structure of the conversion layer module provided by the present invention;

[0053] Figure 4 A schematic diagram of the structure of a priori encoder module provided by the present invention;

[0054] Figure 5 A schematic diagram of the structure of a pronunciation duration estimation module provided by the present invention;

[0055] Figure 6 A schematic diagram of the structure of a decoder module provided by the present invention;

[0056] Figure 7A flowchart of an embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present invention;

[0057] Figure 8 A flowchart of another embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present invention;

[0058] Fig. 9 A flowchart of another embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present invention;

[0059] Fig.10 A schematic structural diagram of an embodiment of the device provided by the present invention;

[0060] Fig.11 A schematic structural diagram of an embodiment of the medium provided by the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] like Figure 1 As shown, Figure 1 A schematic diagram of the structure of an embodiment of a speech synthesis system based on a multi-subband generation strategy provided by the present invention. Figure 1 A speech synthesis system 10 based on a multi-subband generation strategy includes a preprocessing module 11, a posterior encoder module 12, a conversion layer module 13, a priori encoder module 14, a pronunciation duration estimation module 15 and a decoder module 16.

[0063] The preprocessing module 11 is used to convert the input text into a phoneme sequence, and convert the real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform.

[0064] Exemplarily, the preprocessing module 11 converts the input text into a phoneme sequence, i.e., a Chinese syllable sequence, performs word segmentation on the input text, and divides the text into words or phrases. Pinyin is annotated on the text after word segmentation to generate a pinyin sequence. According to the pinyin sequence, each pinyin is converted into a corresponding phoneme sequence, i.e., a Chinese syllable sequence. Furthermore, the Chinese syllable sequence is optimized, including adjusting the connection relationship between syllables and stress marks.

[0065] In addition, the speech waveform corresponding to the input text is preprocessed, including filtering, framing and windowing. The preprocessed speech waveform is short-time Fourier transformed to obtain a time-frequency spectrum. The time-frequency spectrum is frequency mapped to map the frequency axis from a linear scale to a Mel scale. The energy in each Mel frequency band is calculated to obtain a Mel spectrum. The Mel spectrum is logarithmically compressed to better represent the dynamic range of the signal, and finally a linear spectrum is obtained.

[0066] The posterior encoder module 12 is used to convert the linear spectrum into a posterior distribution, and obtain latent variables by sampling according to the mean and standard deviation in the posterior distribution.

[0067] Exemplarily, in the posterior encoder module 12, the mean and standard deviation of the posterior distribution in the linear spectrum are obtained according to the recursive neural network and the fully connected layer, and data points are sampled by the mean and standard deviation of the posterior distribution to obtain the latent variable z, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of the posterior encoder module provided by the present invention. The linear spectrum is passed through the recursive neural network WaveNet to obtain the hidden variable after feature extraction, and the segmentation layer outputs the hidden variable of the linear spectrum as the mean and standard deviation of the posterior distribution through a fully connected layer. Subsequently, data points are sampled through a sampling function according to the mean and standard deviation of the obtained posterior distribution, and the sampled data is the latent variable z.

[0068] The conversion layer module 13 is used to convert the potential variables output by the posterior encoder module into sampled data.

[0069] Exemplarily, in the conversion layer module 13, the sampling data f of the latent variable on the prior distribution is obtained through the residual coupling layer and the flip layer. θ (z), such as Figure 3 As shown, Figure 3 Schematic diagram of the structure of the conversion layer module provided by the present invention. The latent variable z is used as input, where the latitude of z is [B, 192, T]. The latent variable z is divided into the first segmentation variable x0 and the second segmentation variable x1, and then the mean μ of the latent variable z on the prior distribution is obtained through the recurrent neural network waveNet and the one-dimensional convolutional layer conv1d. p According to the mean μ of the latent variable z p The second splitting variable x1 is used to determine the second splitting variable on the prior distribution. The second splitting variable on the prior distribution is shown as follows:

[0070]

[0071] in, is the second splitting variable on the prior distribution, The dimension of is [B, 96, T], x0 is the first segmentation variable, x1 is the second segmentation variable, the dimension of x1 is [B, 96, T], μ p is the mean of the latent variable on the prior distribution, and σ is the standard deviation of the latent variable on the prior distribution.

[0072] Furthermore, the first segmentation variable x0[B, 96, T] is connected to the second segmentation variable on the prior distribution through the connection layer. After the connection, the positions of the first segmentation variable x0 and the second segmentation variable on the prior distribution are reversed through the flip layer. The first segmentation variable on the prior distribution is obtained again through the residual coupling layer. The first segmentation variable on the prior distribution and the second segmentation variable on the prior distribution are connected to obtain z p [B, 192, T], after 4 residual coupling layers and flipping, the sampled data f of the output latent variable on the prior distribution is obtained θ (z).

[0073] The prior encoder module 14 is used to extract the deep features of the phoneme sequence output by the preprocessing module and map the deep features into a prior distribution.

[0074] Exemplarily, while obtaining the mean and standard deviation of the posterior distribution in the linear spectrum according to the recursive neural network and the fully connected layer, sampling data points by the mean and standard deviation of the posterior distribution to obtain the latent variables, the prior encoder module 14 determines the prior distribution of the speed sequence according to the deep features of the phoneme sequence, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of the a priori encoder module provided by the present invention. The phoneme sequence is converted into a 192-dimensional feature vector by the embedding layer and embedded into the original sequence to obtain a sequence [B, T, 192]. The transposition layer performs dimensional conversion on the sequence to obtain a conversion sequence [B, 192, T]. The attention layer is then used to perform deep information extraction on the conversion sequence to obtain the deep feature h corresponding to each sample. text Then, the depth feature h text Through a mapping layer and segmentation operation, the mean μ of the prior distribution of the phoneme sequence is obtained θ and standard deviation σ θ , according to the mean μ of the prior distribution of the phoneme sequence θ and standard deviation σ θ Determine the prior distribution.

[0075] The pronunciation duration inference module 15 is used to align the prior distribution with the sampled data through monotone alignment search, obtain the alignment matrix, sum the alignment matrix to obtain the duration sequence, conditionally encode the deep features and the duration sequence as conditional information, and convert the Gaussian noise sequence into the pronunciation duration distribution through variational dequantization according to the conditional information.

[0076] For example, the sample data f is first aligned using a monotone alignment search method. θ (z) Align with the prior distribution of the phoneme sequence to achieve the correspondence between the text and the speech waveform signal. According to the monotonicity constraint idea, that is, the pronunciation of the synthesized speech is consistent with the order of the text, the phonemes and the speech waveform are monotonically aligned. One phoneme can correspond to multiple sampling points of the speech waveform, but one speech waveform sampling point can only correspond to one phoneme. Use the dynamic programming algorithm to solve the maximum likelihood value of each corresponding strategy. The corresponding strategy with the largest maximum likelihood value is regarded as the sampling data f θ (z) corresponds to the prior distribution of the phoneme sequence. This correspondence sequentially converts each sample data f θ (z) corresponds to the prior distribution of the sequence and generates an alignment matrix.

[0077] Furthermore, the duration sequence d corresponding to each phoneme is obtained by summing each row of the alignment matrix, and the duration sequence d and the depth feature h corresponding to the text are combined. text as input.

[0078] It should be noted that the present invention uses the depth feature h corresponding to the text text When inputting, a gradient stop controller is used to prevent the update of the gradient so as not to affect the gradient calculation in the training process of the prior encoder. And because the input duration sequence d is a discrete integer sequence, the application of maximum likelihood estimation is difficult. The present invention adopts variational dequantization and variational data enhancement methods.

[0079] Further, in the pronunciation duration estimation module 15, the alignment matrix is ​​converted into a pronunciation duration distribution by summing and encoding, such as Figure 5 As shown, Figure 5 The structural diagram of the pronunciation duration estimation module provided by the present invention. The duration sequence d and the depth feature h corresponding to the text text As input, it is first encoded by the conditional encoder and then input into the variational encoder as conditional information; the variational encoder converts a Gaussian noise sequence into a first random variable v and a second random variable u according to the conditional information, and uses the first random variable v and the second random variable u to represent the approximate distribution; the duration sequence d is subtracted from the second random variable u to determine the dequantized continuous variable du; then, the dequantized continuous variable du and the first random variable v are converted into the pronunciation duration distribution Noise~N(0,I) to represent the dequantized and enhanced log-likelihood distribution. Among them, the pronunciation duration distribution Noise~N(0,I) can be the pronunciation duration corresponding to the sampled text in the distribution sequence.

[0080] It should be noted that the first random variable v is a noise variable for data enhancement, and the second random variable u is a background noise variable of duration, ranging from [0, 1).

[0081] The decoder module 16 is used to sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the sub-band signal through linear transformation of the sampled data, generate the sub-band signal through short-time Fourier inverse transformation according to the amplitude and phase information, and synthesize multiple sub-band signals into a speech waveform signal through a synthesis filter.

[0082] Exemplarily, the decoder module 16 is first trained, the amplitude and phase information of the subband signal in the latent variable z is extracted in the decoder module 16, the inverse operation of the short-time Fourier transform is performed on the amplitude and phase information of the subband signal to generate the subband signal, and the multiple subband signals are synthesized into a speech waveform signal through a synthesis filter. like Figure 6 As shown, Figure 6 The schematic diagram of the decoder module provided by the present invention. The latent variable z is upsampled by two residual convolution blocks with parameter s, where parameter s is [4, 4] and is the parameter of the upsampling scale. Subsequently, a linear transformation is performed through the fully connected layer to generate the amplitude information and phase information of the four sub-band signals.

[0083] Furthermore, the amplitude information and phase information of each sub-band signal are subjected to an iSTFT (Inverse Short-Time Fourier Transform) operation to generate a sub-band signal. The number of fast Fourier transform sampling points, the jump length of the iSTFT, and the window length are 16, 4, and 16, respectively. The 4 sub-band signals are up-sampled to match the sampling rate of the original signal by adding 0 between samples. Subsequently, the trainable synthesis filter is used to synthesize the above 4 sub-band signals into a speech waveform signal. The trainable synthesis filter includes a pseudo-orthogonal mirror filter bank.

[0084] Furthermore, the trained decoder module 16 samples the prior distribution according to the pronunciation duration distribution Noise~N(0,I), generates the amplitude and phase information of the subband signal through linear transformation, generates the subband signal through short-time inverse Fourier transform according to the amplitude and phase information, and synthesizes the multiple subband signals into a speech waveform signal through a synthesis filter. Specifically, according to the pronunciation duration distribution Noise~N(0,I) and the prior distribution of deep features h textThe continuous pronunciation duration of the corresponding sample is obtained, and the continuous pronunciation duration is rounded up to obtain the duration sequence d. According to the duration sequence d, the prior distribution generated by the prior encoder 13 is sampled to obtain the sampling data f of the potential variable z on the prior distribution. θ (z) (i.e., the sampled data). The sampled data is transformed linearly to generate the amplitude and phase information of the sub-band signal. The sub-band signal is generated by short-time inverse Fourier transform based on the amplitude and phase information. Multiple sub-band signals are synthesized into a speech waveform signal through a synthesis filter. And use the speaker to make sounds.

[0085] Furthermore, a speech synthesis system based on a multi-subband generation strategy also includes a discriminator module, which is used to form adversarial training with the decoder module 16, discriminate between the real speech waveform signal and the speech waveform signal, and optimize the speech waveform signal. The discriminator module is used to form adversarial training with the decoder module 16, discriminate between the real speech waveform signal and the speech waveform signal, and optimize the speech waveform signal, so that the speech waveform signal output by the decoder module 16 is closer to the real speech waveform.

[0086] It can be seen from the above description that the present invention first obtains the phoneme sequence and linear spectrum of the input text through the preprocessing module, and uses the inverse operation of the short-time Fourier transform to effectively convert the frequency domain to the time domain. Further, the potential variables and deep features are output by the prior encoder module and the posterior encoder module to obtain the relationship between the text and the speech waveform. Further, the pronunciation duration distribution is obtained according to the deep features, and the pronunciation duration rhythm distribution of different texts is learned through the pronunciation duration distribution, which helps the speech synthesis system based on the multi-subband generation strategy to generate a more natural and high-quality speech waveform. Finally, in the decoder module, based on the multi-subband generation strategy, the prior distribution is sampled as a multi-subband signal according to the pronunciation duration distribution and synthesized into a speech waveform signal, which greatly improves the calculation efficiency and reasoning speed. The speech synthesis system based on the multi-subband generation strategy provided by the present invention adopts an end-to-end training method, directly learns from the text to the speech waveform, without explicitly learning the intermediate representation or features, and realizes the direct mapping of text to speech.

[0087] Figure 7 A flowchart of an embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present invention. Figure 7 , a speech synthesis method based on a multi-subband generation strategy, the method comprising:

[0088] S101: Convert the text into a phoneme sequence, and convert the real speech waveform corresponding to the text into a linear spectrum.

[0089] Exemplarily, the input text is converted into a phoneme sequence, i.e., a Chinese syllable sequence, and the input text is segmented to divide the text into words or phrases. The segmented text is annotated with pinyin to generate a pinyin sequence. According to the pinyin sequence, each pinyin is converted into a corresponding phoneme sequence, i.e., a Chinese syllable sequence. In addition, the input speech waveform is preprocessed with the real speech waveform corresponding to the input text, and the preprocessed speech waveform is short-time Fourier transformed to obtain a time-frequency spectrum. The time-frequency spectrum is frequency mapped, and the energy in each Mel frequency band is calculated to obtain a Mel spectrum. The Mel spectrum is logarithmically compressed to finally obtain a linear spectrum.

[0090] S102: Convert the linear spectrum into a posterior distribution, and obtain latent variables by sampling according to the mean and standard deviation in the posterior distribution.

[0091] Exemplarily, the linear spectrum is input into a recursive neural network WaveNet, which includes multiple consecutive convolutional layers. After each convolutional layer, the linear spectrum is added to the output of the convolutional layer to obtain a residual signal. The residual signal is normalized, usually using batch normalization. An activation function, such as ReLU (Rectified Linear Unit), is applied to the normalized residual signal, and the above steps are repeated until the output of the residual block, and the output of the residual block is the extracted linear spectrum latent variable. The linear spectrum latent variable is input into a fully connected layer, which includes multiple consecutive linear layers. The fully connected layer passes through a linear layer to obtain the mean of the posterior distribution; and passes through another linear layer to obtain the standard deviation of the posterior distribution. Data points are sampled through a sampling function according to the mean and standard deviation of the obtained posterior distribution, and the sampled data is recorded as a latent variable z.

[0092] S103: Convert the output potential variables into sampled data.

[0093] Exemplarily, in the conversion layer module 13, the sampling data f of the potential variable z on the prior distribution is obtained through the residual coupling layer and the flip layer. θ (z). First, the latent variable z is taken as input, and the latent variable z is divided into the first segmentation variable x0 and the second segmentation variable x1. Then, the mean μ of the latent variable z on the prior distribution is obtained through the recurrent neural network waveNet and the one-dimensional convolutional layer conv1d. p According to the mean μ of the latent variable z p and the second splitting variable x1 determine the second splitting variable on the prior distribution.

[0094] Furthermore, the first segmentation variable x0 is connected to the second segmentation variable on the prior distribution through the connection layer. After the connection, the positions of the first segmentation variable x0 and the second segmentation variable on the prior distribution are reversed through the flip layer. The first segmentation variable on the prior distribution is obtained again through the residual coupling layer. The first segmentation variable on the prior distribution is connected to the second segmentation variable on the prior distribution. After 4 residual coupling layers and flipping, the sampling data f of the output potential variable on the prior distribution is obtained. θ (z).

[0095] S104: Extract deep features of the output phoneme sequence, and map the deep features into a prior distribution.

[0096] For example, the phoneme sequence is converted into a feature vector through the embedding layer, and the feature vector is embedded into the phoneme sequence to obtain an embedded sequence. The embedded sequence is transformed in latitude through the transposition layer to obtain a transformed sequence. The information of the transformed sequence is extracted through the attention layer to obtain the deep feature h of the phoneme sequence. text Through the mapping layer, the deep feature h text Segmentation is performed to obtain the prior distribution of phoneme sequences.

[0097] S105: Align the prior distribution with the sampled data through monotone alignment search to obtain the alignment matrix, sum the alignment matrix to obtain the duration sequence, conditionally encode the deep features and the duration sequence as conditional information, and convert the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization according to the conditional information.

[0098] Exemplarily, the prior distribution is aligned with the sampled data f by monotone alignment search. θ (z) Align and obtain the alignment matrix; sum each row of the alignment matrix to obtain the duration sequence d corresponding to each phoneme; text Conditionally encode the duration sequence d as conditional information; convert the Gaussian noise sequence into a first random variable v and a second random variable u according to the conditional information; subtract the duration sequence d from the second random variable u to determine the dequantized continuous variable du; convert the dequantized continuous variable du and the first random variable v into a pronunciation duration distribution Noise~N(0,I).

[0099] S106: Sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the subband signal through linear transformation of the sampled data, generate the subband signal through short-time Fourier inverse transform according to the amplitude and phase information, and synthesize the multiple subband signals into a speech waveform signal through a synthesis filter.

[0100] For example, according to the prior distribution of depth features h textand pronunciation duration distribution Noise~N(0,I) to obtain continuous pronunciation duration d; round up the continuous pronunciation duration d to obtain a duration sequence; sample the prior distribution according to the duration sequence, generate the amplitude and phase information of the sub-band signal through linear transformation of the sampled data, generate the sub-band signal through short-time inverse Fourier transform according to the amplitude and phase information, and synthesize multiple sub-band signals into a speech waveform signal through a synthesis filter And use the speaker to make sounds.

[0101] like Figure 8 As shown, Figure 8 A flowchart of another embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present invention. Figure 8 , a speech synthesis method based on a multi-subband generation strategy, the method comprising:

[0102] S201: Define a reconstruction loss function according to the real speech waveform and the mel-spectrogram of the speech waveform signal.

[0103] Exemplarily, to evaluate the decoder module 16, the latent variable z is converted into a speech waveform signal The effect is obtained by updating the parameters in the decoder module 16 during the training phase, so the reconstruction loss function is defined as follows:

[0104]

[0105] The present invention uses Mel spectrogram instead of speech waveform to calculate the reconstruction loss function. recon is the reconstruction loss function, x mel is the Mel spectrogram corresponding to the real speech waveform, It is the Mel-spectrogram corresponding to the speech waveform signal.

[0106] S202: Define a KL divergence loss function based on the posterior distribution and the prior distribution.

[0107] For example, in order to allow the prior encoder module 14 to generate the sampled data f obtained by sampling the prior distribution using the phoneme sequence as input θ (z) is closer to the potential variable z obtained by sampling the posterior distribution generated by the posterior encoder module 12 using the linear spectrum corresponding to the speech waveform as input, so that the sampled data f output by the prior encoder module 14 is θ (z) is closer to the real distribution of speech waveform, thereby improving the quality of text-to-speech waveform generation. Therefore, the KL divergence (Kullback-Leibler Divergence, relative entropy) loss function is defined as follows:

[0108] L kl =log qφ (z|x lin )-log p θ (z|c text ,A) (3)

[0109] Among them, L kl is the KL divergence loss function, q φ (z|x lin ) is the posterior distribution, x lin is the linear spectrum, z is the latent variable, p θ (z|c text ,A) is the prior distribution under the alignment matrix condition, A is the alignment matrix, c text is a phoneme sequence.

[0110] S203: Define a pronunciation duration inference loss function according to the first random variable and the second random variable.

[0111] For example, in order to detect the correctness of the duration of the speech waveform corresponding to the text and adjust the parameters in the process of reconstruction loss to improve the prediction accuracy, the pronunciation duration estimation loss function L is defined as dur :

[0112]

[0113] Where v is the first random variable, u is the second random variable, and d is the duration sequence. is an approximate distribution, logp θ (du,v|c text ) is the log-likelihood distribution, is the logarithmic expectation of the likelihood distribution, is the expectation of the log-likelihood distribution.

[0114] S204: defining an adversarial loss function and a feature matching loss function based on the identification of the real speech waveform signal and the speech waveform signal.

[0115] Exemplarily, a speech synthesis system based on a multi-subband generation strategy further includes a discriminator module for performing adversarial training with the decoder module 16 to perform adversarial training on a real speech waveform signal and a speech waveform signal. The discriminator module and the decoder module 16 are used to form adversarial training so that the speech waveform signal output by the decoder module 16 is Closer to the real speech waveform. Compare the real speech waveform with the speech waveform signal output by the decoder module 16 The discriminator module determines whether the input waveform is a real waveform. The decoder module 16 adjusts the parameters during the training process according to the feedback from the discriminator module until the discriminator module cannot determine whether the input waveform is a real speech waveform or a speech waveform signal generated by the decoder module 16.

[0116] In order to detect the speech waveform signal generated by the decoder module 16 The effect of the adversarial loss function L is defined by adjusting the parameters during the training of the discriminator module and the decoder module 16. adv (DI) and L adv (DE) is shown in the following formula:

[0117] L adv (DI)=E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] (5)

[0118] L adv (DE)=E z [(DI(DE(z))-1) 2 ] (6)

[0119] Among them, DI(y) is the probability of identifying the real speech waveform as true, DI(DE(z)) is the probability of identifying the speech waveform signal as true, DE(z) is the speech waveform signal, E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] is the expected error caused by the discriminator’s error in distinguishing between the real speech waveform and the generated speech waveform signal, E z [(DI(DE(z))-1) 2 ] is the expected error between the probability that the discriminator determines that the generated speech waveform is the real speech waveform and 1.

[0120] Furthermore, the feature matching loss function is defined based on the identification of the features generated by the speech waveform signal after each layer of convolution and the identification of the features generated by the real speech waveform after each layer of convolution, as shown in the following formula:

[0121]

[0122] Among them, DI l (y) is the probability of identifying the feature generated by the real speech waveform after each layer of convolution as true, DI l(DE(z)) is the probability that the feature generated by the speech waveform signal after each layer of convolution is true, T is the number of speech samples, Nl is the number of convolution layers in the identification process, is the expectation of the weighted sum of the errors between the generated speech waveform and the real speech waveform generated by each convolution layer in the discriminator, and ||·||1 is the absolute value norm, which is used to measure DI l (y) and DI l The sum of the absolute value differences of the elements between (DE(z)).

[0123] S205: Determine the total loss function by using the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the adversarial loss function and the feature matching loss function.

[0124] Exemplarily, the total loss function for speech synthesis is determined according to the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the defined adversarial loss function and the feature matching loss function, as shown in the following formula:

[0125] L total =L recon +L kl +L dur +L adv (DI)+L adv (DE)+L fm (DE) (8)

[0126] Among them, L total is the total loss function for speech synthesis.

[0127] Since the discriminator module is used to improve the quality of the speech waveform generated by the decoder module 16, it does not directly participate in the text-to-speech generation process. Therefore, the total speech synthesis loss function used in the text-to-speech generation part is summarized as formula (9):

[0128] L Gen =L recon +L kl +L dur +L adv (DE)+L fm (DE) (9)

[0129] Among them, L Gen is the final total loss function for speech synthesis.

[0130] Furthermore, the parameters of the preprocessing module 11, the posterior encoder module 12, the conversion layer module 13, the a priori encoder module 14, the pronunciation duration estimation module 15, the decoder module 16 and the discriminator module are updated according to the total loss function, and the iterations are repeated to obtain the trained preprocessing module 11, the posterior encoder module 12, the conversion layer module 13, the a priori encoder module 14, the pronunciation duration estimation module 15, the decoder module 16 and the discriminator module.

[0131] like Fig. 9 As shown, Fig. 9 A flowchart of another embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present invention. Fig. 9 , a speech synthesis method based on a multi-subband generation strategy, the method comprising:

[0132] S301: Convert the input text into a phoneme sequence, and convert the real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform.

[0133] It should be noted that step S301 is already in Figure 1 The implementation scenario shown is discussed in detail and will not be repeated here.

[0134] S302: Extracting latent variables of the linear spectrum through a recursive neural network.

[0135] Exemplarily, the linear spectrum is input into a recurrent neural network, which includes multiple consecutive convolutional layers. After each convolutional layer, the linear spectrum is added to the output of the convolutional layer to obtain a residual signal. The residual signal is normalized, usually using batch normalization. An activation function, such as ReLU, is applied to the normalized residual signal, and the above steps are repeated until the output of the residual block, which is the latent variable of the extracted linear spectrum.

[0136] S303: Perform linear transformation through the fully connected layer to output the latent variable as the mean and standard deviation of the posterior distribution.

[0137] S304: Sampling data points using a sampling function according to the mean and standard deviation of the posterior distribution to obtain latent variables.

[0138] Exemplarily, the latent variable of the linear spectrum is input into a fully connected layer, which includes multiple consecutive linear layers. The fully connected layer passes through a linear layer to obtain the mean of the posterior distribution; passes through another linear layer to obtain the standard deviation of the posterior distribution. According to the mean and standard deviation of the obtained posterior distribution, data points are sampled through a sampling function, and the sampled data is recorded as the latent variable z.

[0139] S305: Extract deep features of the output phoneme sequence, and map the deep features into a priori distribution.

[0140] It should be noted that steps S302-S304 and step S305 are performed simultaneously. Figure 1 The implementation scenario shown is discussed in detail and will not be repeated here.

[0141] S306: Convert the output potential variables into sampled data.

[0142] For example, the latent variable z is first taken as input, and the latent variable z is divided into a first segmentation variable x0 and a second segmentation variable x1, and then the mean μ of the latent variable z on the prior distribution is obtained through a recurrent neural network and a one-dimensional convolutional layer. p According to the mean μ of the latent variable p and the second splitting variable x1 determine the second splitting variable on the prior distribution.

[0143] Furthermore, the first segmentation variable x0 is connected to the second segmentation variable on the prior distribution through the connection layer. After the connection, the positions of the first segmentation variable x0 and the second segmentation variable on the prior distribution are reversed through the flip layer. The first segmentation variable on the prior distribution is obtained again through the residual coupling layer. The first segmentation variable on the prior distribution is connected to the second segmentation variable on the prior distribution. After 4 residual coupling layers and flipping, the sampling data f of the output potential variable on the prior distribution is obtained. θ (z).

[0144] S307: Align the prior distribution with the sampled data through a monotone alignment search to obtain an alignment matrix.

[0145] In one implementation scenario, the sampled data f is searched using a monotone alignment search method. θ (z) Align with the prior distribution of the phoneme sequence to achieve the correspondence between the text and the speech waveform signal. According to the monotonicity constraint idea, that is, the pronunciation of the synthesized speech is consistent with the order of the text, the phonemes and the speech waveform are monotonically aligned. One phoneme can correspond to multiple sampling points of the speech waveform, but one speech waveform sampling point can only correspond to one phoneme. Use the dynamic programming algorithm to solve the maximum likelihood value of each corresponding strategy. The corresponding strategy with the largest maximum likelihood value is regarded as the sampling data f θ (z) corresponds to the prior distribution of the phoneme sequence. This correspondence sequentially converts each sample data f θ (z) corresponds to the prior distribution of the sequence and generates an alignment matrix.

[0146] Specifically, we first construct a dynamic programming table whose size is the input text sequence length × target speech length. Each table entry represents a possible alignment.

[0147] Furthermore, the first row of the table is initialized to indicate the starting position where the first input text unit can be aligned.

[0148] Further, traverse backward from the first row of the table and fill in the values ​​in the table by repeating the following steps:

[0149] For each unfilled table entry (i, j);

[0150] Calculate its likelihood value score(i, j);

[0151] Select the maximum value table entry of the previous layer to fill the unfilled table entry (i, j), or keep it empty;

[0152] The maximum value position of the unfilled table entry (i, j) is recorded, and the maximum value is the maximum likelihood value.

[0153] Furthermore, after the traversal is completed, the position path corresponding to the maximum value in the last row of the record table is traced back to obtain the optimal alignment relationship.

[0154] Furthermore, an optimal alignment relationship is output, where the optimal alignment relationship is the corresponding strategy with the largest maximum likelihood value.

[0155] Furthermore, an alignment matrix is ​​generated according to the optimal alignment relationship.

[0156] Specifically, we first initialize an empty alignment matrix with a size of input text sequence length × target audio length.

[0157] The optimal correspondence obtained by traversal:

[0158] For each (text unit i, audio position j) pair:

[0159] Assign the value of 1 to the element in the i-th row and j-th column of the alignment matrix, keep the other elements as 0, and output the constructed alignment matrix.

[0160] For example: If the best correspondence record is:

[0161] Input text 1 corresponds to audio clips 1-3

[0162] Input text 2 corresponds to audio clips 4-6

[0163] Input text 3 corresponds to audio clips 7-9

[0164] The constructed alignment matrix is: [[1 1 1 0 0 0 0 0 0] [0 0 0 1 1 1 0 0 0] [0 0 0 0 0 0 1 1 1]

[0168] Each row represents an input text unit, and each column represents an audio time step. The non-zero element position indicates that the input text unit generates audio at that time step.

[0169] S308: Sum each row of the alignment matrix to obtain a duration sequence corresponding to each phoneme.

[0170] S309: Conditionally encode the depth feature and the duration sequence as conditional information.

[0171] Exemplarily, each row of the alignment matrix is ​​summed to obtain a duration sequence d corresponding to each phoneme. Exemplarily, referring to Figure 5 , duration sequence d and the corresponding deep feature h of the text text As input, it is first encoded by the conditional encoder and then input into the variational encoder as conditional information.

[0172] S310: Converting the Gaussian noise sequence into a first random variable and a second random variable according to the condition information.

[0173] Exemplarily, the conditional encoder is based on the deep feature h text and the conditional information of the duration sequence d, transform the Gaussian noise sequence into a first random variable v and a second random variable u, and use the first random variable v and the second random variable u to approximate the distribution.

[0174] S311: Subtract the duration sequence from the second random variable to determine a dequantized continuous variable.

[0175] S312: Convert the dequantized continuous variable and the first random variable into pronunciation duration distribution.

[0176] Exemplarily, the duration sequence d is subtracted from the second random variable u to obtain a dequantized continuous variable du; then, the dequantized continuous variable du and the first random variable v are converted into pronunciation duration distributions to represent the dequantized and enhanced log-likelihood distribution.

[0177] S313: Sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the subband signal through linear transformation of the sampled data, generate the subband signal through short-time Fourier inverse transform according to the amplitude and phase information, and synthesize multiple subband signals into a speech waveform signal through a synthesis filter.

[0178] It should be noted that step S313 is already in Figure 7 The implementation scenario shown is discussed in detail and will not be repeated here.

[0179] like Fig.10 As shown, Fig.10The device 20 includes a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 executes the computer program when working to implement the following. Figure 7 , Figure 8 and Fig. 9 The method shown.

[0180] The specific technical details of a speech synthesis method based on a multi-subband generation strategy implemented when the above-mentioned device 20 executes a computer program have been discussed in detail in the above-mentioned method steps, so they will not be repeated here.

[0181] like Fig.11 As shown, Fig.11 30 stores at least one computer program 31, which is executed by the processor 22 to implement the following. Figure 7 , Figure 8 and Fig. 9 In one embodiment, the medium 30 may be a storage chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server, etc.

[0182] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0183] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0184] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0185] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0186] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A speech synthesis system based on a multi-subband generation strategy, characterized in that: The system includes a preprocessing module, a conversion layer module, a priori encoder module, a posteriori encoder module, a pronunciation duration estimation module and a decoder module. The preprocessing module is used to convert the input text into a phoneme sequence, and convert the real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform; The posterior encoder module is used to convert the linear spectrum into a posterior distribution, and obtain latent variables by sampling according to the mean and standard deviation in the posterior distribution; The conversion layer module is used to convert the latent variables output by the posterior encoder module into sampled data; The prior encoder module is used to extract the deep features of the phoneme sequence output by the preprocessing module and map the deep features into a prior distribution; The pronunciation duration inference module is used to align the prior distribution with the sampled data through monotone alignment search, obtain an alignment matrix, sum the alignment matrix to obtain a duration sequence, conditionally encode the deep features and the duration sequence as condition information, and convert the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization according to the condition information; The decoder module is used to sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the subband signal through linear transformation of the sampled data, generate the subband signal through short-time Fourier inverse transformation according to the amplitude and phase information, and synthesize multiple subband signals into a speech waveform signal through a synthesis filter.

2. The speech synthesis system based on the multi-subband generation strategy according to claim 1, characterized in that: The system further comprises: The discriminator module is used to form adversarial training with the decoder module, discriminate between a real speech waveform signal and the speech waveform signal, and optimize the speech waveform signal.

3. A speech synthesis method based on a multi-subband generation strategy, characterized in that: The method comprises: The input text is converted into a phoneme sequence, and the real speech waveform corresponding to the input text is converted into a linear spectrum through short-time Fourier transform; The linear spectrum is converted into a posterior distribution, and a latent variable is obtained by sampling according to a mean and a standard deviation in the posterior distribution; Convert the output latent variables into sample data; Extracting deep features of the output phoneme sequence, and mapping the deep features into a prior distribution; Aligning the prior distribution with the sampled data through monotone alignment search to obtain an alignment matrix, summing the alignment matrix to obtain a duration sequence, conditionally encoding the deep features and the duration sequence as conditional information, and converting the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization according to the conditional information; The prior distribution is sampled according to the pronunciation duration distribution, the sampled data is linearly transformed to generate amplitude and phase information of subband signals, subband signals are generated according to the amplitude and phase information through short-time Fourier inverse transform, and a plurality of subband signals are synthesized into a speech waveform signal through a synthesis filter.

4. The speech synthesis method based on the multi-subband generation strategy according to claim 3, characterized in that: The converting the linear spectrum into a posterior distribution and sampling according to the mean and standard deviation in the posterior distribution to obtain a latent variable specifically includes: Extracting latent variables of the linear spectrum through a recurrent neural network; Performing linear transformation through a fully connected layer to output the latent variable as the mean and standard deviation of the posterior distribution; Data points are sampled using a sampling function according to the mean and standard deviation of the posterior distribution to obtain potential variables.

5. The speech synthesis method based on the multi-subband generation strategy according to claim 4, characterized in that: The prior distribution is aligned with the sampled data through monotone alignment search, an alignment matrix is ​​obtained, the alignment matrix is ​​summed to obtain a duration sequence, the deep features and the duration sequence are conditionally encoded as condition information, and the Gaussian noise sequence is converted into a pronunciation duration distribution through variational dequantization according to the condition information, specifically including: Aligning the prior distribution with the sampled data through a monotone alignment search to obtain an alignment matrix; Summing each row of the alignment matrix to obtain a duration sequence corresponding to each phoneme; Conditionally encoding the depth feature and the duration sequence as conditional information; Converting the Gaussian noise sequence into a first random variable and a second random variable according to the condition information; The duration sequence is subtracted from a second random variable to determine a dequantized continuous variable; The dequantized continuous variable and the first random variable are converted into pronunciation duration distribution.

6. The speech synthesis method based on the multi-subband generation strategy according to claim 5, characterized in that: The method further specifically comprises: Defining a reconstruction loss function according to a real speech waveform signal and a mel-spectrogram of the speech waveform signal; Define a KL divergence loss function according to the posterior distribution and the prior distribution; Defining a pronunciation duration inference loss function according to the first random variable and the second random variable; Defining an adversarial loss function and a feature matching loss function according to the identification of the real speech waveform signal and the speech waveform signal; The total loss function is determined by the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the adversarial loss function and the feature matching loss function.

7. The speech synthesis method based on the multi-subband generation strategy according to claim 6, characterized in that: The total loss function is determined by using the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the adversarial loss function and the feature matching loss function, specifically including: According to L total =L recon +L kl +L dur +L adv (DI)+L adv (DE)+L fm (DE) Determine the total loss function of speech synthesis, where L total is the total loss function of speech synthesis, L recon To reconstruct the loss function, x mel is the Mel spectrogram corresponding to the real speech waveform, It is the Mel spectrogram corresponding to the speech waveform signal; L kl is the KL divergence loss function, L kl =logq φ (z|x lin )-logp θ (z|c text ,A),q φ (z|x lin ) is the posterior distribution, x lin is the linear spectrum, z is the latent variable, p θ (z|c text ,A) is the prior distribution under the alignment matrix condition, A is the alignment matrix, c text is a phoneme sequence; L dur is the pronunciation duration inference loss function, is the log-likelihood distribution, v is the first random variable, u is the second random variable, d is the duration sequence, is the expectation of the log-likelihood distribution, logp θ (du,v|c text ) is Gaussian noise distribution, is the expectation of the Gaussian noise distribution; L adv (DI) and L adv (DE) is the adversarial loss function, L adv (DI)=E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ], L adv (DE)=E z [(DI(DE(z))-1) 2 ], DI(y) is the probability of identifying the real speech waveform as true, DI(DE(z)) is the probability of identifying the speech waveform signal as true, DE(z) is the speech waveform signal, E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] is the expected error caused by the discriminator’s error in distinguishing between the real speech waveform and the generated speech waveform signal, E z [(DI(DE(z))-1) 2 ] is the expected error between the probability that the discriminator determines that the generated speech waveform is the real speech waveform and 1; L fm (DE) is the feature matching loss function, DI l (y) is the probability of identifying the feature generated by the real speech waveform after each layer of convolution as true, DI l (DE(z)) is the probability that the feature generated by the speech waveform signal after each layer of convolution is true, T is the number of speech samples, Nl is the number of convolution layers in the identification process, is the expectation of the weighted sum of the errors between the generated speech waveform and the real speech waveform produced by each convolution layer in the discriminator, and ||·||1 is the absolute value norm.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 3 to 7.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 3 to 7.

Images