Speech synthesis system, method, medium, and device based on multi-subband generation strategy

Abstract

The embodiment of the application discloses a speech synthesis system, method, medium and equipment based on a multi-subband generation strategy, the system comprises: a preprocessing module, which is used for converting text into a phoneme sequence and converting corresponding speech into a linear spectrum through a short-time Fourier transform; a posterior encoder module, which is used for converting the linear spectrum into a posterior distribution and sampling potential variables according to a mean value and a standard deviation in the posterior distribution; a conversion layer module, which is used for converting the potential variables into sampling data; a prior encoder module, which is used for extracting the phoneme sequence into deep features and mapping the deep features into a prior distribution; a pronunciation duration estimation module, which is used for aligning the prior distribution with the sampling data, obtaining an alignment matrix, and obtaining a pronunciation duration distribution through coding according to the deep features and the alignment matrix; and a decoder module, which is used for sampling the prior distribution into multi-subband signals according to the pronunciation duration distribution and synthesizing the multi-subband signals into a speech waveform signal, so that direct mapping from text to speech is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and in particular to a speech synthesis system and method based on a multi-subband generation strategy, a medium and equipment. BACKGROUND

[0002] The field of speech synthesis is currently undergoing rapid development, but still faces a series of key problems. Among them, the problems of speech quality and generation speed are particularly prominent. Existing speech synthesis systems still have room for improvement in terms of naturalness, fluency and emotional expression. Synthetic speech often sounds stiff and unnatural, with challenges such as accent problems, unreasonable pauses and abnormal tone. The expression of emotion and tone also needs to be further improved to meet the higher requirements of different application scenarios for speech synthesis quality. Support for multiple languages and dialects is also a key issue. Different languages and dialects have different pronunciation rules and speech characteristics, requiring more refined modeling and adaptation. In addition, achieving personalized synthesis to adapt to the voice characteristics of specific users is the direction of future development, which is crucial for personalized assistants and personalized speech synthesis.

[0003] In terms of generation speed, especially in applications that require real-time response, such as voice assistants and video calls, generation speed becomes a crucial issue. Some existing speech synthesis techniques may require a large amount of computing resources, resulting in slower generation speed, which is unacceptable for real-time applications. In addition, in embedded systems, mobile devices and edge computing environments, hardware resources are limited, and more lightweight speech synthesis models and algorithms are needed to meet the needs of these resource-constrained environments. SUMMARY

[0004] Therefore, it is necessary to propose a speech synthesis system and method based on a multi-subband generation strategy to solve the above problems.

[0005] A speech synthesis system based on a multi-subband generation strategy, the system comprising a preprocessing module, a conversion layer module, a prior encoder module, a posterior encoder module, a pronunciation duration prediction module and a decoder module,

[0006] The preprocessing module is configured to convert the input text into a phoneme sequence and convert the real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform.

[0007] The posterior encoder module is configured to convert the linear spectrum into a posterior distribution and sample the latent variable according to the mean and standard deviation in the posterior distribution.

[0008] The conversion layer module is configured to convert the latent variable output by the posterior encoder module into sampling data.

[0009] The prior encoder module is configured to extract deep features of the phoneme sequence output by the preprocessing module, and map the deep features to a prior distribution.

[0010] The pronunciation duration speculation module is configured to align the prior distribution with the sampling data by monotonic alignment search, obtain an alignment matrix, sum the alignment matrix to obtain a duration sequence, conditionally encode the deep features and the duration sequence as condition information, and convert a Gaussian noise sequence into a pronunciation duration distribution by variational dequantization according to the condition information.

[0011] The decoder module is configured to sample the prior distribution according to the pronunciation duration distribution, generate amplitude and phase information of a subband signal by linear transformation on the sampled data, generate a subband signal by inverse short-time Fourier transform according to the amplitude and phase information, and synthesize a plurality of the subband signals into a speech waveform signal by a synthesis filter.

[0012] The system further comprises:

[0013] The discriminator module is configured to form an adversarial training with the decoder module, discriminate between a real speech waveform signal and the speech waveform signal, and optimize the speech waveform signal.

[0014] A speech synthesis method based on a multi-subband generation strategy, the method comprising:

[0015] Converting an input text into a phoneme sequence, and converting a real speech waveform corresponding to the input text into a linear spectrum by short-time Fourier transform.

[0016] Converting the linear spectrum into a posterior distribution, and sampling latent variables according to a mean value and a standard deviation in the posterior distribution.

[0017] Converting the output latent variables into sampling data.

[0018] Extracting deep features of the output phoneme sequence, and mapping the deep features to a prior distribution.

[0019] Aligning the prior distribution with the sampling data by monotonic alignment search, obtaining an alignment matrix, summing the alignment matrix to obtain a duration sequence, conditionally encoding the deep features and the duration sequence as condition information, and converting a Gaussian noise sequence into a pronunciation duration distribution by variational dequantization according to the condition information.

[0020] The prior distribution is sampled according to the pronunciation duration distribution, the sampled data is linearly transformed to generate amplitude and phase information of a sub-band signal, the sub-band signal is generated by inverse short-time Fourier transform according to the amplitude and phase information, and a plurality of sub-band signals are synthesized into a speech waveform signal by a synthesis filter.

[0021] The linear spectrum is converted into a posterior distribution, and a latent variable is obtained by sampling according to a mean value and a standard deviation in the posterior distribution, and the method specifically comprises:

[0022] The latent variable of the linear spectrum is extracted by a recurrent neural network.

[0023] The latent variable is output as a mean value and a standard deviation of the posterior distribution by linear transformation through a fully connected layer.

[0024] Data points are sampled by a sampling function according to the mean value and the standard deviation of the posterior distribution, and a latent variable is obtained.

[0025] The prior distribution is aligned with the sampled data by monotonic alignment search to obtain an alignment matrix, the alignment matrix is summed to obtain a duration sequence, the deep feature and the duration sequence are conditionally encoded as condition information, and a Gaussian noise sequence is converted into a pronunciation duration distribution by variational dequantization according to the condition information, and the method specifically comprises:

[0026] The prior distribution is aligned with the sampled data by monotonic alignment search to obtain an alignment matrix.

[0027] Each row of the alignment matrix is summed to obtain a duration sequence corresponding to each phoneme.

[0028] The deep feature and the duration sequence are conditionally encoded as condition information.

[0029] The Gaussian noise sequence is converted into a first random variable and a second random variable according to the condition information.

[0030] The duration sequence is subtracted from the second random variable to determine a dequantized continuous variable.

[0031] The dequantized continuous variable and the first random variable are converted into a pronunciation duration distribution.

[0032] The method further specifically comprises:

[0033] A reconstruction loss function is defined according to a real speech waveform signal and a mel spectrogram of the speech waveform signal.

[0034] A KL divergence loss function is defined according to the posterior distribution and the prior distribution.

[0035] define a duration speculation loss function according to the first random variable and the second random variable.

[0036] define an adversarial loss function and a feature matching loss function according to the discrimination of the real speech waveform signal and the speech waveform signal.

[0037] determine a total loss function through the reconstruction loss function, the KL divergence loss function, the duration speculation loss function, the adversarial loss function and the feature matching loss function.

[0038] wherein the determination of the total loss function through the reconstruction loss function, the KL divergence loss function, the duration speculation loss function, the adversarial loss function and the feature matching loss function specifically comprises:

[0039] according to L total = L recon + L kl + L dur + L adv (DI) + L adv (DE) + L fm (DE) determine a speech synthesis total loss function, wherein L total is the speech synthesis total loss function, L recon is a reconstruction loss function, x mel is a mel-spectrogram corresponding to a real speech waveform, is a mel-spectrogram corresponding to a speech waveform signal.

[0040] L kl is a KL divergence loss function, L kl = log q φ (z|x lin )-log p θ (z|c text ,A), q φ (z|x lin ) is a posterior distribution, x lin is a linear spectrum, z is a latent variable, p θ (z|c text ,A) is a prior distribution under an alignment matrix condition, A is an alignment matrix, and c text is a phoneme sequence.

[0041] L dur is a duration speculation loss function, is a log-likelihood distribution, v is a first random variable, u is a second random variable, and d is a duration sequence, logp θ (d-u,v|c text ) is a Gaussian noise distribution, is an expectation of a Gaussian noise distribution.

[0042] L adv (DI) and L adv (DE) are adversarial loss functions, L adv (DI) = E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ], L adv (DE) = E z [(DI(DE(z))-1) 2 ], DI(y) is a probability of discriminating a real speech waveform as real, DI(DE(z)) is a probability of discriminating a speech waveform signal as real, DE(z) is a speech waveform signal, E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] is an expectation of an error generated when a discriminator misjudges a real speech waveform and a generated speech waveform signal, E z [(DI(DE(z))-1) 2 ] is an expectation of an error between a probability of a discriminator judging a generated speech waveform as a real speech waveform and 1.

[0043] L fm (DE) is a feature matching loss function, DI l (y) is a probability of discriminating a feature generated after each layer of convolution of a real speech waveform as real, DI l (DE(z)) is a probability of discriminating a feature generated after each layer of convolution of a speech waveform signal as real, T is a number of speech samples, Nl is a number of convolution layers in a discrimination process, is an expectation of a weighted sum of errors between a generated speech waveform and a real speech waveform generated by each layer of convolution in a discriminator, ||·||1 is an absolute value norm.

[0044] A computer readable storage medium storing a computer program, the computer program being executed by a processor to cause the processor to perform steps of the method as described above.

[0045] A computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to perform steps of the method as described above.

[0046] By adopting the embodiment of the present application, the following beneficial effects are achieved:

[0047] The present application firstly obtains the phoneme sequence and linear spectrum of the input text through the preprocessing module, and effectively converts the frequency domain to the time domain by using the inverse operation of the short-time Fourier transform. Further, the latent variable and the deep feature are output through the prior encoder module and the posterior encoder module, and the relationship between the text and the speech waveform is obtained. Further, the pronunciation duration distribution is obtained according to the deep feature, and the pronunciation duration prosody distribution of different texts is learned through the pronunciation duration distribution, which helps the speech synthesis system based on the multi-subband generation strategy to generate more natural and high-quality speech waveforms. Finally, in the decoder module, based on the multi-subband generation strategy, the prior distribution is sampled into a multi-subband signal according to the pronunciation duration distribution and is synthesized into a speech waveform signal, which greatly improves the calculation efficiency and inference speed. The speech synthesis system based on the multi-subband generation strategy provided by the present application adopts an end-to-end training method, directly learns from the text to the speech waveform, without explicitly learning the intermediate representation or feature, and realizes the direct mapping from the text to the speech. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Among them:

[0050] Figure 1 The flowchart of an embodiment of the speech synthesis system based on the multi-subband generation strategy provided by the present application is shown in the figure.

[0051] Figure 2 The structure diagram of the posterior encoder module provided by the present application is shown in the figure.

[0052] Figure 3 The structure diagram of the conversion layer module provided by the present application is shown in the figure.

[0053] Figure 4 The structure diagram of the prior encoder module provided by the present application is shown in the figure.

[0054] Figure 5 The structure diagram of the pronunciation duration estimation module provided by the present application is shown in the figure.

[0055] Figure 6 The structure diagram of the decoder module provided by the present application is shown in the figure.

[0056] Figure 7A flowchart of an embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present application is shown in FIG. 1.

[0057] Figure 8 A flowchart of another embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present application is shown in FIG. 2.

[0058] Figure 9 A flowchart of still another embodiment of a speech synthesis method based on a multi-subband generation strategy provided by the present application is shown in FIG. 3.

[0059] Figure 10 A structural diagram of an embodiment of a device provided by the present application is shown in FIG. 4.

[0060] Figure 11 A structural diagram of an embodiment of a medium provided by the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0062] As shown in FIG. 1, Figure 1 as shown in FIG. 2, Figure 1 A structural diagram of an embodiment of a speech synthesis system based on a multi-subband generation strategy provided by the present application is shown in FIG. 1. Referring to FIG. 1, Figure 1 a speech synthesis system 10 based on a multi-subband generation strategy, the system comprising a preprocessing module 11, a posteriori encoder module 12, a conversion layer module 13, an a priori encoder module 14, a pronunciation duration speculation module 15 and a decoder module 16,

[0063] The preprocessing module 11 is configured to convert an input text into a phoneme sequence and convert a real speech waveform corresponding to the input text into a linear frequency spectrum through a short-time Fourier transform.

[0064] Exemplarily, the preprocessing module 11 converts an input text into a phoneme sequence, i.e., a Chinese syllable sequence, performs word segmentation on the input text, and divides the text into words or phrases. The segmented text is annotated with pinyin to generate a pinyin sequence. According to the pinyin sequence, each pinyin is converted into a corresponding phoneme sequence, i.e., a Chinese syllable sequence. Further, the Chinese syllable sequence is optimized, including adjusting the connection relationship between syllables and marking stress.

[0065] And, the speech waveform corresponding to the input text is preprocessed, including filtering, framing and windowing. The preprocessed speech waveform is subjected to short-time Fourier transform to obtain a time-frequency spectrogram. The frequency axis of the time-frequency spectrogram is mapped from a linear scale to a mel scale. The energy in each mel band is calculated to obtain a mel spectrum. The mel spectrum is subjected to logarithmic compression to better represent the dynamic range of the signal, and finally a linear spectrum is obtained.

[0066] The posterior encoder module 12 is used to convert the linear spectrum into a posterior distribution, and the latent variable z is obtained by sampling the mean and standard deviation in the posterior distribution.

[0067] Exemplarily, in the posterior encoder module 12, the mean and standard deviation of the posterior distribution in the linear spectrum are obtained according to the recurrent neural network and the fully connected layer, and the latent variable z is obtained by sampling the data points according to the mean and standard deviation of the posterior distribution, as shown in Figure 2 , Figure 2 The structure diagram of the posterior encoder module provided by the present application is shown. The linear spectrum is subjected to the extraction feature of the recurrent neural network WaveNet to obtain the hidden variable, and the hidden variable of the linear spectrum is output as the mean and standard deviation of the posterior distribution by a fully connected layer in the segmentation layer. Then, the data points are sampled according to the mean and standard deviation of the obtained posterior distribution by a sampling function, and the sampled data is the latent variable z.

[0068] The conversion layer module 13 is used to convert the latent variable output by the posterior encoder module into sampling data.

[0069] Exemplarily, in the conversion layer module 13, the sampling data f θ (z) of the latent variable on the prior distribution is obtained by the residual coupling layer and the flip layer, as shown in Figure 3 , Figure 3 The structure diagram of the conversion layer module provided by the present application is shown. The latent variable z is taken as the input, wherein the latitude of z is [B, 192, T]. The latent variable z is divided into a first segmentation variable x0 and a second segmentation variable x1, and then the mean μ p of the latent variable z on the prior distribution is obtained by the recurrent neural network waveNet and the one-dimensional convolution layer conv1d. The second segmentation variable on the prior distribution is determined according to the mean μ p of the latent variable z and the second segmentation variable x1, and the second segmentation variable on the prior distribution is as follows:

[0070]

[0071] wherein, is the second segmentation variable on the prior distribution, , x0 is the first split variable, x1 is the second split variable, the dimension of x1 is [B, 96, T], μ p is the mean of the latent variable on the prior distribution, and σ is the standard deviation of the latent variable on the prior distribution.

[0072] Further, the first split variable x0[B, 96, T] and the second split variable on the prior distribution are connected through a connection layer, and after connection, the first split variable x0 and the second split variable on the prior distribution are reversed in position through a flip layer, and the first split variable on the prior distribution is obtained again through a residual coupling layer. After connecting the first split variable on the prior distribution and the second split variable on the prior distribution, z p is obtained. θ (z).

[0073] The prior encoder module 14 is used to extract the deep features of the phoneme sequence output by the preprocessing module, and map the deep features to the prior distribution.

[0074] Exemplarily, the mean and standard deviation of the posterior distribution in the linear spectrum are obtained according to the recurrent neural network and the fully connected layer, the data points are sampled through the mean and standard deviation of the posterior distribution, the latent variable is obtained, and in the prior encoder module 14, the prior distribution of the phoneme sequence is determined according to the deep features of the phoneme sequence, as shown in Figure 4 , Figure 4 is the structure diagram of the prior encoder module provided by the application. The phoneme sequence is converted into a 192-dimensional feature vector through an embedding layer, and the phoneme is embedded into the original sequence to obtain a sequence [B, T, 192]. The dimension conversion sequence obtained by the transposition layer is [B, 192, T], and the attention layer is used to extract the information depth of the conversion sequence to obtain the deep feature h text of each sample. Subsequently, the deep feature h text is mapped through a mapping layer and is subjected to a split operation to obtain the mean μ θ and the standard deviation σ θ of the prior distribution of the phoneme sequence. θ θ The prior distribution is determined according to the mean μ

[0075] The pronunciation duration estimation module 15 is used to align the prior distribution with the sampling data through monotonic alignment search to obtain an alignment matrix, sum the alignment matrix to obtain a duration sequence, conditionally encode the deep feature and the duration sequence as conditional information, and convert the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization according to the conditional information. ​

[0076] Exemplarily, the sampling data f θ (z) is aligned with the prior distribution of the phoneme sequence, and the correspondence between the text and the speech waveform signal is realized. According to the monotonicity constraint idea, that is, the pronunciation of the synthesized speech is consistent with the order of the text, the phonemes and the speech waveform are monotonically aligned, that is, one phoneme can correspond to multiple sampling points of the speech waveform, but one sampling point of the speech waveform can only correspond to one phoneme. The maximum likelihood value of each corresponding strategy is solved by using a dynamic programming algorithm. The corresponding strategy with the maximum likelihood value is regarded as the sampling data f θ (z) and the prior distribution of the phoneme sequence. The correspondence sequentially corresponds each sampling data f θ (z) and the prior distribution of the sequence, and generates an alignment matrix.

[0077] Further, the sum of each row of the alignment matrix is obtained, and the duration sequence d corresponding to each phoneme is obtained. The duration sequence d and the deep feature h text corresponding to the text are input.

[0078] It should be noted that, when the deep feature h text corresponding to the text is input, a gradient stop controller is used to prevent the update of the gradient, so as to avoid affecting the gradient calculation in the prior encoder training process. Since the input duration sequence d is a discrete integer sequence, it is difficult to apply maximum likelihood estimation, and the method of variational dequantization and variational data enhancement is adopted.

[0079] Further, in the pronunciation duration speculation module 15, the alignment matrix is converted into a pronunciation duration distribution by summation and encoding, as shown in Figure 5 . Figure 5 is a structural schematic diagram of the pronunciation duration speculation module provided by the present application. The duration sequence d and the deep feature h text corresponding to the text are input into the variational encoder as conditional information after being encoded by the conditional encoder; the variational encoder converts a Gaussian noise sequence into a first random variable v and a second random variable u according to the conditional information, and uses the first random variable v and the second random variable u to represent the approximate distribution; the duration sequence d is subtracted from the second random variable u to determine the dequantized continuous variable d-u; subsequently, the dequantized continuous variable d-u and the first random variable v are converted into a pronunciation duration distribution Noise ~ N(0, I) to represent the dequantized and enhanced log-likelihood distribution. The pronunciation duration distribution Noise ~ N(0, I) can sample the pronunciation duration corresponding to the text in the distribution sequence.

[0080] It should be noted that the first random variable v is the noise variable for data augmentation, and the second random variable u is the background noise variable for duration, ranging from [0, 1).

[0081] The decoder module 16 is used to sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the sub-band signal by linear transformation of the sampled data, generate the sub-band signal by short-time Fourier inverse transform based on the amplitude and phase information, and synthesize the multiple sub-band signals into a speech waveform signal by synthesis filter.

[0082] For example, the decoder module 16 is first trained, and the amplitude and phase information of the sub-band signal in the latent variable z are extracted in the decoder module 16. The amplitude and phase information of the sub-band signal are then subjected to the inverse operation of short-time Fourier transform to generate the sub-band signal. Finally, the multiple sub-band signals are synthesized into a speech waveform signal by a synthesis filter. like Figure 6 As shown, Figure 6 This is a schematic diagram of the decoder module provided by the present invention. The latent variable z is upsampled through two residual convolutional blocks with parameter s, where parameter s is [4, 4] and is the upsampling scale parameter. Subsequently, it undergoes a linear transformation through a fully connected layer to generate amplitude and phase information of four sub-band signals.

[0083] Furthermore, the amplitude and phase information of each sub-band signal are subjected to an iSTFT (Inverse Short-Time Fourier Transform) operation to generate the sub-band signal. The number of Fast Fourier Transform sampling points, the iSTFT jump length, and the window length are 16, 4, and 16, respectively. Upsampling is performed on the four sub-band signals by adding zeros between samples to match the sampling rate of the original signal. Subsequently, a trainable synthesis filter is used to synthesize the four sub-band signals into a speech waveform signal. Among them, the trainable synthetic filters include pseudo-orthogonal mirror filter banks.

[0084] Furthermore, the trained decoder module 16 samples the prior distribution according to the pronunciation duration distribution Noise~N(0,I). The sampled data is then transformed linearly to generate the amplitude and phase information of the sub-band signal. Based on the amplitude and phase information, the sub-band signal is generated through short-time Fourier inverse transform. Finally, the multiple sub-band signals are synthesized into a speech waveform signal through a synthesis filter. Specifically, based on the pronunciation duration distribution Noise~N(0,I) and the depth features h of the prior distribution... textobtaining a duration sequence d by rounding up the continuous pronunciation time length of the corresponding sample. θ (z) (i.e., the sampled data). The sampled data is transformed linearly to generate amplitude and phase information of the subband signal, the subband signal is generated by inverse short-time Fourier transform according to the amplitude and phase information, and the multiple subband signals are synthesized into a speech waveform signal by a synthesis filter and sound through a loudspeaker.

[0085] Further, the speech synthesis system based on the multi-subband generation strategy further comprises a discriminator module for forming an adversarial training with the decoder module 16, discriminating between the real speech waveform signal and the speech waveform signal, and optimizing the speech waveform signal. The discriminator module and the decoder module 16 form an adversarial training, discriminate between the real speech waveform signal and the speech waveform signal, and optimize the speech waveform signal, so that the speech waveform signal output by the decoder module 16 is closer to the real speech waveform.

[0086] As described above, the present application first obtains the phoneme sequence and linear spectrum of the input text through the preprocessing module, and effectively converts the frequency domain to the time domain by inverse operation of the short-time Fourier transform. Further, the latent variable and the deep feature are output by the prior encoder module and the posterior encoder module, and the relationship between the text and the speech waveform is obtained. Further, the pronunciation time length distribution is obtained according to the deep feature, and the pronunciation time length rhythm distribution of different texts is learned through the pronunciation time length distribution, which helps the speech synthesis system based on the multi-subband generation strategy to generate more natural and high-quality speech waveforms. Finally, in the decoder module, based on the multi-subband generation strategy, the prior distribution is sampled into a multi-subband signal according to the pronunciation time length distribution and synthesized into a speech waveform signal, which greatly improves the calculation efficiency and inference speed. The speech synthesis system based on the multi-subband generation strategy provided by the present application adopts an end-to-end training method, directly learns from the text to the speech waveform, without explicitly learning the intermediate representation or feature, and realizes the direct mapping from the text to the speech.

[0087] Figure 7 The flowchart of an embodiment of the speech synthesis method based on the multi-subband generation strategy provided by the present application is shown in FIG. 1. Figure 7 A speech synthesis method based on a multi-subband generation strategy, the method comprising:

[0088] S101: converting the text into a phoneme sequence, and converting the real speech waveform corresponding to the text into a linear spectrum.

[0089] Exemplarily, the input text is converted into a phoneme sequence, i.e., a Chinese syllable sequence, the input text is segmented into words or phrases, and the segmented text is annotated with pinyin to generate a pinyin sequence. According to the pinyin sequence, each pinyin is converted into a corresponding phoneme sequence, i.e., a Chinese syllable sequence. The real speech waveform corresponding to the input text is preprocessed, and the preprocessed speech waveform is subjected to short-time Fourier transform to obtain a time-frequency spectrogram. The time-frequency spectrogram is subjected to frequency mapping to calculate the energy in each mel frequency band to obtain a mel spectrum. The mel spectrum is subjected to logarithmic compression to finally obtain a linear spectrum.

[0090] S102: The linear spectrum is converted into a posterior distribution, and the latent variable is obtained by sampling according to the mean and standard deviation in the posterior distribution.

[0091] Exemplarily, the linear spectrum is input into a recurrent neural network WaveNet, which includes multiple consecutive convolution layers. After each convolution layer, the linear spectrum is added to the output of the convolution layer to obtain a residual signal. The residual signal is subjected to normalization processing, usually batch normalization. An activation function such as ReLU (Rectified Linear Unit) is applied to the normalized residual signal, and the above steps are repeated until the output of the residual block, which is the extracted linear spectrum latent variable. The linear spectrum latent variable is input into a fully connected layer, which includes multiple consecutive linear layers. The fully connected layer obtains the mean of the posterior distribution through one linear layer, and obtains the standard deviation of the posterior distribution through another linear layer. According to the obtained mean and standard deviation of the posterior distribution, data points are sampled by a sampling function, and the sampled data is denoted as the latent variable z.

[0092] S103: The output latent variable is converted into sampled data.

[0093] Exemplarily, in the conversion layer module 13, the sampled data f(z) of the latent variable z on the prior distribution is obtained through a residual coupling layer and a flip layer. First, the latent variable z is input, and the latent variable z is divided into a first divided variable x0 and a second divided variable x1. Then, the latent variable z is obtained on the prior distribution through a recurrent neural network waveNet and a one-dimensional convolution layer conv1d. θ p The mean μ(z) of the latent variable z on the prior distribution is determined according to the mean μ(z) of the latent variable z and the second divided variable x1. p

[0094] ​​Further, the first split variable x0 and the second split variable on the prior distribution are connected through a connection layer, and after connection, the first split variable x0 and the second split variable on the prior distribution are reversed in position through a flip layer, and the first split variable on the prior distribution is obtained again through a residual coupling layer, and the first split variable on the prior distribution and the second split variable on the prior distribution are connected, and after 4 times of residual coupling layer and flip, the sampling data f of the output latent variable on the prior distribution is obtained θ (z).

[0095] S104: Extract the deep feature of the output phoneme sequence, and map the deep feature to the prior distribution.

[0096] Illustratively, the phoneme sequence is converted into a feature vector through an embedding layer, the feature vector is embedded into the phoneme sequence to obtain an embedded sequence. The embedded sequence is subjected to latitude conversion through a transpose layer to obtain a converted sequence. The information of the converted sequence is extracted through an attention layer to obtain the deep feature h of the phoneme sequence text The deep feature h text is divided to obtain the prior distribution of the phoneme sequence.

[0097] S105: Align the prior distribution with the sampling data through monotonic alignment search to obtain an alignment matrix, sum the alignment matrix to obtain a duration sequence, conditionally encode the deep feature and the duration sequence as condition information, and convert the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization according to the condition information.

[0098] Illustratively, the prior distribution is aligned with the sampling data f θ (z) through monotonic alignment search to obtain an alignment matrix; each row of the alignment matrix is summed to obtain a duration sequence d corresponding to each phoneme; the deep feature h text and the duration sequence d are conditionally encoded as condition information; the Gaussian noise sequence is converted into a first random variable v and a second random variable u according to the condition information; the duration sequence d is subtracted from the second random variable u to determine the dequantized continuous variable d-u; the dequantized continuous variable d-u and the first random variable v are converted into a pronunciation duration distribution Noise ~ N(0, I).

[0099] S106: Sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the subband signal through linear transformation on the sampled data, generate the subband signal through inverse short-time Fourier transform according to the amplitude and phase information, and synthesize the subband signal into a speech waveform signal through a synthesis filter.

[0100] Illustratively, the deep feature h textand the pronunciation duration distribution Noise ~ N(0, I) to obtain continuous pronunciation duration d; the continuous pronunciation duration d is rounded up to obtain a duration sequence; the prior distribution is sampled according to the duration sequence; the sampled data is generated through linear transformation to generate amplitude and phase information of the sub-band signal; the sub-band signal is generated through inverse short-time Fourier transform according to the amplitude and phase information; and the multiple sub-band signals are synthesized into a speech waveform signal through a synthesis filter and sound through a loudspeaker.

[0101] As Figure 8 shown, Figure 8 is a flowchart of another embodiment of a speech synthesis method based on a multi-sub-band generation strategy provided by the present application. Referring to Figure 8 , a speech synthesis method based on a multi-sub-band generation strategy, the method comprises:

[0102] S201: defining a reconstruction loss function according to a real speech waveform and a mel-spectrogram of a speech waveform signal.

[0103] Exemplarily, in order to evaluate the effect of the decoder module 16 on converting the latent variable z into a speech waveform signal and update the parameters in the decoder module 16 in the training stage, the reconstruction loss function is defined as follows:

[0104]

[0105] The present application uses a mel-spectrogram instead of a speech waveform to calculate the reconstruction loss function. Wherein, L recon is the reconstruction loss function, x mel is the mel-spectrogram corresponding to the real speech waveform, is the mel-spectrogram corresponding to the speech waveform signal.

[0106] S202: defining a KL divergence loss function according to a posterior distribution and a prior distribution.

[0107] Exemplarily, in order to make the sampling data f θ (z) generated by sampling the prior distribution generated by the prior encoder module 14 with the phoneme sequence as input closer to the latent variable z sampled by the posterior distribution generated by the posterior encoder module 12 with the linear spectrum corresponding to the speech waveform as input, so that the sampling data f θ (z) output by the prior encoder module 14 is closer to the real distribution of the speech waveform, thereby improving the generation quality of the text to speech waveform, and the KL divergence (Kullback-Leibler Divergence, relative entropy) loss function is defined as follows:

[0108] L kl = log qφ (z|x lin )-log p θ (z|c text ,A) (3)

[0109] where L kl is the KL divergence loss function, q φ (z|x lin ) is the posterior distribution, x lin is the linear spectrum, z is the latent variable, p θ (z|c text ,A) is the prior distribution under the alignment matrix condition, A is the alignment matrix, c text is the phoneme sequence.

[0110] S203: defining a pronunciation duration prediction loss function according to the first random variable and the second random variable.

[0111] Exemplarily, in order to detect the correctness of the duration of the speech waveform corresponding to the text, and adjust the parameters in the process of reconstruction loss, so as to improve the prediction accuracy, the pronunciation duration prediction loss function L dur is defined as follows:

[0112]

[0113] where v is the first random variable, u is the second random variable, d is the duration sequence, is the approximate distribution, logp θ (d-u,v|c text ) is the log-likelihood distribution, is the log-likelihood expectation, is the expectation of the log-likelihood distribution.

[0114] S204: defining an adversarial loss function and a feature matching loss function according to the discrimination of the real speech waveform signal and the speech waveform signal.

[0115] Exemplarily, the speech synthesis system based on the multi-subband generation strategy further comprises an discriminator module, which is used to form an adversarial training with the decoder module 16, and discriminate the real speech waveform signal and the speech waveform signal to optimize the speech waveform signal. The discriminator module is used to form an adversarial training with the decoder module 16, so that the speech waveform signal output by the decoder module 16 is closer to the real speech waveform. The real speech waveform and the speech waveform signal output by the decoder module 16 as an input of the discriminator module. The discriminator module judges whether the input waveform is a real waveform, and the decoder module 16 adjusts the parameters in the training process according to the feedback result of the discriminator module until the discriminator module cannot judge whether the input waveform is a real speech waveform or a speech waveform signal generated by the decoder module 16

[0116] In order to detect the effect of the speech waveform signal generated by the decoder module 16 , and adjust the parameters in the training process of the discriminator module and the decoder module 16, the adversarial loss function L adv (DI) and L adv (DE) are defined as follows:

[0117] L adv (DI) = E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] (5)

[0118] L adv (DE) = E z [(DI(DE(z))-1) 2 ] (6)

[0119] Where DI(y) is the probability of identifying a real speech waveform as real, DI(DE(z)) is the probability of identifying a speech waveform signal as real, DE(z) is a speech waveform signal, E (y,z) [(DI(y)-1) 2 +(DI(DE(z))) 2 ] is the expected error generated when the discriminator incorrectly distinguishes between real speech waveforms and generated speech waveform signals, and E z [(DI(DE(z))-1) 2 ] is the expected error between the probability of the discriminator judging the generated speech waveform as a real speech waveform and 1.

[0120] Further, the feature matching loss function is defined according to the identification of the features generated after each layer of convolution of the speech waveform signal and the identification of the features generated after each layer of convolution of the real speech waveform, as follows:

[0121]

[0122] Where DI l (y) is the probability of identifying the features generated after each layer of convolution of the real speech waveform as real, and DI l(DE(z)) is the probability of the feature discrimination being true after the speech waveform signal is convolved through each layer, T is the number of speech samples, Nl is the number of convolution layers in the discrimination process, is the expectation of the weighted sum of the error between the generated speech waveform and the real speech waveform produced by each layer of convolution in the discriminator, ||·||1 is the absolute value norm, used to measure DI l (y) and DI l (DE(z)) between elements.

[0123] S205: Determine the total loss function by the reconstruction loss function, the KL divergence loss function, the pronunciation duration estimation loss function, the adversarial loss function and the feature matching loss function.

[0124] Exemplarily, determine the speech synthesis total loss function according to the reconstruction loss function, the KL divergence loss function, the pronunciation duration estimation loss function, the definition of the adversarial loss function and the feature matching loss function, as shown in the following formula:

[0125] L total = L recon + L kl + L dur + L adv (DI) + L adv (DE) + L fm (DE) (8)

[0126] Wherein, L total is the speech synthesis total loss function.

[0127] Since the discriminator module is set to improve the quality of the speech waveform generated by the decoder module 16. It does not directly participate in the text-to-speech generation process. Therefore, the speech synthesis total loss function used in the text-to-speech generation part is summarized as formula (9):

[0128] L Gen = L recon + L kl + L dur + L adv (DE) + L fm (DE) (9)

[0129] Wherein, L Gen is the final speech synthesis total loss function.

[0130] Further, parameters of the preprocessing module 11, the posterior encoder module 12, the conversion layer module 13, the prior encoder module 14, the pronunciation duration speculation module 15, the decoder module 16 and the discriminator module are updated according to the total loss function, and iteration is repeated to obtain the trained preprocessing module 11, the posterior encoder module 12, the conversion layer module 13, the prior encoder module 14, the pronunciation duration speculation module 15, the decoder module 16 and the discriminator module.

[0131] As shown in Figure 9 , Figure 9 FIG. 1 is a flowchart of a speech synthesis method based on a multi-subband generation strategy according to an embodiment of the present application. As shown in Figure 9 , the speech synthesis method based on the multi-subband generation strategy comprises the following steps.

[0132] S301: converting an input text into a phoneme sequence and converting a real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform.

[0133] It should be noted that step S301 has been discussed in detail in the embodiment scenario shown in Figure 1 , which will not be repeated here.

[0134] S302: extracting hidden variables of the linear spectrum through a recurrent neural network.

[0135] Exemplarily, the linear spectrum is input into the recurrent neural network, and the recurrent neural network comprises a plurality of continuous convolution layers. After each convolution layer, the linear spectrum is added to the output of the convolution layer to obtain a residual signal. The residual signal is normalized, and batch normalization is usually used. An activation function such as ReLU is applied to the normalized residual signal, and the above steps are repeated until the output of the residual block, which is the extracted hidden variable of the linear spectrum.

[0136] S303: outputting the hidden variable as the mean and standard deviation of the posterior distribution through linear transformation by a fully connected layer.

[0137] S304: obtaining latent variables through data point sampling by a sampling function according to the mean and standard deviation of the posterior distribution.

[0138] Exemplarily, the hidden variable of the linear spectrum is input into the fully connected layer, and the fully connected layer comprises a plurality of continuous linear layers. The fully connected layer obtains the mean of the posterior distribution through one linear layer, and obtains the standard deviation of the posterior distribution through another linear layer. According to the obtained mean and standard deviation of the posterior distribution, data points are sampled by a sampling function, and the sampled data is recorded as latent variables z.

[0139] S305: extracting deep features of the output phoneme sequence and mapping the deep features to a prior distribution.

[0140] It should be noted that steps S302-S304 and step S305 are performed simultaneously, and they have already been... Figure 1 The implementation scenarios shown are discussed in detail and will not be repeated here.

[0141] S306: Convert the output latent variables into sampled data.

[0142] For example, the latent variable z is first taken as input and split into a first splitting variable x0 and a second splitting variable x1. Then, the mean μ of the latent variable z on the prior distribution is obtained through a recurrent neural network and a one-dimensional convolutional layer. p Based on the mean μ of the latent variables p The second splitting variable x1 determines the second splitting variable on the prior distribution.

[0143] Furthermore, a connection layer is used to connect the first segmentation variable x0 and the second segmentation variable on the prior distribution. After the connection, a flip layer is used to reverse the positions of the first segmentation variable x0 and the second segmentation variable on the prior distribution. The first segmentation variable on the prior distribution is obtained again through a residual coupling layer. The first segmentation variable on the prior distribution and the second segmentation variable on the prior distribution are then connected. After four residual coupling layers and flips, the sampling data f of the output latent variable on the prior distribution is obtained. θ (z).

[0144] S307: Align the prior distribution with the sampled data through monotonic alignment search to obtain the alignment matrix.

[0145] In one implementation scenario, the monotonic alignment search method is used to sample the data f. θ (z) Alignment is performed with the prior distribution of the phoneme sequence to achieve the correspondence between the text and the speech waveform signal. Based on the monotonicity constraint, i.e., the pronunciation of the synthesized speech is consistent with the order of the text, phoneme and speech waveform are monotonically aligned. One phoneme can correspond to multiple sampling points of the speech waveform, but one speech waveform sampling point can only correspond to one phoneme. A dynamic programming algorithm is used to solve for the maximum likelihood value of each correspondence strategy. The correspondence strategy with the largest maximum likelihood value is considered as the sampling data f. θ (z) represents the correspondence between the prior distribution of the phoneme sequence and the data. This correspondence sequentially assigns each sampled data f... θ (z) is mapped to the prior distribution of the sequence and an alignment matrix is ​​generated.

[0146] Specifically, a dynamic programming table is first constructed, with a size equal to the length of the input text sequence multiplied by the length of the target speech. Each entry represents a possible alignment.

[0147] Further, the first row of the table is initialized, indicating the starting position that the first input text unit can be aligned with.

[0148] Further, the first row of the table is initialized, indicating the starting position that the first input text unit can be aligned with.

[0149] For each unfilled table item (i, j);

[0150] Calculate its likelihood value score(i, j);

[0151] Select the last layer maximum value table item to fill in the unfilled table item (i, j), or keep it empty;

[0152] Record the maximum value position of the unfilled table item (i, j), which is the maximum likelihood value.

[0153] Further, after the traversal is completed, the path corresponding to the maximum value of the last row of the record table is traced back to obtain the optimal alignment relationship.

[0154] Further, output the optimal alignment relationship, which is the corresponding strategy with the maximum likelihood value.

[0155] Further, generate an alignment matrix according to the optimal alignment relationship.

[0156] Specifically, first initialize an empty alignment matrix with a size of input text sequence length x target audio length.

[0157] Traverse the obtained optimal correspondence:

[0158] For each (text unit i, audio position j) pair:

[0159] Assign the value of the i-th row and j-th column element of the alignment matrix to 1, and keep other elements as 0, and output the constructed alignment matrix.

[0160] For example: if the optimal correspondence record is:

[0161] Input text 1 corresponds to audio segment 1-3

[0162] Input text 2 corresponds to audio segment 4-6

[0163] Input text 3 corresponds to audio segment 7-9

[0164] The constructed alignment matrix is: [[1 1 1 0 0 0 0 0 0] [0 0 0 1 1 1 0 0 0] [0 0 0 0 0 0 1 1 1]

[0168] wherein each row represents an input text unit, and each column represents an audio time step. The position of non-zero elements indicates that the input text unit generates audio at the time step.

[0169] S308: Sum each row of the alignment matrix to obtain a duration sequence corresponding to each phoneme.

[0170] S309: Conditionally encode the deep feature and the duration sequence as conditional information.

[0171] Exemplarily, sum each row of the alignment matrix to obtain a duration sequence d corresponding to each phoneme, and refer to Figure 5 , the duration sequence d and the deep feature h corresponding to the text text are encoded by a conditional encoder as conditional information input into a variational encoder.

[0172] S310: Convert a Gaussian noise sequence into a first random variable and a second random variable according to the conditional information.

[0173] Exemplarily, the conditional encoder converts a Gaussian noise sequence into a first random variable v and a second random variable u according to the conditional information of the deep feature h text and the duration sequence d, and uses the first random variable v and the second random variable u to approximate the distribution.

[0174] S311: Subtract the duration sequence from the second random variable to determine a dequantized continuous variable.

[0175] S312: Convert the dequantized continuous variable and the first random variable into a pronunciation duration distribution.

[0176] Exemplarily, subtract the duration sequence d from the second random variable u to obtain a dequantized continuous variable d-u; then, convert the dequantized continuous variable d-u and the first random variable v into a pronunciation duration distribution to represent a dequantized and enhanced log-likelihood distribution.

[0177] S313: Sample the prior distribution according to the pronunciation duration distribution, generate amplitude and phase information of a subband signal through linear transformation of the sampled data, generate the subband signal through inverse short-time Fourier transform according to the amplitude and phase information, and synthesize the subband signal into a speech waveform signal through a synthesis filter.

[0178] It should be noted that step S313 has been discussed in detail in the implementation scenario shown in Figure 7 , and will not be repeated here.

[0179] As shown in Figure 10 , the speech waveform signal is output. Figure 10A structural schematic diagram of an embodiment of the device provided by the present application. The device 20 comprises a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 executes the computer program when working to implement the method shown in Figure 7 、 Figure 8 and Figure 9 .

[0180] The specific technical details of the speech synthesis method based on a multi-subband generation strategy implemented by the device 20 executing the computer program have been discussed in detail in the above method steps, and thus will not be repeated here.

[0181] As shown in Figure 11 , Figure 11 is a structural schematic diagram of an embodiment of the medium provided by the present application. The medium 30 stores at least a computer program 31, and the computer program 31 is executed by the processor 22 to implement the method shown in Figure 7 、 Figure 8 and Figure 9 . For details, please refer to the above, which will not be repeated here. In an embodiment, the medium 30 can be a storage chip, a hard disk or a mobile hard disk or an optical disc, or other readable and writable storage tools, and can also be a server, etc.

[0182] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments of the method. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0183] Any technical features in the above-described embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations are described, but it is understood that the scope of the present specification includes all possible combinations.

[0184] The above-described embodiments are merely exemplary and do not limit the scope of the application. It should be apparent to those skilled in the art that various modifications and improvements can be made to the embodiments without departing from the scope of the application. Accordingly, the scope of the application should be determined by the appended claims and their legal equivalents rather than by the embodiments described above.

[0185] The various embodiments in the present specification are described by way of progression, and for the sake of brevity, the same or similar parts in the various embodiments are not described in detail again. Each embodiment focuses on the differences from other embodiments. In particular, the system embodiments are not described in detail because they are substantially similar to the method embodiments.

[0186] The above-described embodiments are merely exemplary and do not limit the scope of the application. It should be apparent to those skilled in the art that various modifications and improvements can be made to the embodiments without departing from the scope of the application. Accordingly, the scope of the application should be determined by the appended claims and their legal equivalents rather than by the embodiments described above.

Claims

1. A speech synthesis method based on a multi-subband generation strategy, characterized in that, The method includes: The input text is converted into a phoneme sequence, and the real speech waveform corresponding to the input text is converted into a linear spectrum through short-time Fourier transform. The linear spectrum is transformed into a posterior distribution, and latent variables are obtained by sampling based on the mean and standard deviation of the posterior distribution. The output latent variables are converted into sampled data; Extract the depth features of the output phoneme sequence and map the depth features to a prior distribution; Aligning the prior distribution with the sampled data using monotonic alignment search to obtain an alignment matrix, summing the alignment matrix to obtain a duration sequence, conditionally encoding the depth features and the duration sequence as conditional information, and converting the Gaussian noise sequence into a speech duration distribution through variational dequantization based on the conditional information, specifically includes: aligning the prior distribution with the sampled data using monotonic alignment search to obtain an alignment matrix, summing the alignment matrix to obtain a duration sequence, conditionally encoding the depth features and the duration sequence as conditional information, and converting the Gaussian noise sequence into a speech duration distribution through variational dequantization based on the conditional information. Alignment matrix; summing each row of the alignment matrix to obtain the duration sequence corresponding to each phoneme; conditionally encoding the deep features and the duration sequence as conditional information; transforming the Gaussian noise sequence into a first random variable and a second random variable based on the conditional information, specifically including: dividing the latent variable into a first segmentation variable and a second segmentation variable, obtaining the mean of the latent variable on the prior distribution, determining the second segmentation variable on the prior distribution based on the mean of the latent variable and the second segmentation variable, and obtaining the sampled data of the output latent variable on the prior distribution after passing through 4 residual coupling layers and flipping; subtracting the duration sequence from the second random variable to determine the dequantized continuous variable; transforming the dequantized continuous variable and the first random variable into a pronunciation duration distribution; The prior distribution is sampled according to the pronunciation duration distribution. The sampled data is transformed linearly to generate the amplitude and phase information of the sub-band signal. The sub-band signal is generated by inverse short-time Fourier transform according to the amplitude and phase information. The multiple sub-band signals are synthesized into a speech waveform signal by a synthesis filter. The method further includes: defining a loss function for predicting pronunciation duration based on the first random variable and the second random variable. To infer the loss function for pronunciation duration, , Let v be a log-likelihood distribution, v be the first random variable, u be the second random variable, and d be the duration sequence. Let the expected value be the log-likelihood distribution. The noise distribution is Gaussian. This represents the expectation of the Gaussian noise distribution.

2. The speech synthesis method based on a multi-subband generation strategy according to claim 1, characterized in that, The process of transforming the linear spectrum into a posterior distribution and sampling based on the mean and standard deviation of the posterior distribution to obtain latent variables specifically includes: The latent variables of the linear spectrum are extracted using a recurrent neural network; The latent variables are output as the mean and standard deviation of the posterior distribution by performing a linear transformation through a fully connected layer. Data points are sampled using a sampling function based on the mean and standard deviation of the posterior distribution to obtain latent variables.

3. The speech synthesis method based on a multi-subband generation strategy according to claim 1, characterized in that, The method further includes: The reconstruction loss function is defined based on the real speech waveform signal and the Mel spectrogram of the speech waveform signal; Define the KL divergence loss function based on the posterior distribution and the prior distribution; Based on the identification of the real speech waveform signal and the speech waveform signal, an adversarial loss function and a feature matching loss function are defined; The total loss function is determined by the reconstruction loss function, the KL divergence loss function, the pronunciation duration inference loss function, the adversarial loss function, and the feature matching loss function.

4. The speech synthesis method based on a multi-subband generation strategy according to claim 3, characterized in that, The determination of the total loss function through the reconstruction loss function, the KL divergence loss function, the pronunciation duration estimation loss function, the adversarial loss function, and the feature matching loss function specifically includes: according to Determine the total loss function for speech synthesis, where, Let be the total loss function for speech synthesis. To reconstruct the loss function, , The Mel spectrogram corresponds to the actual speech waveform. The Mel spectrum corresponding to the speech waveform signal; Let KL divergence loss function be used. , For the posterior distribution, Let z be a linear spectrum, and z be a latent variable. Let A be the prior distribution given the alignment matrix, and A be the alignment matrix. It is a phoneme sequence; and To counter the loss function, , , The probability of identifying a real speech waveform as genuine. The probability of identifying a speech waveform signal as true. It is a voice waveform signal. The expected error caused by the discriminator misjudging when distinguishing between real and generated speech waveform signals. The expected error between the probability that the discriminator determines the generated speech waveform as a real speech waveform and 1; For feature matching loss function, , This represents the probability of identifying the features generated after each convolutional layer of a real speech waveform as true. Let T be the probability that the features generated after each convolutional layer of the speech waveform signal are true, and Nl be the number of speech samples and the number of convolutional layers in the discrimination process. This is the expected value of the weighted sum of the errors between the generated speech waveform and the real speech waveform produced by each convolutional layer in the discriminator. It is the absolute value norm.

5. A speech synthesis system based on a multi-subband generation strategy, characterized in that, The system performs the method as described in any one of claims 1-4, and the system includes a preprocessing module, a conversion layer module, a priori encoder module, a posterior encoder module, a pronunciation duration estimation module, and a decoder module. The preprocessing module is used to convert the input text into a phoneme sequence and convert the real speech waveform corresponding to the input text into a linear spectrum through short-time Fourier transform. The posterior encoder module is used to convert the linear spectrum into a posterior distribution and to sample the latent variables based on the mean and standard deviation of the posterior distribution. The conversion layer module is used to convert the latent variables output by the posterior encoder module into sampled data; The prior encoder module is used to extract the depth features of the phoneme sequence output by the preprocessing module and map the depth features to a prior distribution. The pronunciation duration estimation module is used to align the prior distribution with the sampled data through monotonic alignment search to obtain an alignment matrix, sum the alignment matrix to obtain a duration sequence, conditionally encode the depth features and the duration sequence as conditional information, and convert the Gaussian noise sequence into a pronunciation duration distribution through variational dequantization based on the conditional information. The decoder module is used to sample the prior distribution according to the pronunciation duration distribution, generate the amplitude and phase information of the sub-band signal by linear transformation of the sampled data, generate the sub-band signal by short-time Fourier inverse transform according to the amplitude and phase information, and synthesize the multiple sub-band signals into a speech waveform signal by synthesis filter.

6. The speech synthesis system based on a multi-subband generation strategy according to claim 5, characterized in that, The system also includes: The discriminator module is used to conduct adversarial training with the decoder module to distinguish between the real speech waveform signal and the speech waveform signal, and to optimize the speech waveform signal.

7. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 4.

8. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 4.

Images