A speech synthesis method, system, terminal and medium of a two-stage neural vocoder

Abstract

The application discloses a speech synthesis method, system, terminal and medium of a two-stage neural vocoder, relates to the technical field of speech synthesis, and the two stages include an amplitude reconstruction stage and a phase generation stage. The method comprises the following steps: in the amplitude reconstruction stage, an estimated amplitude spectrum is obtained by using a supervised learning manner based on an ICCRN model; in the phase generation stage, a phase spectrum highly fitted with the estimated amplitude spectrum is generated by combining a generative adversarial network based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the ICCRN model; and a synthesized speech sample is obtained based on the estimated amplitude spectrum and the phase spectrum. The ICCRN model is applied to the amplitude spectrum reconstruction stage and the phase spectrum generation stage, and in the phase spectrum generation stage, the GAN mode is innovatively proposed, the estimated amplitude spectrum is used to generate a phase spectrum highly fitted with the amplitude, the one-to-many relationship inherent to the phase-amplitude is solved, and the audio fidelity and the device compatibility are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and particularly relates to a speech synthesis method, system, terminal and medium of a two-stage neural vocoder. BACKGROUND

[0002] As a core component of speech synthesis, the neural vocoder is mainly used for converting low-dimensional acoustic features (such as mel-spectrogram) into high-quality waveform signals. The existing neural vocoder mainly includes the following five types: autoregressive model, flow-based model, GAN (Generative Adversarial Network)-based model, diffusion model and variational autoencoder.

[0003] However, the mel-spectrogram only provides amplitude information, and the phase information is missing and does not have a one-to-one correspondence with the amplitude. Due to the limited transmission of amplitude and phase information, it is easy to cause performance collapse on larger datasets, which will lead to spectral artifacts, metallic sound distortion and weak generalization.

[0004] Therefore, the prior art still has defects. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a speech synthesis method, system, terminal and medium of a two-stage neural vocoder to solve the above defects of the prior art. The technical solution adopted by the present application is as follows:

[0006] In a first aspect, the present application provides a speech synthesis method of a two-stage neural vocoder, wherein the two-stage neural vocoder adopts an in-situ cepstral convolution recurrent neural network model, and the two-stage includes an amplitude reconstruction stage and a phase generation stage. The method comprises:

[0007] In the amplitude reconstruction stage, the mel-spectrogram of the original speech sample is obtained, the pseudo-inverse mel filter bank is used to interpolate to the feature dimension of the original amplitude, and the amplitude spectrum is recovered based on the in-situ cepstral convolution recurrent neural network model using a supervised learning method to obtain an estimated amplitude spectrum, so as to realize amplitude spectrum reconstruction.

[0008] In the phase generation stage, based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstral convolution recurrent neural network model, a phase spectrum fitted with the estimated amplitude spectrum is generated in combination with a generative adversarial network, so as to realize the generation of the phase spectrum. The generative adversarial network includes a generator and a discriminator. The generator is used in the inference stage and adopts the architecture of the in-situ cepstral convolution recurrent neural network model. The discriminator is used in the training stage and is composed of a plurality of parallel discriminator modules with the same structure. The output value of each discriminator module is used to drive the adversarial training process of the generator.

[0009] Based on the estimated amplitude spectrum and the phase spectrum, a synthesized speech sample is obtained.

[0010] In an implementation manner, in the amplitude reconstruction stage, a mel spectrogram of the original speech sample is obtained, interpolated to the feature dimension of the original amplitude by using a pseudo-inverse mel filter bank, and the amplitude spectrum is recovered by using a supervised learning manner based on the original cepstrum convolution recurrent neural network model to obtain an estimated amplitude spectrum, so as to realize amplitude spectrum reconstruction, comprising:

[0011] The frequency domain information of the original speech sample is extracted based on a Hanning window to obtain a mel spectrogram of the original speech sample, wherein the dimension of the mel spectrogram is 80.

[0012] The mel spectrogram is interpolated by using a pseudo-inverse mel filter bank to obtain a linear spectrum with the same dimension as the original amplitude, and the dimension of the linear spectrum is 513.

[0013] The linear spectrum is input into the original cepstrum convolution recurrent neural network model, the amplitude spectrum is recovered by using a supervised learning manner, and an estimated amplitude spectrum is output.

[0014] In an implementation manner, the core layer structure of the original cepstrum convolution recurrent neural network model comprises: 5 cepstrum frequency blocks and a frequency domain bidirectional long short-term memory network; wherein the cepstrum frequency block is used to extract the frequency domain local feature of the linear spectrum, and the frequency domain bidirectional long short-term memory network is used to capture the long dependence relationship of the frequency dimension and map the linear spectrum to a high-dimensional space.

[0015] In the amplitude reconstruction stage, the number of channels of the original cepstrum convolution recurrent neural network model is 10, and the Softplus activation function is used in the network output layer.

[0016] In the phase generation stage, the number of channels of the original cepstrum convolution recurrent neural network model is 20, and the phase calculation formula is used in the network output layer.

[0017]

[0018] wherein, the real part is represented by, the imaginary part is represented by, is an arctangent function, is used to process the quadrant discrimination and principal value interval constraint in phase calculation.

[0019] In an implementation manner, the loss of the generator comprises an adversarial loss and a multi-resolution short-time Fourier transform loss, and the loss function of the generator is:

[0020]

[0021] wherein, = 0.8, = 0.2, denotes the generator, denotes the discriminator, denotes real speech samples, denotes synthesized speech samples, is the adversarial loss, is the multi-resolution short-time Fourier transform loss used to guide the gradual generation of phase information matching the estimated magnitude spectrum during the training process.

[0022] In an implementation, the adversarial loss is represented as:

[0023]

[0024] wherein, denotes the mathematical expectation, and Softplus denotes the Softplus activation function;

[0025] The multi-resolution short-time Fourier transform loss is represented as:

[0026]

[0027]

[0028]

[0029] wherein, = 3, used to represent the configuration of the window length and window shift for the short-time Fourier transform, is the frequency dimension, is a very small value used to prevent division by zero errors.

[0030] In an implementation, the loss function of the discriminator is:

[0031]

[0032] wherein, denotes the dynamic attenuation coefficient, denotes the gradient of the discriminator input to the real sample, denotes the gradient of the discriminator input to the generated sample, and denotes two sets of zero-centered gradient penalty terms, denotes the square of the L2 norm.

[0033] In a second aspect, the embodiments of the present application further provide a speech synthesis system of a two-stage neural vocoder, which is used to implement the steps of the speech synthesis method of the two-stage neural vocoder described in the above scheme, the two-stage neural vocoder adopts an in-situ cepstral convolution recurrent neural network model, and the two stages include an amplitude reconstruction stage and a phase generation stage, and the system comprises:

[0034] an amplitude spectrum reconstruction module, which is configured to, in the amplitude reconstruction stage, acquire a mel spectrogram of an original speech sample, interpolate to a feature dimension of an original amplitude by using a pseudo-inverse mel filter bank, and recover an amplitude spectrum based on the in-situ cepstral convolution recurrent neural network model by using a supervised learning manner to obtain an estimated amplitude spectrum, so as to implement amplitude spectrum reconstruction;

[0035] a phase spectrum generation module, which is configured to, in the phase generation stage, generate a phase spectrum fitted with the estimated amplitude spectrum based on the in-situ cepstral convolution recurrent neural network model and a generative adversarial network, so as to implement phase spectrum generation, wherein the generative adversarial network comprises a generator and a discriminator, the generator is used in an inference stage and adopts an architecture of the in-situ cepstral convolution recurrent neural network model, and the discriminator is used in a training stage and is composed of a plurality of parallel discriminator modules with the same structure, and an output value of each discriminator module is used to drive an adversarial training process of the generator;

[0036] a speech synthesis module, which is configured to obtain a synthesized speech sample based on the estimated amplitude spectrum and the phase spectrum.

[0037] In a third aspect, the embodiments of the present application further provide a terminal, wherein the terminal comprises a memory, a processor, and a speech synthesis program of a two-stage neural vocoder stored in the memory and executable on the processor, and the processor implements the steps of the speech synthesis method of the two-stage neural vocoder of any one of the above schemes when executing the speech synthesis program of the two-stage neural vocoder.

[0038] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium stores a speech synthesis program of a two-stage neural vocoder, and the speech synthesis program of the two-stage neural vocoder implements the steps of the speech synthesis method of the two-stage neural vocoder of any one of the above schemes on the computer readable storage medium.

[0039] Beneficial effects: compared with the prior art, the speech synthesis method of the two-stage neural vocoder provided by the application adopts an in-situ cepstrum convolution recurrent neural network model, the two stages include an amplitude reconstruction stage and a phase generation stage, the application first acquires a mel spectrogram of an original speech sample in the amplitude reconstruction stage, interpolates to the feature dimension of the original amplitude by using a pseudo-inverse mel filter bank, and restores the amplitude spectrum based on the in-situ cepstrum convolution recurrent neural network model by using a supervised learning manner to obtain an estimated amplitude spectrum. Then, in the phase generation stage, based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstrum convolution recurrent neural network model, a phase spectrum fitted with the estimated amplitude spectrum is generated by combining a generative adversarial network, so as to realize the generation of the phase spectrum, wherein the generative adversarial network includes a generator and a discriminator, the generator is used in the inference stage and adopts the architecture of the in-situ cepstrum convolution recurrent neural network model, and the discriminator is used in the training stage and is composed of a plurality of parallel discriminator modules with the same structure, and the output value of each discriminator module is used to drive the adversarial training process of the generator. Finally, based on the estimated amplitude spectrum and the phase spectrum, a synthesized speech sample is obtained.

[0040] The application of the in-situ cepstrum convolution recurrent neural network model (ICCRN model) in the amplitude spectrum reconstruction stage and the phase spectrum generation stage, although the ICCRN model is an existing network model, in the generators in the amplitude spectrum reconstruction stage and the phase spectrum generation stage of the application, the network output layer of the ICCRN model is uniquely designed, and the training targets of the two are not the same. In addition, in the phase spectrum generation stage, the application innovatively proposes a generative adversarial network (GAN) manner, which uses the estimated amplitude spectrum to generate a phase spectrum highly fitted with the amplitude. In the discriminator in the phase generation stage, the loss function is redesigned to enable the generator to output high-quality data. The application provides a lightweight and stable amplitude and phase reconstruction mechanism, solves the one-to-many relationship inherent in phase-amplitude, and is beneficial to improving audio fidelity and device compatibility. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The flowchart of the inference stage of the speech synthesis method of the two-stage neural vocoder provided by the embodiment of the application.

[0042] Figure 2 The overall scheme flowchart of the speech synthesis method of the two-stage neural vocoder provided by the embodiment of the application.

[0043] Figure 3 The structure diagram of the in-situ cepstrum convolution recurrent neural network model in the speech synthesis method of the two-stage neural vocoder provided by the embodiment of the application.

[0044] Figure 4 A principle block diagram of a speech synthesis system of a two-stage neural vocoder provided for an embodiment of the present application.

[0045] Figure 5 A principle block diagram of a terminal provided for an embodiment of the present application. DETAILED DESCRIPTION

[0046] For the purpose of making the object, technical solutions and effects of the present application more clear and explicit, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0047] The flowcharts shown in the drawings are only exemplary and do not necessarily include all contents and operations or steps, nor are they necessarily executed in the order described. For example, some operations or steps can be further divided, combined or partially merged, so that the actual execution order may be changed according to the actual situation.

[0048] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clear from the context, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0049] It should be understood that, in order to facilitate clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second" and the like are used to distinguish the same or similar items with basically the same function and effect. For example, the first control information and the second control information are only used to distinguish different control information, and do not limit the order.

[0050] Those skilled in the art can understand that the terms "first", "second" and the like do not limit the quantity and execution order, and the terms "first", "second" and the like do not necessarily mean different.

[0051] It should also be understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0052] Considering the one-to-many relationship between the phase and the amplitude, the traditional method is to use time-domain modeling or to use the label phase to constrain, which does not fundamentally solve the problem. In order to solve the problem of the prior art, the embodiment provides a speech synthesis method of a two-stage neural vocoder, and the core of the embodiment is a two-stage neural vocoder. The two-stage neural vocoder includes an amplitude reconstruction stage and a phase generation stage, and the two-stage neural vocoder adopts an in-situ cepstral convolution recurrent neural network model. Specifically, in the amplitude reconstruction stage, a mel spectrogram of an original speech sample is obtained, a pseudo-inverse mel filter bank is used to interpolate to a feature dimension of an original amplitude, and a supervised learning method is used to recover an amplitude spectrum based on the in-situ cepstral convolution recurrent neural network model (ICCRN model) to obtain an estimated amplitude spectrum. In the phase generation stage, based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstral convolution recurrent neural network model, a phase spectrum that fits the estimated amplitude spectrum is generated by combining a generative adversarial network to realize generation of the phase spectrum. The entire phase generation stage in the embodiment is named PhaseGAN, where Phase represents the phase, and PhaseGAN represents the combination of the phase and the generative adversarial network. This also reflects the innovative application of the generative adversarial network (GAN) in the phase generation stage in the embodiment. In inference, the PhaseGAN generator is used to generate a highly fitted phase spectrum according to the estimated amplitude spectrum, as shown in the second stage: the flow of the phase generation stage in Figure 1 It can be seen that the amplitude reconstruction and the phase generation are decoupled into two branches (i.e., the amplitude reconstruction stage and the phase generation stage) and are processed respectively in the embodiment. Compared with the prior art, the amplitude spectrum is recovered to obtain an estimated amplitude spectrum in the embodiment, and then the estimated amplitude spectrum is used to generate a phase spectrum that is highly fitted to the amplitude, so that efficient and artifact-free waveform synthesis is realized.

[0053] The speech synthesis method of the two-stage neural vocoder in the embodiment can be applied to a terminal. The terminal can be a computer or other intelligent product terminal, as shown in Figure 2 The speech synthesis method of the two-stage neural vocoder includes the following steps:

[0054] In the amplitude reconstruction stage, a mel spectrogram of an original speech sample is obtained, a pseudo-inverse mel filter bank is used to interpolate to a feature dimension of an original amplitude, and a supervised learning method is used to recover an amplitude spectrum based on the in-situ cepstral convolution recurrent neural network model (ICCRN model) to obtain an estimated amplitude spectrum, so as to realize amplitude spectrum reconstruction.

[0055] In specific applications, the speech synthesis method of the two-stage neural vocoder can be combined with Figure 1As shown in FIG. 1, in the first stage of the amplitude reconstruction stage, the embodiment first extracts the frequency domain information of the original speech sample based on the Hanning window to obtain a Mel spectrogram of the original speech sample, wherein the Mel spectrogram has a dimension of 80, which retains the key frequency domain information of the speech. Then, the embodiment performs interpolation processing on the Mel spectrogram based on a pseudo-inverse Mel filter bank to obtain a linear spectrum with the same dimension as the original amplitude, which has a dimension of 513, matches the dimension of the original amplitude spectrum, and ensures that the input and output dimensions of the in-place cepstral convolution recurrent neural network model in the subsequent amplitude reconstruction process are consistent. The pseudo-inverse Mel filter bank is an inverse operation of the Mel filter bank, which is used to recover the original spectrum from the Mel spectrum. In another implementation manner, the embodiment can also randomly crop the audio sample into a fixed length of 16,384 samples at a sampling rate of 22.05 kHz to form training samples with consistent sample length, and use the training samples for model training to avoid unstable model training due to too long or too short samples, and at the same time provide standardized real-time samples for subsequent loss calculation.

[0056] The linear spectrum after interpolation of the Mel spectrogram is only a rough frequency domain feature, and a high-precision estimated amplitude spectrum needs to be generated through model optimization. The estimated amplitude spectrum is the energy skeleton of the speech, which directly determines the frequency energy distribution of the speech (such as the energy difference between vowels and consonants), and its accuracy is the premise of subsequent phase generation. The phase needs to match the energy distribution of the amplitude, otherwise the synthesized speech will be distorted.

[0057] Further, the embodiment adopts an in-place cepstral convolution recurrent neural network model (Inplace Cepstral Convolutional Recurrent Neural Network, ICCRN), and the structure of the in-place cepstral convolution recurrent neural network model is as shown in FIG. 2. Figure 3 Figure 3 [b, c, f, t] in FIG. 2 represents the shape of a tensor, wherein b represents the batch size, c represents the number of channels, f represents the frequency dimension, and t represents the time dimension. The ICCRN model of the embodiment avoids any frequency downsampling or upsampling operation, which reduces information loss, prevents the introduction of extraneous information, and directly avoids the artifact problem caused by the upsampling operation. Therefore, the frequency dimension of the ICCRN model remains unchanged, and the output channel number of all convolution layers is the same. In order to further analyze the time-frequency domain features, the ICCRN model applies cepstral analysis, as shown in FIG. 3. Figure 3 ​The cepstrum frequency block splicing in the ICCRN model is provided with n groups, and the ICCRN model also utilizes LSTM (Long Short-Term Memory) along the frequency axis and the time axis to perform double-path modeling. Figure 3 The generator part in the ICCRN model. In order to ensure the causality of the model, the LSTM along the time axis is unidirectional, which means that the future frames are not used. This cepstrum analysis capability enables the ICCRN model to examine the relationship between the amplitude information and the phase information from the perspective of the harmonic structure.

[0058] Specifically, in the amplitude reconstruction stage, the number of channels c of the ICCRN model is 10, and the core layer structure includes: 5 cepstrum frequency blocks and a frequency domain bidirectional long short-term memory network; wherein the cepstrum frequency block is used to extract the frequency domain local features of the linear spectrum, and the frequency domain bidirectional long short-term memory network is used to capture the long dependence relationship in the frequency dimension and map the linear spectrum to a high-dimensional space. The network output layer of the ICCRN model adopts a Softplus activation function to ensure that the amplitude spectrum value output by the model is non-negative, avoiding the emergence of invalid features that do not conform to the acoustic law. The loss function of the amplitude reconstruction stage in the embodiment is:

[0059]

[0060] wherein, is a mean square error loss, is the number of frames, is the frequency dimension, and are the estimated and target amplitude spectrums, and represent the time and frequency units, respectively.

[0061] When the linear spectrum is input into the ICCRN model, the ICCRN model restores the amplitude spectrum in a supervised learning manner. Through the above loss function, the overall difference between the estimated amplitude spectrum output by the ICCRN model and the true amplitude spectrum can be minimized, and the ICCRN model is forced to learn the energy distribution characteristics of the amplitude spectrum.

[0062] In the phase generation stage, based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstrum convolution recurrent neural network model, a phase spectrum fitted with the estimated amplitude spectrum is generated by combining a generative adversarial network, so as to realize the generation of the phase spectrum, wherein the generative adversarial network includes a generator and a discriminator, the generator is used in the inference stage and adopts the architecture of the in-situ cepstrum convolution recurrent neural network model, and the discriminator is used in the training stage and is composed of a plurality of parallel discriminator modules with the same structure. The output value of each discriminator module is used to drive the adversarial training process of the generator.

[0063] Since the estimated amplitude spectrum only provides frequency domain energy information, the temporal continuity of speech (such as syllable transitions and intonation details) is determined by the phase spectrum. This step uses generator-discriminator adversarial learning to generate a true phase spectrum that highly matches the estimated amplitude spectrum, solving the one-to-many problem of inherent amplitude and phase in traditional methods. Compared to traditional time-frequency (TF) vocoder methods, this embodiment does not use label information about the phase, but uses adversarial generation to make the phase more closely fit the recovered estimated amplitude spectrum.

[0064] In the second phase generation stage, this embodiment innovatively proposes a Generative Adversarial Network (GAN). The GAN generates a phase spectrum that fits the estimated amplitude spectrum. The GAN includes a generator and a discriminator. The generator is used in the inference stage, forming the PhaseGAN generator part in this embodiment, while the discriminator is used in the training stage. Specifically... Figure 3 As shown, the PhaseGAN generator uses an in-situ cepstral convolutional recurrent neural network model (ICCRN model), primarily used to learn the amplitude-phase mapping relationship and generate a phase that matches the amplitude. Specifically, the ICCRN model in the phase generation stage differs from the ICCRN model in the amplitude spectrum reconstruction stage, mainly in the network output layer and model size. In the phase generation stage, the ICCRN model has 20 channels (c=20), and the phase calculation (equivalent to the activation function) formula used in the network output layer is:

[0065]

[0066] in, The Reality Department Indicates the imaginary part. It is the arctangent function. Used for quadrant discrimination and principal value interval constraints when processing phase calculations.

[0067] When the estimated amplitude spectrum is input into the in-situ cepstral convolutional recurrent neural network model, the real and imaginary parts are output in parallel based on the bilinear convolutional layers in the model. Then, based on the real and imaginary parts and a preset phase calculation formula, the phase spectrum is obtained.

[0068] Furthermore, the generator's loss includes adversarial loss and multi-resolution short-time Fourier transform loss, and the generator's loss function is:

[0069]

[0070] in, =0.8, =0.2, Represents a generator. denotes the discriminator, denotes the real speech samples, denotes the synthesized speech samples, is the adversarial loss, is the multi-resolution short-time Fourier transform loss, which is used to guide the gradual generation of phase information matching the estimated amplitude spectrum during the training process. The formula is as follows:

[0071]

[0072] wherein, denotes the mathematical expectation, Softplus denotes the Softplus activation function, and the Softplus function is a smoothed version of the ReLU function, which can still produce a non-zero output when the input is negative, so as to limit the value in the interval greater than 0.

[0073] To enhance the matching degree between the generated phase and the reconstructed amplitude, a multi-resolution short-time Fourier transform (STFT) loss (i.e. ) is introduced into the loss function of the generator. This loss term indirectly constrains the phase gradient, guiding the model to gradually reconstruct the phase information highly matching the estimated amplitude spectrum during the training process. The mathematical expression of is defined as:

[0074]

[0075]

[0076]

[0077] wherein, = 3 denotes the configuration of the window length and window shift of the short-time Fourier transform. In this embodiment, three different time-frequency resolution configuration schemes are adopted, denoted as (window length, frame shift, time-frequency point number), specifically: (400, 80, 512), (800, 200, 1024) and (1600, 400, 2048). is a very small value, which is used to prevent division by zero error.

[0078] Further, the discriminator adopted in this embodiment is composed of a plurality of parallel discriminator modules with the same structure, and each module discriminator contains a three-layer one-dimensional convolution stack. These discriminator modules perform down-sampling on the input speech features through multi-layer convolution operations, and the output directly drives the adversarial training process of the generator. Specifically, the loss function of the discriminator is:

[0079]

[0080] wherein, denotes the dynamic attenuation coefficient, denotes the gradient of the discriminator input with respect to the real sample, denotes the gradient of the discriminator input with respect to the generated sample, and denotes the two sets of zero-centered gradient penalty terms, denotes the square of the L2 norm, i.e., the square of the Euclidean norm.

[0081] Step S300, obtaining the synthesized speech sample based on the estimated amplitude spectrum and the phase spectrum.

[0082] Based on the estimated amplitude spectrum and the phase spectrum obtained in the above steps S100 and S200, the embodiment can combine the estimated amplitude spectrum and the phase spectrum, generate a time-domain speech waveform through inverse short-time Fourier transform, and obtain the synthesized speech sample. The inverse short-time Fourier transform can be represented as:

[0083]

[0084] wherein, is the inverse short-time Fourier transform, is the imaginary unit, satisfying . is the complex exponential form of Euler's formula.

[0085] As can be seen, the estimated amplitude spectrum ( ) in the embodiment provides the energy skeleton of the speech, and the phase spectrum ( ) provides the timing detail modulation of the speech; both are combined through the complex exponential to restore the complex spectrum of the STFT, and then through the inverse iSTFT process, a natural and continuous time-domain speech waveform is finally generated.

[0086] Further, the embodiment also adopts comprehensive objective indicators for system evaluation, which specifically include: the F1 value of the classification of clear and dull sound (V / UV F1, optimal value: 1.0), the greater the F1 value of the classification of clear and dull sound, the better; the periodicity error (Periodicity error, optimal value: 0.0), the smaller the periodicity error, the better; the root mean square error of the fundamental frequency (Pitch RMSE, optimal value: 0.0), the smaller the root mean square error of the fundamental frequency, the better; the root mean square error of the fundamental frequency track (F0 RMSE, optimal value: 0.0), the smaller the root mean square error of the fundamental frequency track, the better; the unified speech quality score (UTMOS, optimal value: 5.0), the greater the unified speech quality score, the better; the wideband perceptual speech quality evaluation (WB-PESQ, optimal value: 4.5), the greater the wideband perceptual speech quality evaluation, the better; the short-time objective intelligibility (STOI, optimal value: 1.0), the greater the short-time objective intelligibility, the better; the mel-cepstral distortion (MCD, optimal value: 0.0), the smaller the mel-cepstral distortion, the better. Details are shown in Tables 1 and 2 and Table 3. Table 1 is the objective evaluation result of the speech synthesis test set in the single speaker scenario. Table 2 is the objective indicator evaluation result of the speech synthesis test set in the unseen speaker scenario. Table 3 is the objective indicator evaluation result for the singing speech data set. The best performance structure is indicated in bold.

[0087] Table 1

[0088]

[0089] Table 2

[0090]

[0091] Table 3

[0092]

[0093] In the above Table 1, Table 2 and Table 3, HiFi-GAN is a vocoder based on a generative adversarial network, iSTFTNet is a vocoder using inverse short-time Fourier transform (iSTFT) for speech waveform synthesis, APNet is a speech synthesis model focusing on joint amplitude and phase modeling, Vocos is a vocoder characterized by efficient computation and high-quality synthesis effect, which is used to generate time-domain speech from a mel-spectrogram. FreeV is a contrastive model for the speech synthesis task, focusing on the flexibility or generalization ability of the model. PhaseGAN represents the two-stage neural vocoder adopted in the embodiment, and PhaseGAN-s is a lightweight variant of PhaseGAN, which reduces the computational complexity or parameter quantity while ensuring performance. Among them, the models marked with an asterisk in the upper right corner are non-causal models, which can use future frame information during inference, and usually have higher sound quality. Domain represents the processing domain of the model, T represents the time domain, and T-F represents the time-frequency domain. #Mac is the abbreviation of Multiply-Accumulate (multiply-accumulate operation), and #Mac represents the number of multiply-accumulate operations required by the model, which is a core indicator for measuring the computational complexity of the model: the larger the value, the higher the computational overhead during inference and training of the model, and the higher the requirement for hardware computing power. #Params is the abbreviation of Parameters (parameter quantity), and #Params represents the parameter quantity (i.e. the total number of trainable parameters in the model) of the model. The parameter quantity reflects the complexity and expressiveness of the model: the larger the parameter quantity, the more complex the patterns that the model can theoretically learn, but the training difficulty and memory occupancy will also increase accordingly. The upward arrow (↑) after the objective indicator indicates that the larger the value, the better the performance, and the downward arrow (↓) indicates that the smaller the value, the better the performance. The two-stage neural vocoder of the embodiment greatly reduces the computational requirement, significantly improves the processing efficiency, and can achieve superior speech synthesis performance with very small parameter quantity. Under the premise of reducing the parameter quantity by 90%, it can still achieve higher unified speech quality scores in unseen speaker scenarios. Moreover, the two-stage neural vocoder of the embodiment can significantly reduce harmonic interference by eliminating the upsampling operation and introducing a cepstrum structure, thereby solving the artifact problem caused by upsampling. The two-stage neural vocoder has the characteristics of high fidelity, low delay and low parameters, and is suitable for speech synthesis related applications, including but not limited to intelligent voice assistants, vehicle voice broadcast systems, virtual digital humans, TTS voice engines, embedded voice terminals (such as smart speakers, wearable devices), etc. It has good deployment adaptability and industrialization potential.

[0094] Based on the above embodiment, the embodiment also provides a speech synthesis system of a two-stage neural vocoder. The system of the embodiment is used to implement the steps of the above method embodiment, and the two-stage neural vocoder adopts an in-situ cepstrum convolution recurrent neural network model, which is specifically as follows: Figure 4As shown, the system includes: an amplitude spectrum reconstruction module 10, a phase spectrum generation module 20, and a speech synthesis module 30. The amplitude spectrum reconstruction module 10 is used to acquire the Mel spectrogram of the original speech sample during the amplitude reconstruction stage, interpolate it to the feature dimension of the original amplitude using a pseudo-inverse Mel filter bank, and recover the amplitude spectrum using supervised learning based on the in-situ cepstral convolutional recurrent neural network model to obtain an estimated amplitude spectrum, thereby achieving amplitude spectrum reconstruction. The phase spectrum generation module 20 is used to generate a phase spectrum that fits the estimated amplitude spectrum based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstral convolutional recurrent neural network model, combined with a generative adversarial network, during the phase generation stage, thereby achieving phase spectrum generation. The generative adversarial network includes a generator and a discriminator. The generator is used in the inference stage and adopts the architecture of the in-situ cepstral convolutional recurrent neural network model. The discriminator is used in the training stage and consists of multiple parallel discriminator modules with identical structures. The output value of each discriminator module is used to drive the adversarial training process of the generator. The speech synthesis module 30 is used to obtain synthesized speech samples based on the estimated amplitude spectrum and the phase spectrum.

[0095] The speech synthesis system of the two-stage neural vocoder in this embodiment is based on the same principle as the steps in the above method embodiments, and will not be repeated here.

[0096] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 5 As shown. The terminal may include one or more processors 100 ( Figure 5 (Only one is shown in the diagram), memory 101, and computer program 102 stored in memory 101 and executable on one or more processors 100. For example, a speech synthesis program for a two-stage neural vocoder. When one or more processors 100 execute computer program 102, they can implement the various steps in the speech synthesis method embodiment of the two-stage neural vocoder. Alternatively, when one or more processors 100 execute computer program 102, they can implement the functions of various modules / units in the speech synthesis system embodiment of the two-stage neural vocoder, without limitation herein.

[0097] In one embodiment, the processor 100 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0098] In one embodiment, the memory 101 can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The memory 101 can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 101 can include both the internal storage unit and the external storage device of the electronic device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0099] Those skilled in the art can understand that, Figure 5 The block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific terminal can include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0100] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, operating database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0101] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A speech synthesis method using a two-stage neural vocoder, characterized in that, The two-stage neural vocoder employs an in-situ cepstral convolutional recurrent neural network model. The two stages include an amplitude reconstruction stage and a phase generation stage. The method includes: In the amplitude reconstruction stage, the Mel spectrogram of the original speech sample is obtained, and the feature dimension of the original amplitude is interpolated using a pseudo-inverse Mel filter bank. Based on the in-situ cepstral convolutional recurrent neural network model, the amplitude spectrum is recovered using supervised learning to obtain the estimated amplitude spectrum, thereby realizing amplitude spectrum reconstruction. During the phase generation stage, based on the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstral convolutional recurrent neural network model, a generative adversarial network is combined to generate a phase spectrum that fits the estimated amplitude spectrum, thereby realizing the generation of the phase spectrum. The generative adversarial network includes a generator and a discriminator. The generator is used in the inference stage and adopts the architecture of the in-situ cepstral convolutional recurrent neural network model. The discriminator is used in the training stage and consists of multiple parallel discriminator modules with the same structure. The output value of each discriminator module is used to drive the adversarial training process of the generator. Based on the estimated amplitude spectrum and the phase spectrum, the synthesized speech sample is obtained; The core layer structure of the in-situ cepstral convolutional recurrent neural network model includes: 5 layers of cepstral frequency blocks and a frequency domain bidirectional long short-term memory network; wherein, the cepstral frequency blocks are used to extract local frequency domain features of the linear spectrum, and the frequency domain bidirectional long short-term memory network is used to capture long-term dependencies in the frequency dimension and map the linear spectrum to a high-dimensional space. During the amplitude reconstruction stage, the in-situ cepstral convolutional recurrent neural network model has 10 channels, and the network output layer uses the Softplus activation function. In the phase generation stage, the in-situ cepstral convolutional recurrent neural network model has 20 channels, and the phase calculation formula used in the network output layer is as follows: in, The Reality Department Indicates the imaginary part. It is the arctangent function. Used for quadrant discrimination and principal value interval constraints when processing phase calculations.

2. The speech synthesis method of the two-stage neural vocoder according to claim 1, characterized in that, In the amplitude reconstruction stage, the Mel spectrogram of the original speech sample is obtained, interpolated to the feature dimension of the original amplitude using a pseudo-inverse Mel filter bank, and the amplitude spectrum is recovered using supervised learning based on the in-situ cepstral convolutional recurrent neural network model to obtain the estimated amplitude spectrum, thereby achieving amplitude spectrum reconstruction, including: Frequency domain information of the original speech sample is extracted based on the Hanning window to obtain the Mel spectrogram of the original speech sample, wherein the Mel spectrogram has 80 dimensions. The Mel spectrum is interpolated based on the pseudo-inverse Mel filter bank to obtain a linear spectrum with the same amplitude dimension as the original spectrum, which has a dimension of 513. The linear spectrum is input into the in-situ cepstral convolutional recurrent neural network model, and the amplitude spectrum is recovered using supervised learning to output the estimated amplitude spectrum.

3. The speech synthesis method of the two-stage neural vocoder according to claim 1, characterized in that, The generator's loss includes adversarial loss and multi-resolution short-time Fourier transform loss. The generator's loss function is: in, =0.8, =0.2, Represents a generator. Indicates the discriminator, Represents real speech samples, This represents a synthesized speech sample. To combat the losses, This is a multi-resolution short-time Fourier transform loss used to guide the gradual generation of phase information that matches the estimated amplitude spectrum during training.

4. The speech synthesis method using a two-stage neural vocoder according to claim 3, characterized in that, Combating losses Represented as: in, Denotes the mathematical expectation, and Softplus denotes the Softplus activation function; Multi-resolution short-time Fourier transform loss Represented as: in, =3, used to indicate the window length and window shift configuration for the short-time Fourier transform. For frequency dimension, It is a very small value used to prevent division by zero errors.

5. The speech synthesis method of the two-stage neural vocoder according to claim 3, characterized in that, The loss function of the discriminator is: in, Indicates the dynamic attenuation coefficient. This represents the gradient of the discriminator with respect to the real sample input. This represents the gradient of the discriminator with respect to the generated sample input. and Represents two sets of zero-center gradient penalty terms. This represents the square of the L2 norm.

6. The speech synthesis method using a two-stage neural vocoder according to claim 1, characterized in that, Based on the estimated amplitude spectrum and the phase spectrum, the synthesized speech sample is obtained, including: The estimated amplitude spectrum is combined with the phase spectrum, and a time-domain speech waveform is generated by inverse short-time Fourier transform to obtain the synthesized speech sample.

7. A speech synthesis system using a two-stage neural vocoder, characterized in that, The system is used to implement the speech synthesis method of the two-stage neural vocoder according to any one of claims 1-6, wherein the two-stage neural vocoder adopts an in-situ cepstral convolutional recurrent neural network model, and the two stages include an amplitude reconstruction stage and a phase generation stage, and the system includes: The amplitude spectrum reconstruction module is used to obtain the Mel spectrogram of the original speech sample in the amplitude reconstruction stage, interpolate it to the feature dimension of the original amplitude using a pseudo-inverse Mel filter bank, and recover the amplitude spectrum using supervised learning based on the in-situ cepstral convolutional recurrent neural network model to obtain the estimated amplitude spectrum, so as to realize amplitude spectrum reconstruction. A phase spectrum generation module is used to generate a phase spectrum that fits the estimated amplitude spectrum obtained in the amplitude reconstruction stage and the in-situ cepstral convolutional recurrent neural network model, combined with a generative adversarial network, during the phase generation stage, so as to realize the generation of the phase spectrum. The generative adversarial network includes a generator and a discriminator. The generator is used in the inference stage and adopts the architecture of the in-situ cepstral convolutional recurrent neural network model. The discriminator is used in the training stage and consists of multiple parallel discriminator modules with the same structure. The output value of each discriminator module is used to drive the adversarial training process of the generator. The speech synthesis module is used to obtain synthesized speech samples based on the estimated amplitude spectrum and the phase spectrum.

8. A terminal, characterized in that, The terminal includes a memory, a processor, and a speech synthesis program for a two-stage neural vocoder stored in the memory and executable on the processor. When the processor executes the speech synthesis program for the two-stage neural vocoder, it implements the steps of the speech synthesis method for the two-stage neural vocoder as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a speech synthesis program for a two-stage neural vocoder, the speech synthesis program for the two-stage neural vocoder implementing the steps of the speech synthesis method for the two-stage neural vocoder as described in any one of claims 1-6 on the computer-readable storage medium.

Images