Speech synthesis method, apparatus, device, and storage medium

Abstract

The application provides a speech synthesis method and device, equipment and storage medium, and relates to the technical field of speech synthesis. The method comprises the following steps: obtaining a mel spectrogram corresponding to a target text; generating a conditional audio according to the mel spectrogram and a random phase spectrum corresponding to the mel spectrogram; and generating target audio corresponding to the target text by using a preset vocoder according to the conditional audio and the mel spectrogram. Compared with the prior art, the problem of artifacts in the audio generated by the vocoder is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a speech synthesis method and device, equipment and storage medium. BACKGROUND

[0002] A vocoder is a technology widely used in the fields of speech synthesis, sound conversion and speech translation, etc., and its main function is to convert acoustic features, such as mel-spectrogram

[0003] into the final waveform. The vocoder based on neural network is called neural vocoder.

[0004] With the development of neural network, the generation quality of neural vocoder has been significantly improved. First, the neural vocoder based on autoregressive model (AR), such as autoregressive neural vocoder, but since this kind of vocoder has a faster inference speed, it needs a large number of neural network parameters for training, which is not suitable for practical application; therefore, the neural vocoder based on generative adversarial networks (GAN) is proposed to solve the above problems, which not only has a faster inference speed, but also uses a relatively small number of network parameters, which meets the practical application scenario.

[0005] Although these neural vocoders based on GAN have many advantages, in practical application, there are various types of artifacts in the generated audio, such as spectral discontinuity, pitch error, electric sound and other problems.

[0006] Although these neural vocoders based on GAN have many advantages, in practical application, there are various types of artifacts in the generated audio, such as spectral discontinuity, pitch error, electric sound and other problems. SUMMARY

[0007] The present application aims at the deficiencies in the prior art, and provides a speech synthesis method, device, equipment and storage medium to solve the problem of artifacts in the audio generated by the vocoder in the prior art.

[0008] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0009] In a first aspect, an embodiment of the present application provides a speech synthesis method, which comprises:

[0010] obtaining a mel-spectrogram corresponding to a target text;

[0011] generating a conditional audio according to the mel-spectrogram and a random phase spectrum corresponding to the mel-spectrogram;

[0012] The preset vocoder is used to generate target audio corresponding to the target text according to the condition audio and the mel-spectrogram.

[0013] In a second aspect, another embodiment of the present application provides a speech synthesis device, the device comprising: an acquisition module and a generation module, wherein:

[0014] The acquisition module is configured to acquire a mel-spectrogram corresponding to a target text.

[0015] The generation module is configured to generate condition audio according to the mel-spectrogram and a random phase spectrum corresponding to the mel-spectrogram, and use a preset vocoder to generate target audio corresponding to the target text according to the condition audio and the mel-spectrogram.

[0016] In a third aspect, another embodiment of the present application provides a speech synthesis device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, and the processor and the storage medium communicate through the bus when the speech synthesis device is running, and the processor executes the machine-readable instructions to perform the steps of any of the methods of the first aspect.

[0017] In a fourth aspect, another embodiment of the present application provides a storage medium, the storage medium storing a computer program, and the computer program is executed by the processor to perform the steps of any of the methods of the first aspect.

[0018] The speech synthesis method provided by the present application has the following beneficial effects: after acquiring a mel-spectrogram corresponding to a target text, the method generates condition audio according to the mel-spectrogram and a random phase spectrum corresponding to the mel-spectrogram, and then inputs the condition audio and the mel-spectrogram into a preset vocoder, wherein the condition audio is randomly generated without introducing an additional network to predict, and after obtaining the random phase spectrum, the target audio corresponding to the target text can be generated by the preset vocoder according to the condition audio and the mel-spectrogram. Since the random phase spectrum is obtained according to the mel-spectrogram, the condition audio obtained according to the random phase spectrum and the mel-spectrogram can not only provide accurate amplitude information but also provide accurate phase information, thereby reducing artifacts and improving the audio quality of the target audio generated by the preset vocoder. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0020] Figure 1 The flowchart of the speech synthesis method provided by an embodiment of the present application is shown in the figure.

[0021] Figure 2 The flowchart of the speech synthesis method provided by another embodiment of the present application is shown in the figure.

[0022] Figure 3 The flowchart of the speech synthesis method provided by another embodiment of the present application is shown in the figure.

[0023] Figure 4 The flowchart of the speech synthesis method provided by another embodiment of the present application is shown in the figure.

[0024] Figure 5 The structural diagram of the speech synthesis device provided by an embodiment of the present application is shown in the figure.

[0025] Figure 6 The structural diagram of the speech synthesis device provided by another embodiment of the present application is shown in the figure.

[0026] Figure 7 The structural diagram of the speech synthesis device provided by another embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments.

[0028] The components of the embodiments of the present application generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of the present application.

[0029] In addition, the flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in order, and steps that have no logical context relationship can be reversed in order or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0030] For the convenience of understanding the present application, the following explains some of the terms involved in the present application:

[0031] Vocoder: a speech analysis and synthesis system of a certain model of speech signal, mainly used for digital telephone communication, especially for secure telephone communication. In transmission, only model parameters are used, and in coding and decoding, speech signal coding and decoding are used. A coder and decoder for analyzing and synthesizing voice, also known as a voice analysis and synthesis system or a voice frequency band compression system. It is a powerful tool for compressing communication frequency band and conducting secure communication.

[0032] In the neural homomorphic vocoder (NHV): NHV includes a neural network to control the linear time-varying (LTV) filter in the source filter model. Similar to the harmonic plus noise model, NHV generates harmonic and noise components separately. Harmonic component: contains periodic vibration of sound production, modeled by a sequence of pulses filtered by LTV. Noise component: includes background noise, unvoiced sound and random components in voiced sound, modeled by LTV filtered noise.

[0033] Unified source-filter GAN based on source filter model: a unified source-filter GAN. In the proposed neural vocoder, the vocoder takes sinusoidal signals as additional input, and pitch-dependent dilated convolution and spectral envelope regularization loss have been implemented to decompose the entire network into source network and filter network.

[0034] RefineGAN: uses the F0 generated speech template as an additional conditional input in addition to the Mel-spectrogram to ensure the correctness and quality of the generated speech pitch.

[0035] In the prior art, in order to improve the reasoning speed, a small number of network parameters are used, and in the speech synthesis process, a vocoder based on a generative adversarial network (GAN vocoder) is generally used for audio prediction, and the neural vocoder based on GAN usually takes the mel amplitude spectrum as the input, but in actual application, the audio generated by them will have various types of artifacts, such as spectral discontinuity, pitch error or electric sound and the like.

[0036] In order to improve the quality of the generated audio and reduce the occurrence of artifacts, researchers have begun to consider adding the fundamental frequency information as the input condition of the vocoder, so that the neural vocoder acts as a source filter model to improve the generation quality of the neural vocoder and reduce the occurrence of artifacts. Neural Homomorphic Vocoder, Unified Source-Filter GAN, RefineGAN and the like are typical representatives.

[0037] Although these models relatively improve the quality of the neural vocoder based on GAN, they also bring new problems, that is, in the speech synthesis task, the acoustic model usually only predicts the mel amplitude spectrum, and the frame-level fundamental frequency (F0) is usually difficult to predict and has certain errors, so when the fundamental frequency with errors is introduced as the conditional input, it will also cause the quality of these models to decrease, that is, the prior art cannot guarantee the accuracy of the predicted fundamental frequency, so as to guarantee the accuracy of the obtained audio, resulting in the problem of artifacts in the generated audio.

[0038] Speech synthesis is generally a process of converting target text into target audio, and in the conversion process, the target text information is usually first converted into acoustic feature information, and the mel spectrogram is one of the acoustic feature information. After the mel spectrogram corresponding to the target text information is obtained, the target audio is obtained by restoring the mel spectrogram frame by frame, which is the step of speech synthesis, wherein the process of restoring the mel spectrogram is the working process of the vocoder. The following focuses on a detailed description of the process of restoring the mel spectrogram. The target audio restored by the speech synthesis method provided in the present application has better accuracy and no artifact problem.

[0039] The speech synthesis method provided by the embodiment of the present application is explained and described as follows in combination with a plurality of specific application examples. Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present application is shown in Figure 1 The method comprises the following steps.

[0040] S101: Obtain the mel spectrogram corresponding to the target text.

[0041] In an embodiment of the present application, the manner of obtaining the mel-spectrogram may be, for example, that a preset speech synthesis model is adopted to generate a mel-spectrogram corresponding to the target text according to the target text corresponding to the speaker.

[0042] In another embodiment of the present application, the manner of obtaining the mel-spectrogram may also be that a preset speech synthesis model is adopted to generate a mel-spectrogram corresponding to the target text under the emotional label according to the target text and the emotional label corresponding to the speaker, wherein it should be understood that for the target text with the same content, the generated mel-spectrogram may be different when the emotional label is different, for example, for the target text with the same content, the corresponding mel-spectrogram is different when the emotional label is happy and the emotional label is angry, such a setting manner makes the subsequent synthesized target audio more emotional and more expressive because of the guidance of the emotional label when generating the mel-spectrogram corresponding to the target text, for example, for the emotional label of happy, the subsequently generated target audio may be more active; when the emotional label is angry, the subsequently generated target audio may be more low and serious, it should be understood that the above embodiment is only an example, and the specific manner of obtaining the mel-spectrogram can be flexibly adjusted according to the user's needs, and is not limited to the above embodiment.

[0043] In some possible embodiments, the manner of determining the emotional label may be, for example, pre-input by the user; or, may also be determined after recognizing the target text corresponding to the speaker, for example, for the target text of "I am really happy to see you today", the recognized and determined emotional label may be happy; if the target text is "you do this really make me disappointed", the recognized and determined emotional label may be sad, it should be understood that the above embodiment is only an example, and the specific manner of obtaining or determining the emotional label can be flexibly adjusted according to the user's needs, and is not limited to the above embodiment.

[0044] S102: generating a conditional audio according to the mel-spectrogram and the random phase spectrum corresponding to the mel-spectrogram.

[0045] In an embodiment of the present application, the conditional audio may include, for example, pitch information, voice information and target speaker information.

[0046] Since the present application does not introduce any network prediction when generating the conditional audio, but directly determines a random phase spectrum based on the mel-spectrogram, and directly generates based on the mel-spectrogram and the random phase spectrum corresponding to the mel-spectrogram, the complexity of generating the conditional audio in the present application is low and the efficiency is high.

[0047] And, in the embodiments of the present application, the generated conditional audio has audio key information required for subsequent prediction, that is, the conditional audio generated by the method provided in the embodiments of the present application includes information consistent with the mel-spectrogram, and there is no prediction error, so that the accuracy of subsequent prediction can be improved.

[0048] S103: generating target audio corresponding to the target text by using a preset vocoder according to the conditional audio and the mel-spectrogram.

[0049] In the embodiments of the present application, the selection of the preset vocoder may, for example, be a generative adversarial network-based vocoder (GAN Vocoder), which may, for example, be a neural homomorphic vocoder (NHV), a unified neural vocoder framework based on a source filter model (Unified Source-Filter GAN), a mainstream neural vocoder HIFIGAN, or RefineGAN, etc. It should be understood that the above embodiments are only exemplary descriptions, and the selection of the specific preset vocoder is not limited to the above embodiments. Any input including a mel-spectrogram, and the input can be a multi-input vocoder can use the method provided in the present application. The selection of the specific preset vocoder can be flexibly adjusted according to user needs, and is not limited to the above embodiments.

[0050] The audio with the above pitch information, voice information, and target speaker information is input into the preset vocoder as the conditional audio, so that the conditional audio can provide accurate fundamental frequency information, voice information, and speaker information for the preset vocoder, thereby making the quality of the target audio finally predicted by the preset vocoder higher and reducing the occurrence of artifacts.

[0051] Using the speech synthesis method provided in the present application, after obtaining the mel-spectrogram corresponding to the target text, the conditional audio is generated according to the mel-spectrogram and the corresponding random phase spectrum. Then, the conditional audio and the mel-spectrogram are jointly input into the preset vocoder. Since the generation of the conditional audio does not require an additional network to be introduced for prediction, it is randomly generated. After obtaining the random phase spectrum, the target audio corresponding to the target text can be generated by the preset vocoder according to the conditional audio and the mel-spectrogram. Such a way can reduce artifacts and improve the audio quality of the target audio generated by the preset vocoder, because the random phase spectrum is obtained according to the mel-spectrogram, so that the conditional audio obtained according to the random phase spectrum and the mel-spectrogram can provide accurate amplitude information and accurate phase information.

[0052] Optionally, based on the above-mentioned embodiments, the embodiments of the present application can further provide a speech synthesis method, and the implementation process of the method will be exemplarily described below in combination with the accompanying drawings. Figure 2 As shown in the flowchart of a speech synthesis method provided by another embodiment of the present application, Figure 2 Before S102, the method can further include:

[0053] S111: converting the mel-spectrogram into a linear amplitude spectrum.

[0054] S112: generating a random phase spectrum according to the linear amplitude spectrum.

[0055] In the embodiments of the present application, the random phase spectrum is generated in the following manner: a random phase spectrum with the same resolution as the linear amplitude spectrum is randomly generated, and the random phase spectrum is generated in a main manner, which only needs to ensure that the frequency of the generated random phase spectrum is the same as that of the linear amplitude spectrum, that is, the amplitude of the random phase spectrum is randomly generated, and the amplitude of the random phase spectrum is not limited in the present application.

[0056] In the embodiments of the present application, the random phase spectrum can be generated in the following manner: a random phase spectrum generated by Gaussian distribution; or a random phase spectrum obtained by randomly sampling uniformly on Gaussian distribution; it should be understood that the above-mentioned embodiments are only exemplarily described, and the manner of generating the random phase spectrum can be flexibly adjusted according to the needs of users, and is not limited to the above-mentioned embodiments.

[0057] Optionally, based on the above-mentioned embodiments, the embodiments of the present application can further provide a speech synthesis method, and the implementation process of generating the conditional audio in the method will be exemplarily described below in combination with the accompanying drawings. Figure 3 As shown in the flowchart of a speech synthesis method provided by another embodiment of the present application, Figure 3 S102 can include:

[0058] S121: generating the conditional audio according to the linear amplitude spectrum and the random phase spectrum.

[0059] In the embodiments of the present application, the random phase spectrum with the same resolution as the linear amplitude spectrum is generated according to the linear amplitude spectrum, that is, a phase spectrum with a random phase but the same resolution as the linear amplitude spectrum is generated.

[0060] In the embodiments of the present application, the way of generating the conditional audio may be, for example, inverse Fourier transform (IFT) is performed on the linear amplitude spectrum and the random phase spectrum to obtain the conditional audio. That is, in the embodiments of the present application, the linear amplitude spectrum and the random phase spectrum are collectively taken as the input of the inverse Fourier transform, and after the inverse Fourier transform is performed on them, the output obtained is the conditional audio.

[0061] In an embodiment of the present application, the inverse Fourier transform used may be, for example, an inverse Short Time Fourier transform (ISTFT). It should be understood that the above embodiment is only an example, and the specific inverse Fourier transform may be flexibly adjusted according to user needs, and is not limited to the above embodiment.

[0062] Since the conditional audio is generated in the present application, the generation of the conditional audio does not depend on any prediction network or prediction algorithm or prediction model, but after the mel spectrogram is converted into the linear amplitude spectrum, a random phase spectrum with the same resolution as the linear amplitude spectrum and random phase is randomly generated. This way of randomly obtaining the random phase spectrum does not require additional operations or prediction when obtaining the random phase spectrum, but only needs to directly generate and obtain a random phase spectrum with the same resolution as the linear amplitude spectrum. In the embodiments of the present application, there is no requirement for the amplitude of the randomly generated random phase spectrum.

[0063] Subsequently, the conditional audio is generated according to the random phase spectrum and the linear amplitude spectrum. This way of determining the conditional audio is very convenient, and in the embodiments of the present application, the amplitude information and the phase information can be directly determined according to the generated conditional audio and the mel spectrogram.

[0064] The conditional audio obtained in this way has correct pitch information, voice information and speaker information although the phase is random. That is, the conditional audio obtained in this way has all the information of the linear amplitude spectrum and the random phase spectrum, and the process of obtaining the conditional audio does not involve network prediction, and the complexity is only twice matrix multiplication, and the time consumed is very short. That is, the efficiency of generating the conditional audio is very high by using the method provided in the present application. In the process of predicting the target audio, the randomly generated conditional audio is taken as an input of the preset vocoder, and is input into the preset vocoder together with the mel spectrogram to obtain the target audio, which can also reduce the occurrence of artifacts when the preset vocoder generates the target audio, and improve the audio quality of the target audio obtained by the preset vocoder.

[0065] Figure 4 A flowchart of a speech synthesis method provided for another embodiment of the present application is shown in FIG. 2. Figure 4As shown, the speech synthesis method provided in the present application has the following completion process:

[0066] The random phase spectrogram and the Mel-spectrogram are jointly used as the input of the inverse Fourier transform, and the condition audio is output after the inverse Fourier transform.

[0067] The generation process of the random phase spectrogram is as follows: first, the Mel-spectrogram is converted into a linear amplitude spectrum, and then a random phase spectrogram with the same resolution as the linear amplitude spectrum and random phase is randomly generated according to the resolution of the linear amplitude spectrum. The random phase spectrogram obtained in this way has correct pitch information, voice information and speaker information, so that the information included in the condition audio obtained according to the random phase spectrogram and the linear amplitude spectrum is consistent with the Mel-spectrogram. Therefore, there is no problem of prediction error in the subsequent prediction process of the target audio.

[0068] Compared with the prior art, the prediction method of the target audio provided in the present application not only does not need to introduce additional network prediction, but also the condition audio can provide accurate information for subsequent speech prediction, thereby improving the accuracy of subsequent speech prediction and avoiding the problems of prediction error and artifacts in the predicted audio.

[0069] Subsequently, the condition audio and the Mel-spectrogram are input into the preset vocoder as inputs of the preset vocoder, and the preset vocoder restores the audio according to the input condition audio and the Mel-spectrogram. The final target audio corresponding to the target text is obtained after restoration. Since the condition audio with correct pitch information, voice information and speaker information is introduced in the process of obtaining the target audio, the accuracy of the target audio obtained by the processing method of the present application is higher than that of the prior art.

[0070] In the embodiments of the present application, the preset vocoder used in the present application can be taken as an example of the current mainstream neural vocoder HIFIGAN. Since the condition audio is introduced as a new condition feature in the process of determining the target audio corresponding to the target text in the embodiments of the present application, a new module is added to the preset vocoder of the present application in order to process the new condition feature.

[0071] Specifically, the preset vocoder in the present application can be composed of a generator and multiple discriminators, wherein the generator and the discriminators are obtained after adversarial training.

[0072] In the embodiments of the present application, the generator in the preset vocoder can be composed of two parts: one part of the generator can have a structure such as an original HIFIGAN structure, in which the mel-spectrogram is used as the conditional input, and a one-layer convolutional layer is used to encode the mel-spectrogram into a hidden layer, and then the encoded mel-spectrogram is input into the generator network, each layer in the generator network first performs up-sampling on the feature, and then uses a residual block-based module to encode the up-sampled part, and then inputs it to the next layer until the output audio is obtained. The residual block is composed of several groups of convolutional layers with the same convolution kernel size and different dilation coefficient sizes, the outputs of the convolutional layers in different groups are linked by residual, and the convolution kernel sizes and dilation coefficients of different residual blocks are different. In the embodiments of the present application, the hyperbolic tangent activation function Tanh is used before the audio output to limit the numerical range to 0-1, and in addition, the activation function Leaky Relu is set between adjacent convolutional layers with different dilation coefficient sizes to adjust the zero gradient problem of the negative value, thereby stabilizing the training of the preset vocoder GAN Vocoder.

[0073] The other part of the generator uses the conditional audio generated by the mel-spectrogram and the random phase spectrum corresponding to the mel-spectrogram as the conditional input, and the conditional audio is input into a plurality of down-sampling networks with the same structure as the up-sampling network. The output of each layer of the down-sampling network is spliced with the output of the up-sampling network, and the down-sampling network and the up-sampling network are in a symmetrical form, so that in the training of the entire preset vocoder GAN Vocoder, the generated feature at each resolution level is guided by the conditional audio, and the accuracy of the target audio generated in the final generation in terms of pitch, UV and speaker information is ensured.

[0074] In the embodiments of the present application, the loss function in the preset vocoder includes two parts: the loss function of the generator and the loss function of the discriminator.

[0075] The loss function part of the generator is the mean square error (MSE) of the mel-spectrogram of the real audio and the predicted audio.

[0076] The loss function of the discriminator part, in addition to the multi-scale and multi-period discriminators of the original HIFIGAN, in the embodiments of the present application, a multi-frequency discriminator is additionally added to discriminate the effect of the generated audio in the high frequency; specifically, the discriminator first performs Fourier transform on the input generated audio, converts the features of the generated audio to the frequency domain, and then discriminates the frequency domain features to determine the effect in the high frequency. That is, in the training part of the generative adversarial network (GAN), the least squares generative adversarial network (LSGAN) is used as the training target in the embodiments of the present application, and a feature matching loss function is added to improve the stability of the training. These two training targets are widely used in the training of GAN models.

[0077] For the proportion setting of each loss function, in the embodiments of the present application, for example, the proportion coefficient corresponding to the multi-frequency discriminator loss function (Feature matching loss) can be 2, that is, multiplying 2 on the numerical value of the original discriminator loss value, and the proportion coefficient corresponding to the Mel loss function (Mel loss) can be 45, that is, multiplying 45 on the numerical value of the original Mel loss function. Finally, each loss function adjusted according to the preset proportion coefficient is used to adjust the preset vocoder. Such proportion can balance the proportion between the generator loss function and the discriminator loss function of the preset vocoder, thereby stabilizing the entire training process of the preset vocoder.

[0078] By using the speech synthesis method provided in the present application, since a random phase spectrum with the same resolution as the linear amplitude spectrum converted from the Mel spectrogram but with random phase is randomly generated before generating the target audio, and the linear amplitude spectrum and the random phase spectrum are restored to obtain the conditional audio by using inverse Fourier transform, and then the conditional audio and the Mel spectrogram are simultaneously input into the preset vocoder as conditions, the quality of the final target audio is improved. The speech synthesis method mentioned in the present application can improve the audio quality of the final target audio and reduce the occurrence of artifacts when applied to any existing GAN-based neural decoder.

[0079] The speech synthesis device provided in the present application is explained and described below in conjunction with the accompanying drawings. The speech synthesis device can perform the above Figures 1-4 Any speech synthesis method, its specific implementation and beneficial effects are referred to the above, and will not be described below.

[0080] Figure 5A structural schematic diagram of a speech synthesis device provided by an embodiment of the present application is shown in FIG. 1, which includes an acquisition module 201 and a generation module 202, wherein: Figure 5

[0081] The acquisition module 201 is configured to acquire a mel-spectrogram corresponding to a target text.

[0082] The generation module 202 is configured to generate a conditional audio according to the mel-spectrogram and a random phase spectrum corresponding to the mel-spectrogram, and generate a target audio corresponding to the target text according to the conditional audio and the mel-spectrogram by using a preset vocoder.

[0083] Optionally, based on the above-mentioned embodiments, an embodiment of the present application can further provide a speech synthesis device, and the implementation process of the device given by the above-mentioned embodiments will be exemplarily described below in combination with the accompanying drawings. Figure 5 Figure 6 A structural schematic diagram of a speech synthesis device provided by another embodiment of the present application is shown in FIG. 2, which further includes a conversion module 203 configured to convert the mel-spectrogram into a linear amplitude spectrum. Figure 6

[0084] The generation module 202 is specifically configured to generate the random phase spectrum according to the linear amplitude spectrum.

[0085] Optionally, the generation module 202 is specifically configured to generate the conditional audio according to the linear amplitude spectrum and the random phase spectrum.

[0086] Optionally, the generation module 202 is specifically configured to perform inverse Fourier transform on the linear amplitude spectrum and the random phase spectrum to obtain the conditional audio.

[0087] Optionally, the generation module 202 is specifically configured to generate the random phase spectrum with the same resolution as the linear amplitude spectrum according to the linear amplitude spectrum.

[0088] Optionally, the generation module 202 is specifically configured to generate the mel-spectrogram corresponding to the target text according to the target text by using a preset speech synthesis model.

[0089] Optionally, the conditional audio includes pitch information, voice information and target speaker information.

[0090] The above-mentioned device is used to execute the method provided by the above-mentioned embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0091] ​​​The above modules can be one or more integrated circuits configured to implement the above methods, for example, one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of a processing element scheduling code, the processing element can be a general purpose processor, such as a central processing unit (CPU) or other processor that can invoke program code. For another example, the modules can be integrated together to implement a system-on-a-chip (SOC).

[0092] Figure 7 A structural diagram of a speech synthesis device provided by an embodiment of the present application is provided, which can be integrated into a terminal device or a chip of a terminal device.

[0093] As shown in Figure 7 , the speech synthesis device includes a processor 501, a bus 502, and a storage medium 503.

[0094] The processor 501 is configured to store a program, and the processor 501 invokes the program stored in the storage medium 503 to execute the above Figures 1-4 corresponding method embodiments. The specific implementation manners and technical effects are similar, and will not be described here.

[0095] Optionally, the present application also provides a program product, for example, a storage medium, which stores a computer program including a program that, when executed by a processor, performs the above method corresponding embodiments.

[0096] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely schematic, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0097] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0098] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0099] The integrated unit realized in the form of software functional unit can be stored in a computer readable storage medium. The software functional unit stored in a storage medium includes a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated as: ROM), random access memory (English: Random Access Memory, abbreviated as: RAM), magnetic disk or optical disk and various program code storage media.

Claims

1. A speech synthesis method characterized by, The method comprises: obtaining a mel-spectrogram corresponding to a target text; generating a conditional audio according to the mel-spectrogram and a random phase spectrum corresponding to the mel-spectrogram; generating a target audio corresponding to the target text according to the conditional audio and the mel-spectrogram by using a preset vocoder; Before the generating of the conditional audio according to the mel-spectrogram and the random phase spectrum corresponding to the mel-spectrogram, the method further comprises: converting the mel-spectrogram into a linear amplitude spectrum; generating a random phase spectrum with the same resolution as the linear amplitude spectrum according to the linear amplitude spectrum.

2. The method of claim 1, wherein, The generating of the conditional audio according to the mel-spectrogram and the random phase spectrum corresponding to the mel-spectrogram comprises: generating the conditional audio according to the linear amplitude spectrum and the random phase spectrum.

3. The method of claim 2, wherein, The generating of the conditional audio according to the mel-spectrogram and the random phase spectrum corresponding to the mel-spectrogram comprises: performing inverse Fourier transform on the linear amplitude spectrum and the random phase spectrum to obtain the conditional audio.

4. The method of claim 1, wherein, Before the obtaining of the mel-spectrogram corresponding to the target text, the method further comprises: generating the mel-spectrogram corresponding to the target text according to the target text by using a preset speech synthesis model.

5. The method of claim 1, wherein, The conditional audio comprises pitch information, voice information and target speaker information.

6. A speech synthesis apparatus characterized by comprising: The device comprises an obtaining module and a generating module, wherein: The obtaining module is configured to obtain a mel-spectrogram corresponding to a target text; The generating module is configured to generate a conditional audio according to the mel-spectrogram and a random phase spectrum corresponding to the mel-spectrogram; and generate a target audio corresponding to the target text according to the conditional audio and the mel-spectrogram by using a preset vocoder. The device further comprises a converting module configured to convert the mel-spectrogram into a linear amplitude spectrum; The generating module is further configured to generate a random phase spectrum with the same resolution as the linear amplitude spectrum according to the linear amplitude spectrum.

7. A speech synthesis device characterized by comprising: The device comprises a processor, a storage medium and a bus, the storage medium stores machine readable instructions executable by the processor, when the speech synthesis device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine readable instructions to execute the method of any one of claims 1-5.

8. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed by the processor to execute the method of any one of claims 1-5.

Images