Speech Synthesis Method and Related Devices, Equipment, and Storage Media

Abstract

The present application discloses a speech synthesis method and related devices, equipment, and storage media. The method includes: extracting a plurality of frame-level acoustic features based on the text to be synthesized; respectively performing predictions based on each frame-level acoustic feature to obtain acoustic parameters corresponding to each frame-level acoustic feature; fusing the excitation parameters, noise parameters, and acoustic parameters corresponding to the frame-level acoustic features to obtain the spectral values corresponding to the frame-level acoustic features; and obtaining the synthesized speech based on the spectral values corresponding to each frame-level acoustic feature. The above solution can improve the efficiency and quality of speech synthesis.

Description

Technical Field

[0001] The present application relates to the technical field of speech synthesis, and particularly to a speech synthesis method, related devices, equipment, and storage media. Background Art

[0002] Speech synthesis is a method of converting text into speech, mainly including a front end, an acoustic model, and a vocoder part. A vocoder is a method of converting speech features such as spectrograms into speech and is an important part of a speech synthesis system. In recent years, with the continuous development of technology, neural network vocoders with high naturalness and high sound quality have emerged one after another. However, the computational complexity of these vocoders is often large, and the synthesis speed is not ideal. Although the vocoders based on traditional signal processing have a fast synthesis speed, their sound quality is not high. In view of this, how to implement an efficient vocoder while ensuring the naturalness and sound quality of speech synthesis is the focus of current research. Summary of the Invention

[0003] The main technical problem to be solved by the present application is to provide a speech synthesis method, related devices, equipment, and storage media, which can improve the efficiency and quality of speech synthesis.

[0004] To solve the above technical problem, in a first aspect of the present application, a speech synthesis method is provided, including: extracting a plurality of frame-level acoustic features based on the text to be synthesized; respectively performing predictions based on each of the frame-level acoustic features to obtain acoustic parameters corresponding to each of the frame-level acoustic features; performing fusion based on excitation parameters, noise parameters, and the acoustic parameters corresponding to the frame-level acoustic features to obtain a spectral value corresponding to the frame-level acoustic features; and obtaining synthesized speech based on the spectral values corresponding to each of the frame-level acoustic features.

[0005] To solve the above technical problem, in a second aspect of the present application, a speech synthesis device is provided, including: a frame-level feature extraction module, an acoustic parameter prediction module, a parameter fusion module, and a sound synthesis module; the frame-level feature extraction module is configured to extract a plurality of frame-level acoustic features based on the text to be synthesized; the acoustic parameter prediction module is configured to respectively perform predictions based on each of the frame-level acoustic features to obtain acoustic parameters corresponding to each of the frame-level acoustic features; the parameter fusion module is configured to perform fusion based on excitation parameters, noise parameters, and the acoustic parameters corresponding to the frame-level acoustic features to obtain a spectral value corresponding to the frame-level acoustic features; and the sound synthesis module is configured to obtain synthesized speech based on the spectral values corresponding to each of the frame-level acoustic features.

[0006] To solve the above technical problem, in a third aspect of the present application, an electronic device is provided, including a memory and a processor coupled to each other. Program instructions are stored in the memory, and the processor is configured to execute the program instructions to implement the speech synthesis method in the first aspect above.

[0007] To solve the above technical problems, a fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be run by a processor, and the program instructions are used to implement the voice synthesis method in the first aspect above.

[0008] In the above solution, by respectively predicting several frame-level acoustic features extracted, acoustic parameters corresponding to each frame-level acoustic feature are obtained, that is, acoustic parameter prediction is performed in units of frames. Compared with acoustic parameter prediction in units of sampling points, the running speed is fast and the number of runs is small, thereby reducing the computational amount and further improving the efficiency of voice synthesis. In addition, based on the excitation parameters, noise parameters, and acoustic parameters corresponding to the frame-level acoustic features for fusion to obtain the spectral values corresponding to the frame-level acoustic features, so the spectral values of the frame-level acoustic features fuse multiple parameters, making the synthetic speech quality obtained based on the spectral values corresponding to each frame-level acoustic feature higher. Description of the Drawings

[0009] Figure 1 is a schematic flowchart of an embodiment of the voice synthesis method provided by the present application;

[0010] Figure 2 is a schematic structural diagram of an embodiment of the vocoder provided by the present application;

[0011] Figure 3 is a schematic flowchart of an embodiment of the prediction of acoustic parameters provided by the present application;

[0012] Figure 4 is a schematic structural diagram of an embodiment of the parameter prediction network provided by the present application;

[0013] Figure 5 is Figure 1 a schematic flowchart of an embodiment of step S13 shown;

[0014] Figure 6 is a schematic flowchart of another embodiment of the voice synthesis method provided by the present application;

[0015] Figure 7 is a schematic structural diagram of an embodiment of the post-filtering module provided by the present application;

[0016] Figure 8 is a schematic flowchart of an embodiment of the joint training provided by the present application;

[0017] Figure 9 is Figure 8 a schematic flowchart of an embodiment of step S86 shown;

[0018] Figure 10 is a schematic framework diagram of an embodiment of the voice synthesis device provided by the present application;

[0019] Figure 11 is a schematic framework diagram of an embodiment of an electronic device provided by this application;

[0020] Figure 12 is a schematic framework diagram of an embodiment of a computer-readable storage medium provided by this application. Detailed implementation manners

[0021] The following will combine with the accompanying drawings of the specification to detail the solutions of the embodiments of this application.

[0022] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures, interfaces, and technologies are presented to thoroughly understand this application.

[0023] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "plurality" in this article means two or more than two.

[0024] Please refer to Figure 1 , Figure 1 is a schematic flowchart of an embodiment of a speech synthesis method provided by this application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 1 the process sequence shown. As Figure 1 shown, this embodiment includes:

[0025] Step S11: Extract several frame-level acoustic features based on the text to be synthesized.

[0026] The method of this embodiment is used to obtain synthesized speech based on the text to be synthesized. The text to be synthesized described in this article can be, but is not limited to, Chinese text, English text, Japanese text, French text, local dialect text, etc. In other implementation manners, the text to be synthesized can also be a text that includes two or more languages at the same time, which is not limited here and can be specifically set according to actual usage needs.

[0027] In one implementation manner, the text to be synthesized can be any text that needs to be synthesized into synthesized speech, and can specifically be obtained from local storage or cloud storage. It can be understood that in other implementation manners, it can also be obtained by collecting the current text through an image acquisition device.

[0028] In this embodiment, several frame-level acoustic features can be extracted based on the text to be synthesized. In a specific embodiment, an existing acoustic model is used to process the text to be synthesized, so as to generate several frame-level acoustic features corresponding to the text to be synthesized. Optionally, the acoustic model includes but is not limited to the FastSpeech (Fast, Robust and Controllable Text to Speech) model, Tacotron model, Deep Voice model, Transform model, etc., and no specific limitation is made here. It can be understood that in other specific embodiments, a training audio is recorded according to the text to be synthesized, and specifically, it can be recorded through a voice acquisition device such as a mobile phone or a microphone in a recording studio, and acoustic feature extraction is performed on the training audio to obtain several frame-level acoustic features. Among them, the algorithm for acoustic feature extraction of the training audio is not limited here and can be specifically set according to actual usage needs.

[0029] In one embodiment, the number of frame-level acoustic features extracted based on the text to be synthesized is not specifically limited and can be set according to the actual text to be recognized. Optionally, the frame-level acoustic features include but are not limited to spectrum, cepstrum, fundamental frequency, aperiodic information, etc., and no specific limitation is made here.

[0030] Step S12: Perform predictions respectively based on each frame-level acoustic feature to obtain acoustic parameters corresponding to each frame-level acoustic feature.

[0031] In this embodiment, predictions are respectively performed based on each frame-level acoustic feature to obtain acoustic parameters corresponding to each frame-level acoustic feature. In one embodiment, the acoustic parameters include complex frequency domain parameters and aperiodic parameters. Among them, the imaginary part of the complex frequency domain parameter represents phase information, and the real part of the complex frequency domain parameter represents amplitude information; the aperiodic parameter represents the ratio between the periodic harmonic component and the aperiodic noise component. It can be understood that in other embodiments, the acoustic parameters may also include other parameters, and no specific limitation is made here.

[0032] In a specific embodiment, the acoustic parameters are complex frequency domain parameters and aperiodic parameters. Specifically, as Figure 2 shown, Figure 2FIG. 0 is a schematic structural diagram of an embodiment of a vocoder provided by the present application. The vocoder includes a complex frequency domain filter. The function of the complex frequency domain filter is to perform prediction processing on frame-level acoustic features to obtain complex frequency domain parameters. The operation scale of the complex frequency domain filter is predicted in units of frames. Compared with prediction in units of sampling points, the operation scale is large and the number of operations is small. Different from the spectral parameters generated by amplitude spectrum information, the output of the complex frequency domain filter is complex frequency domain parameters, which are complex numbers and include not only amplitude information but also phase information. Different from complex cepstrum information, the generated complex frequency domain parameters have higher flexibility. In one embodiment, the function of the complex frequency domain filter is implemented using a parameter prediction network, that is, the complex frequency domain parameters are predicted by the parameter prediction network. It can be understood that in other embodiments, the phase information and amplitude information corresponding to each frame-level acoustic feature can also be obtained by processing each frame-level acoustic feature through signal processing methods. Specifically, taking the phase information as an example, each frame-level acoustic feature is processed and modeled by the minimum phase method to construct the phase information.

[0033] The vocoder further includes an aperiodic parameter prediction module. The function of the aperiodic parameter prediction module is to perform prediction processing on frame-level acoustic features to obtain aperiodic parameters. Among them, it can be dynamically generated using a parameter prediction network, that is, the aperiodic parameters are predicted by the parameter prediction network, which can more accurately predict harmonic and noise components.

[0034] In one embodiment, the parameter prediction network for predicting the complex frequency domain parameters and the parameter prediction network for predicting the aperiodic parameters can be the same parameter prediction network. That is, using this parameter prediction network to predict each frame-level acoustic feature respectively, the complex frequency domain parameters and aperiodic parameters corresponding to each frame-level acoustic feature can be obtained simultaneously. The two share one parameter prediction network, greatly reducing the calculation amount. It can be understood that in other embodiments, the parameter prediction network for predicting the complex frequency domain parameters and the parameter prediction network for predicting the aperiodic parameters can also be different parameter prediction networks, which can be specifically set according to actual usage needs and are not limited here.

[0035] Step S13: Based on the excitation parameters, noise parameters, and acoustic parameters corresponding to the frame-level acoustic features, perform fusion to obtain the spectral value corresponding to the frame-level acoustic features.

[0036] In this embodiment, based on the excitation parameters, noise parameters, and acoustic parameters corresponding to the frame-level acoustic features, perform fusion to obtain the spectral value corresponding to the frame-level acoustic features. That is to say, the obtained spectral value corresponding to the frame-level acoustic features is a fusion of the excitation parameters, noise parameters, and acoustic parameters corresponding to the frame-level acoustic features. In one specific embodiment, such as Figure 2As shown, the vocoder includes a fusion module that fuses based on excitation parameters, noise parameters, and acoustic parameters corresponding to frame-level acoustic features to obtain the spectral values corresponding to the frame-level acoustic features.

[0037] In the embodiments of the present disclosure, parameter fusion is performed based on the pronunciation principle. Specifically, components constituting speech (such as voiced sounds and voiceless sounds) can be modeled separately. Voiceless sounds can include non-periodic noise components in speech. Exemplarily, a voiceless sound signal can be expressed as the convolution of a noise excitation, a vocal tract model, and a radiation model, specifically as follows:

[0038] x(t) = n(t) * v(t) * r(t)

[0039] Where x(t) represents the voiceless sound signal model; n(t) represents the noise model; v(t) represents the vocal tract model; r(t) represents the radiation model.

[0040] Voiced sounds can include periodic harmonic components in speech. A voiced sound signal can be expressed as the convolution of a harmonic excitation, a glottal pulse model, a vocal tract model, and a radiation model, specifically as follows:

[0041] x(t) = e(t) * g(t) * v(t) * r(t)

[0042] Where x(t) represents the voiced sound signal model; e(t) represents the excitation signal model; g(t) represents the glottal pulse model; v(t) represents the vocal tract model; r(t) represents the radiation model.

[0043] Therefore, since the spectral values corresponding to the frame-level acoustic features are obtained by fusing based on the excitation parameters, noise parameters, and acoustic parameters corresponding to the frame-level acoustic features, the quality of the synthesized speech obtained subsequently based on the spectral values corresponding to each frame-level acoustic feature is higher.

[0044] In addition, please continue to refer to Figure 2 , the vocoder may further include an excitation module that is used to optimize the excitation parameters. It should be noted that the excitation parameters may include, but are not limited to, sine functions corresponding to different harmonics of the fundamental frequency. That is, before fusion, in order to further improve the quality of speech synthesis, the excitation parameters can be optimized to optimize different harmonics corresponding to the fundamental frequency.

[0045] Step S14: Obtain the synthesized speech based on the spectral values corresponding to each frame-level acoustic feature respectively.

[0046] In this embodiment, synthetic speech is obtained according to the spectral values respectively corresponding to each frame-level acoustic feature. In one embodiment, the spectral values respectively corresponding to each frame-level acoustic feature are subjected to frame synchronization superposition, so as to obtain synthetic speech. It can be understood that in other embodiments, synthetic speech can also be obtained by performing other processing on the spectral values respectively corresponding to each frame-level acoustic feature, which is not specifically limited herein.

[0047] In a specific embodiment, as Figure 2 shown, the vocoder includes a fusion module, and the fusion module can obtain synthetic speech based on the spectral values respectively corresponding to each frame-level acoustic feature. Exemplarily, the fusion module can be integrated with the PSOLA (Pitch Synchronous Overlap Add) algorithm, so that the spectral values respectively corresponding to each frame-level acoustic feature can be processed by the PSOLA algorithm to obtain synthetic speech. For the specific process of frame synchronization superposition, reference can be made to the technical details of the PSOLA algorithm, which will not be elaborated herein.

[0048] In the above embodiment, by respectively predicting several extracted frame-level acoustic features, the acoustic parameters respectively corresponding to each frame-level acoustic feature are obtained, that is, the acoustic parameter prediction is performed in units of frames. Compared with the acoustic parameter prediction performed in units of sampling points, the running speed is fast and the number of running times is small, thereby reducing the calculation amount and further improving the efficiency of speech synthesis; in addition, based on the excitation parameters, noise parameters and acoustic parameters corresponding to the frame-level acoustic features for fusion to obtain the spectral values corresponding to the frame-level acoustic features, so the spectral values of the frame-level acoustic features fuse multiple parameters, making the synthetic speech obtained based on the spectral values respectively corresponding to each frame-level acoustic feature of higher quality.

[0049] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of an embodiment of the prediction of acoustic parameters provided by this application. It should be noted that if there are substantially the same results, this embodiment is not limited to the Figure 3 shown process sequence. As Figure 3 shown, in this embodiment, the acoustic parameters include complex frequency domain parameters and aperiodic parameters, and the complex frequency domain parameters and aperiodic parameters are predicted based on the same parameter prediction network, specifically including:

[0050] Step S31: Perform feature extraction on the frame-level acoustic feature to obtain a multiplexed feature for predicting the complex frequency domain parameter and the aperiodic parameter.

[0051] In this embodiment, by performing feature extraction on the frame-level acoustic feature, a multiplexed feature for predicting the complex frequency domain parameter and the aperiodic parameter can be obtained. Specifically, the frame-level acoustic feature is input into the parameter prediction network, so as to predict and obtain the multiplexed feature of the complex frequency domain parameter and the aperiodic parameter.

[0052] In a specific embodiment, as Figure 4 shown, Figure 4 FIG. 1 is a schematic structural diagram of an embodiment of a parameter prediction network provided by the present application. The parameter prediction network is built using a multi-layer convolutional structure and an activation function. The parameter prediction network includes a feature extraction sub-network, and the feature extraction sub-network is used to perform feature extraction. Specifically, the frame-level acoustic features are input into the feature extraction sub-network in the parameter prediction network, and the feature extraction sub-network performs feature extraction on the frame-level acoustic features, so as to obtain a multiplexed feature for predicting complex frequency domain parameters and aperiodic parameters.

[0053] Step S32: Perform a first prediction based on the multiplexed feature to obtain complex frequency domain parameters, and perform a second prediction based on the multiplexed feature to obtain aperiodic parameters.

[0054] In this embodiment, a first prediction is performed on the multiplexed feature to obtain complex frequency domain parameters, and at the same time, a second prediction is performed on the multiplexed feature to obtain aperiodic parameters.

[0055] In a specific embodiment, as Figure 4 shown, the parameter prediction network further includes a first prediction sub-network and a second prediction sub-network, that is, the parameter prediction network includes a feature extraction sub-network, a first prediction sub-network, and a second prediction sub-network. The first prediction sub-network and the second prediction sub-network are respectively connected to the feature extraction sub-network. The first prediction sub-network is used to perform the first prediction, and the second prediction sub-network is used to perform the second prediction. Specifically, the multiplexed feature is input into the first prediction sub-network in the parameter prediction network, and the first prediction sub-network performs a first prediction on the multiplexed feature, so as to obtain complex frequency domain parameters, specifically, the imaginary part representing the phase information and the real part representing the amplitude information of the complex frequency domain parameters; the second prediction sub-network in the parameter prediction network performs a second prediction on the multiplexed feature, so as to obtain aperiodic parameters.

[0056] Please refer to Figure 5 , Figure 5 which is Figure 1 a schematic flowchart of an embodiment of step S13 shown in FIG. 2. It should be noted that if there are substantially the same results, this embodiment is not limited to the Figure 5 flow order shown in FIG. 2. As Figure 5 shown, in this embodiment, the acoustic parameters include complex frequency domain parameters and aperiodic parameters, and according to whether the frame-level acoustic feature is a voiceless acoustic feature or a voiced acoustic feature, an excitation parameter or a noise parameter is selectively selected as the parameter to be fused. Specifically, it includes:

[0057] Step S131: Based on the indication identifier of the frame-level acoustic feature, select an excitation parameter or a noise parameter as the parameter to be fused.

[0058] In this embodiment, according to the indication identifier of the frame-level acoustic feature, an excitation parameter or a noise parameter is selected as the parameter to be fused. The indication identifier is used to characterize whether the frame-level acoustic feature is a voiceless acoustic feature or a voiced acoustic feature. That is to say, whether to selectively select an excitation parameter or a noise parameter as the parameter to be fused will depend on whether the frame-level acoustic feature is a voiceless acoustic feature or a voiced acoustic feature.

[0059] In one embodiment, when the indication identifier indicates that the frame-level acoustic feature is a voiced acoustic feature, an excitation parameter is selected as the parameter to be fused.

[0060] In one embodiment, when the indication identifier characterizes that the frame-level acoustic feature is a voiceless acoustic feature, a noise parameter is selected as the parameter to be fused.

[0061] Step S132: Based on the parameter to be fused and the acoustic parameter corresponding to the frame-level acoustic feature, perform fusion to obtain the spectral value corresponding to the frame-level acoustic feature.

[0062] In this embodiment, according to the parameter to be fused and the acoustic parameter corresponding to the frame-level acoustic feature, perform fusion to obtain the spectral value corresponding to the frame-level acoustic feature. In a specific embodiment, the acoustic parameter includes a complex frequency domain parameter and an aperiodicity parameter. According to the parameter to be fused and the complex frequency domain parameter and aperiodicity parameter corresponding to the frame-level acoustic feature, perform fusion to obtain the spectral value corresponding to the frame-level acoustic feature. The specific formula is as follows:

[0063] spg = filter * e(t) * (1 - AP) * uv + filter * n(t) * AP * (1 - uv)

[0064] Among them, spg represents the spectral value corresponding to the frame-level acoustic feature; filter represents the complex frequency domain parameter; e(t) represents the excitation parameter; AP represents the aperiodicity parameter; n(t) represents the noise parameter.

[0065] In one embodiment, when the indication identifier characterizes that the frame-level acoustic feature is a voiceless acoustic feature, convolve the aperiodicity parameter, the complex frequency domain parameter, and the parameter to be fused to obtain the spectral value corresponding to the frame-level acoustic feature. Specifically, when the indication identifier characterizes that the frame-level acoustic feature is a voiceless acoustic feature, the value of uv is 0, and a noise parameter is selected as the parameter to be fused. That is, convolve the aperiodicity parameter, the complex frequency domain parameter, and the noise parameter to obtain the spectral value corresponding to the frame-level acoustic feature. The corresponding formula is as follows:

[0066] spg = filter * n(t) * AP

[0067] In a specific embodiment, n(t) is white noise that satisfies Gaussian noise, and its probability density function follows a Gaussian distribution, specifically as follows:

[0068] n(t)~N(0,1)

[0069] In one embodiment, when the indication flag indicates that the frame-level acoustic feature is a voiced acoustic feature, the difference between a preset value and an aperiodic parameter, a complex frequency domain parameter, and a parameter to be fused are convolved to obtain a spectral value corresponding to the frame-level acoustic feature. Specifically, when the indication flag indicates that the frame-level acoustic feature is a voiced acoustic feature, the value of uv is 1, and the excitation parameter is selected as the parameter to be fused, that is, the difference between the preset value and the aperiodic parameter, the complex frequency domain parameter, and the excitation parameter are convolved to obtain a spectral value corresponding to the frame-level acoustic feature. Among them, the size of the preset value is not limited and can be specifically set according to actual usage needs. In one embodiment, when the aperiodic parameter is in the range of 0-1, the preset value can be set to 1, and the specific formula is as follows:

[0070] spg = filter * e * (1 - AP)

[0071] In one embodiment, before convolving the difference between the preset value and the aperiodic parameter, the complex frequency domain parameter, and the excitation parameter, the excitation parameter can also be optimized, that is, the difference between the preset value and the aperiodic parameter, the complex frequency domain parameter, and the optimized excitation parameter are convolved to obtain a spectral value corresponding to the frame-level acoustic feature, so as to make the synthesized speech quality obtained based on the spectral values corresponding to each frame-level acoustic feature higher.

[0072] Please refer to Figure 6 , Figure 6 which is a schematic flowchart of another embodiment of the speech synthesis method provided by the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to Figure 6 the process sequence shown. As Figure 6 shown, this embodiment includes:

[0073] Step S61: Based on the text to be synthesized, extract several frame-level acoustic features.

[0074] Step S61 is similar to step S11 and will not be elaborated here.

[0075] Step S62: Respectively perform predictions based on each frame-level acoustic feature to obtain acoustic parameters corresponding to each frame-level acoustic feature.

[0076] Step S62 is similar to step S12 and will not be elaborated here.

[0077] Step S63: Based on the excitation parameter, the noise parameter, and the acoustic parameter corresponding to the frame-level acoustic feature, perform fusion to obtain a spectral value corresponding to the frame-level acoustic feature.

[0078] Step S63 is similar to step S13 and will not be elaborated here.

[0079] Step S64: Obtain the synthesized speech based on the spectral values respectively corresponding to the frame-level acoustic features.

[0080] Step S64 is similar to step S14 and will not be elaborated here.

[0081] Step S65: Perform time-domain enhancement on the synthesized speech to obtain the optimized speech.

[0082] In this embodiment, time-domain enhancement will be performed on the synthesized speech to obtain the optimized speech, so as to obtain an audio with higher sound quality and make the synthesized speech closer to the real level of the recording.

[0083] In a specific embodiment, as Figure 2 and Figure 7 shown, Figure 7 is a schematic structural diagram of an embodiment of the post-filtering module provided by the present application. The vocoder further includes a post-filtering module, and the post-filtering module adopts a residual convolution structure, which is based on a multi-layer convolutional residual connection network. The use of the multi-layer convolutional residual connection network can accelerate the processing of time-domain enhancement of the synthesized speech, so as to obtain the optimized speech.

[0084] In one embodiment, since the vocoder is provided with a complex frequency filter module, the parameter quantity of this post-filtering module can achieve a high-quality vocoder system in the case of a very small value. In one embodiment, the post-filtering module has configurability, and this configurable characteristic can meet the configurations of different hardware. Specifically, for low-end hardware devices, the post-filtering module can not be configured, and for high-end hardware devices, the post-filtering module can be configured.

[0085] In one embodiment, the acoustic parameters are predicted by a parameter prediction network, and the optimized speech is processed by a time-domain enhancement network. Moreover, the parameter prediction network and the time-domain enhancement network are obtained through joint training. The joint training of the two can make the training more flexible, realize the simultaneous control of the frequency domain and time domain of the speech, and can obtain a relatively high-quality speech synthesis effect. In a specific embodiment, the time-domain enhancement network is designed based on a residual network.

[0086] In a specific embodiment, as Figure 8 shown, Figure 8 is a schematic flow diagram of an embodiment of the joint training provided by the present application. The joint training of the parameter prediction network and the time-domain enhancement network specifically includes the following sub-steps:

[0087] Step S81: Obtain a number of sample frame-level acoustic features of the sample speech.

[0088] In this embodiment, a number of sample frame-level acoustic features of the sample speech are obtained. The sample speech includes, but is not limited to, Chinese sample speech, English sample speech, Japanese sample speech, etc. In other embodiments, the sample speech may also be speech that includes two or more languages simultaneously, which is not specifically limited herein and can be specifically set according to actual usage needs. Among them, the algorithm for extracting acoustic features from the sample speech is not limited herein and can be specifically set according to actual usage needs. Exemplarily, the sample speech can be pre-recorded, and the sample frame-level acoustic features can be directly extracted from the sample speech through signal processing algorithms; of course, an acoustic model can also be used to generate corresponding sample frame-level acoustic features for the input text, which is not limited herein. In addition, it should be noted that similar to the frame-level acoustic features, the sample frame-level acoustic features may also include, but are not limited to, the following feature information: spectrum, cepstrum, fundamental frequency, aperiodic information, etc., which is not limited herein.

[0089] In one embodiment, the sample speech can specifically be obtained from local storage or cloud storage. It can be understood that in other embodiments, it can also be obtained by collecting the speech of the speaker through a voice collection device.

[0090] Step S82: Based on the parameter prediction network, predict each sample frame-level acoustic feature respectively to obtain the sample acoustic parameters corresponding to each sample frame-level acoustic feature.

[0091] In this embodiment, based on the parameter prediction network, each sample frame-level acoustic feature is predicted respectively to obtain the sample acoustic parameters corresponding to each sample frame-level acoustic feature. That is to say, by using the parameter prediction network to perform prediction processing on each sample frame-level acoustic feature respectively, the sample acoustic parameters corresponding to each sample frame-level acoustic feature can be obtained.

[0092] In one embodiment, the sample acoustic parameters include complex frequency domain parameters and aperiodic parameters. Among them, the imaginary part of the complex frequency domain parameters represents phase information, and the real part of the complex frequency domain parameters represents amplitude information; the aperiodic parameters represent the ratio between the periodic harmonic components and the aperiodic noise components. It can be understood that in other embodiments, the sample acoustic parameters may also include other parameters, which is not specifically limited herein.

[0093] Step S83: Based on the excitation parameters, noise parameters, and the sample acoustic parameters corresponding to the sample frame-level acoustic features, perform fusion to obtain the sample spectral values corresponding to the sample frame-level acoustic features.

[0094] In this embodiment, the acoustic parameters corresponding to the excitation parameters, noise parameters, and sample frame-level acoustic features are fused to obtain the sample spectral values corresponding to the sample frame-level acoustic features. That is to say, the obtained sample spectral values corresponding to the sample frame-level acoustic features are fused with the excitation parameters, noise parameters, and the acoustic parameters corresponding to the sample frame-level acoustic features, so as to make the quality of the sample synthesized speech obtained subsequently based on the sample spectral values corresponding to each sample frame-level acoustic feature higher.

[0095] Step S84: Based on the sample spectral values respectively corresponding to each sample frame-level acoustic feature, obtain the sample synthesized speech.

[0096] In this embodiment, according to the sample spectral values respectively corresponding to each sample frame-level acoustic feature, the sample synthesized speech is obtained. In one embodiment, the sample spectral values respectively corresponding to each sample frame-level acoustic feature are synchronously superimposed frame by frame to obtain the sample synthesized speech. It can be understood that in other embodiments, other processing may also be performed on the sample spectral values respectively corresponding to each sample frame-level acoustic feature to obtain the sample synthesized speech, which is not specifically limited herein.

[0097] Step S85: Based on the time-domain enhancement network, perform time-domain enhancement on the sample synthesized speech to obtain the sample optimized speech.

[0098] In this embodiment, based on the time-domain enhancement network, perform time-domain enhancement on the sample synthesized speech to obtain the sample optimized speech, so as to obtain an audio with higher sound quality and make the sample synthesized speech closer to the real level of the recording. That is to say, the time-domain enhancement network will be used to perform time-domain enhancement processing on the sample synthesized speech to obtain the sample optimized speech with better sound quality.

[0099] Step S86: Based on the Mel spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively, jointly adjust the network parameters of both the parameter prediction network and the time-domain enhancement network.

[0100] In this embodiment, according to the Mel spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively, jointly adjust the network parameters of both the parameter speech network and the time-domain enhancement network. That is to say, in this embodiment, the network parameters of the parameter prediction network and the time-domain enhancement network are adjusted from multiple dimensions of the speech time-frequency domain, so that the parameter prediction network and the time-domain enhancement network after parameter adjustment have better effects, that is, the quality of the synthesized speech generated subsequently based on the text to be synthesized is higher. Among them, the specific formula for jointly adjusting the network parameters of both the parameter prediction network and the time-domain enhancement network is as follows:

[0101] L = L mel1 + L mel2 + L melf

[0102] Among them, L mel1 represents the Mel-spectrum loss value between the sample synthesized speech and the sample speech; L mel2 represents the Mel-spectrum loss value between the sample optimized speech and the sample speech; L melf represents the Mel-spectrum loss value between the sample acoustic parameters and the sample speech. Specifically, first, the Mel-spectrum of the sample speech, the Mel-spectrum of the sample synthesized speech, the Mel-spectrum of the sample optimized speech, and the Mel-spectrum of the sample acoustic parameters are calculated respectively; then, the difference between the Mel-spectrum of the sample synthesized speech and the Mel-spectrum of the sample speech, that is, L mel1 , the difference between the Mel-spectrum of the sample optimized speech and the Mel-spectrum of the sample speech, that is, L mel2 , and the difference between the Mel-spectrum of the sample acoustic parameters and the Mel-spectrum of the sample speech, that is, L melf are calculated.

[0103] In one embodiment, before jointly adjusting the network parameters of both the parameter prediction network and the time-domain enhancement network, the authenticity of the sample synthesized speech and the sample optimized speech are respectively identified to obtain the first probability of the sample synthesized speech and the second probability of the sample optimized speech. That is to say, a discriminant network is introduced into the intermediate parameters of the parameter prediction network and the time-domain enhancement network, realizing multi-dimensional discrimination of parameters such as time-domain values and frequency-domain values in the intermediate process; in addition, through multi-dimensional constraints in the frequency domain and time domain, the learning difficulty of the parameter prediction network and the time-domain enhancement network is reduced, enabling the parameter prediction network and the time-domain enhancement network to achieve better results. In a specific embodiment, the first discriminator network D1 can be used to identify the authenticity of the sample synthesized speech to obtain the first probability of the sample synthesized speech; the second discriminator network D2 can be used to identify the authenticity of the sample optimized speech to obtain the second probability of the sample optimized speech; both the first discriminator network D1 and the second discriminator network D2 can adopt discriminators with multiple resolutions, where the resolution is not limited and can be specifically set according to actual usage needs.

[0104] Among them, the first probability represents the possibility that the sample synthesized speech is true and / or false, and the second probability represents the possibility that the sample optimized speech is true and / or false.

[0105] In a specific embodiment, as Figure 9 shown, Figure 9 is Figure 8 a schematic flowchart of an embodiment of step S86 shown. After obtaining the first probability that the sample synthesized speech is true and / or false and the second probability that the sample optimized speech is true and / or false, jointly adjusting the network parameters of both the parameter prediction network and the time-domain enhancement network specifically includes the following sub-steps:

[0106] Step S861: Obtain a first loss based on the Mel-spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively.

[0107] In this embodiment, a first loss is obtained according to the Mel-spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively. The specific formula is as follows:

[0108] L1 = L mel1 + L mel2 + L melf

[0109] Wherein, L1 represents the first loss; L mel1 represents the Mel-spectrum loss value between the sample synthesized speech and the sample speech; L mel2 represents the Mel-spectrum loss value between the sample optimized speech and the sample speech; L melf represents the Mel-spectrum loss value between the sample acoustic parameters and the sample speech. Specifically, first calculate the Mel-spectrum of the sample speech, the Mel-spectrum of the sample synthesized speech, the Mel-spectrum of the sample optimized speech, and the Mel-spectrum of the sample acoustic parameters respectively; then calculate the difference between the Mel-spectrum of the sample synthesized speech and the Mel-spectrum of the sample speech, i.e., L mel1 , the difference between the Mel-spectrum of the sample optimized speech and the Mel-spectrum of the sample speech, i.e., L mel2 , and the difference between the Mel-spectrum of the sample acoustic parameters and the Mel-spectrum of the sample speech, i.e., L melf ; according to the difference between the Mel-spectrum of the sample synthesized speech and the Mel-spectrum of the sample speech, i.e., L mel1 , the difference between the Mel-spectrum of the sample optimized speech and the Mel-spectrum of the sample speech, i.e., L mel2 and the difference between the Mel-spectrum of the sample acoustic parameters and the Mel-spectrum of the sample speech, i.e., L melf , obtain the first loss L1.

[0110] Step S862: Obtain a second loss based on the differences between the first probability of the sample synthesized speech and the second probability of the sample optimized speech and a preset probability respectively.

[0111] In this embodiment, a second loss is obtained according to the differences between the first probability of the sample synthesized speech and the second probability of the sample optimized speech and a preset probability respectively. Among them, the preset probability indicates that the possibilities of being true and being false are the same. The specific formula is as follows:

[0112] L2 = L gan (D1)+ L gan (D2)

[0113] Wherein, L2 represents the second loss; L gan (D1) represents the loss value between the first probability of the sample synthesized speech and the preset probability; Lgan (D2) represents the loss value between the second probability of the sample optimized speech and the preset probability.

[0114] Step S863: Based on the first loss and the second loss, jointly adjust the network parameters of both the parameter prediction network and the time domain enhancement network.

[0115] In this embodiment, according to the first loss and the second loss, the network parameters of both the parameter prediction network and the time domain enhancement network are jointly adjusted. By introducing a discriminant network into the intermediate parameters of the parameter prediction network and the time domain enhancement network, multi-dimensional discrimination of parameters such as time domain values and frequency domain values in the intermediate process is achieved; in addition, through multi-dimensional constraints in the frequency domain and time domain, the learning difficulty of the parameter prediction network and the time domain enhancement network is reduced, enabling the parameter prediction network and the time domain enhancement network to achieve better results. Among them, the specific formula for jointly adjusting the network parameters of both the parameter prediction network and the time domain enhancement network is as follows:

[0116] L = L2 + L1 = L gan (D1) + L gan (D2) + L mel1 + L mel2 + L melf

[0117] Among them, L represents the total loss; L1 represents the first loss; L2 represents the second loss; L gan (D1) represents the loss value between the first probability of the sample synthesized speech and the preset probability; L gan (D2) represents the loss value between the second probability of the sample optimized speech and the preset probability; L mel1 represents the Mel spectrum loss value between the sample synthesized speech and the sample speech; L mel2 represents the Mel spectrum loss value between the sample optimized speech and the sample speech; L melf represents the Mel spectrum loss value between the sample acoustic parameters and the sample speech.

[0118] Please refer to Figure 10 , Figure 10 is a schematic framework diagram of an embodiment of the speech synthesis device provided by this application. The speech synthesis device 100 includes a frame-level feature extraction module 101, an acoustic parameter prediction module 102, a parameter fusion module 103, and a voice synthesis module 104; the frame-level feature extraction module 101 is used to extract several frame-level acoustic features based on the text to be synthesized; the acoustic parameter prediction module 102 is used to make predictions respectively based on each frame-level acoustic feature to obtain the acoustic parameters corresponding to each frame-level acoustic feature; the parameter fusion module 103 is used to fuse based on the excitation parameters, noise parameters, and the acoustic parameters corresponding to the frame-level acoustic features to obtain the spectral values corresponding to the frame-level acoustic features; the voice synthesis module 104 is used to obtain the synthesized speech based on the spectral values corresponding to each frame-level acoustic feature.

[0119] Among them, the above acoustic parameters include complex frequency domain parameters and aperiodic parameters, and the complex frequency domain parameters and aperiodic parameters are predicted by a parameter prediction network; among them, the real part of the complex frequency domain parameters represents amplitude information, the imaginary part of the complex frequency domain parameters represents phase information, and the aperiodic parameters represent the ratio between the periodic harmonic components and the aperiodic noise components.

[0120] Among them, the voice synthesis device 100 further includes a prediction module 105, and the prediction module 105 is used for: extracting features from the frame-level acoustic features to obtain a multiplexed feature for predicting the complex frequency domain parameters and aperiodic parameters; performing a first prediction based on the multiplexed feature to obtain the complex frequency domain parameters, and performing a second prediction based on the multiplexed feature to obtain the aperiodic parameters.

[0121] Among them, the above parameter prediction network includes a feature extraction sub-network, a first prediction sub-network, and a second prediction sub-network. The first prediction sub-network and the second prediction sub-network are respectively connected to the feature extraction sub-network, and the feature extraction sub-network is used to perform feature extraction, the first prediction sub-network is used to perform the first prediction, and the second prediction sub-network is used to perform the second prediction.

[0122] Among them, the parameter fusion module 103 is used to fuse the excitation parameters, noise parameters, and the acoustic parameters corresponding to the frame-level acoustic features to obtain the spectral value corresponding to the frame-level acoustic features, specifically including: based on the indication identifier of the frame-level acoustic features, selecting the excitation parameters or noise parameters as the parameters to be fused; among them, the indication identifier is used to characterize whether the frame-level acoustic features are voiceless acoustic features or voiced acoustic features; fusing the parameters to be fused and the acoustic parameters corresponding to the frame-level acoustic features to obtain the spectral value corresponding to the frame-level acoustic features.

[0123] Among them, the parameter fusion module 103 is used to select the excitation parameters or noise parameters as the parameters to be fused based on the indication identifier of the frame-level acoustic features, specifically including: in the case where the indication identifier characterizes the frame-level acoustic features as voiceless acoustic features, selecting the noise parameters as the parameters to be fused; and / or, in the case where the indication identifier represents the frame-level acoustic features as voiced acoustic features, selecting the excitation parameters as the parameters to be fused.

[0124] Among them, the above acoustic parameters include complex frequency domain parameters and aperiodic parameters; the parameter fusion module 103 is used to fuse based on the parameters to be fused and the acoustic parameters corresponding to the frame-level acoustic features to obtain the spectral values corresponding to the frame-level acoustic features, specifically including: when the indication identifier represents that the frame-level acoustic feature is a voiceless acoustic feature, convolving the aperiodic parameter, the complex frequency domain parameter, and the parameter to be fused to obtain the spectral value corresponding to the frame-level acoustic feature; and / or when the indication identifier represents that the frame-level acoustic feature is a voiced acoustic feature, convolving the difference between the preset value and the aperiodic parameter, the complex frequency domain parameter, and the parameter to be fused to obtain the spectral value corresponding to the frame-level acoustic feature.

[0125] Among them, the speech synthesis device 100 further includes a time domain enhancement module 106, and the time domain enhancement module 106 is used to, after obtaining the synthesized speech based on the spectral values corresponding to each frame-level acoustic feature, specifically including: performing time domain enhancement on the synthesized speech to obtain an optimized speech.

[0126] Among them, the above acoustic parameters are predicted by a parameter prediction network, the optimized speech is processed by a time domain enhancement network, and the parameter prediction network and the time domain enhancement network are obtained through joint training.

[0127] Among them, the speech synthesis device 100 further includes a joint training module 107, and the steps for the joint training module 107 to perform joint training include: obtaining a plurality of sample frame-level acoustic features of the sample speech; respectively predicting each sample frame-level acoustic feature based on the parameter prediction network to obtain the sample acoustic parameters corresponding to each sample frame-level acoustic feature; fusing based on the excitation parameter, the noise parameter, and the sample acoustic parameters corresponding to the sample frame-level acoustic feature to obtain the sample spectral values corresponding to the sample frame-level acoustic features; obtaining the sample synthesized speech based on the sample spectral values corresponding to each sample frame-level acoustic feature; performing time domain enhancement on the sample synthesized speech based on the time domain enhancement network to obtain the sample optimized speech; jointly adjusting the network parameters of both the parameter prediction network and the time domain enhancement network based on the Mel spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech.

[0128] Among them, the joint training module 107 is configured to, before jointly adjusting the network parameters of the parameter prediction network and the time-domain enhancement network based on the Mel spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech, specifically include: respectively performing true / false discrimination on the sample synthesized speech and the sample optimized speech to obtain a first probability of the sample synthesized speech and a second probability of the sample optimized speech; where the first probability represents the possibility that the sample synthesized speech is true and / or false, and the second probability represents the possibility that the sample optimized speech is true and / or false; the joint training module 107 is configured to jointly adjust the network parameters of the parameter prediction network and the time-domain enhancement network based on the Mel spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech, specifically include: obtaining a first loss based on the Mel spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech; and obtaining a second loss based on the differences between the first probability of the sample synthesized speech and the second probability of the sample optimized speech and a preset probability respectively; where the preset probability represents that the possibilities of being true and false are the same; jointly adjusting the network parameters of the parameter prediction network and the time-domain enhancement network based on the first loss and the second loss.

[0129] Please refer to Figure 11 , Figure 11 which is a schematic framework diagram of an embodiment of the electronic device provided in this application. The electronic device 110 includes a mutually coupled memory 111 and a processor 112. Program instructions are stored in the memory 111, and the processor 112 is configured to execute the program instructions to implement the steps in any of the above speech synthesis method embodiments. Specifically, the electronic device 110 may include, but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited herein.

[0130] Specifically, the processor 112 is used to control itself and the memory 111 to implement the steps in any of the above-described embodiments of the speech synthesis method. The processor 112 may also be referred to as a CPU (Central Processing Unit). The processor 112 may be an integrated circuit chip with the ability to process signals. The processor 112 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 112 may be implemented jointly by integrated circuit chips.

[0131] Please refer to Figure 12 , Figure 12 which is a schematic framework diagram of an embodiment of the computer-readable storage medium provided by this application. The computer-readable storage medium 120 stores program instructions 121 that can be run by a processor, and the program instructions 121 are used to implement the steps in any of the above-described embodiments of the speech synthesis method.

[0132] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0133] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. Their similarities or similarities can be referred to each other. For the sake of brevity, they will not be repeated in this article.

[0134] In several embodiments provided by this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0135] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that makes a contribution to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0138] The above is only the embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A voice synthesis method, characterized in that, Including: Extracting a plurality of frame-level acoustic features based on the text to be synthesized; Performing predictions respectively based on each of the frame-level acoustic features to obtain acoustic parameters corresponding to each of the frame-level acoustic features; Fusing based on the excitation parameter, the noise parameter, and the acoustic parameter corresponding to the frame-level acoustic feature to obtain the spectral value corresponding to the frame-level acoustic feature; Obtaining the synthesized speech based on the spectral values corresponding to each of the frame-level acoustic features; Wherein, the fusing based on the excitation parameter, the noise parameter, and the acoustic parameter corresponding to the frame-level acoustic feature to obtain the spectral value corresponding to the frame-level acoustic feature includes: Selecting the excitation parameter or the noise parameter as the parameter to be fused based on the indication identifier of the frame-level acoustic feature; wherein, the indication identifier is used to represent whether the frame-level acoustic feature is a voiceless acoustic feature or a voiced acoustic feature; Fusing based on the parameter to be fused and the acoustic parameter corresponding to the frame-level acoustic feature to obtain the spectral value corresponding to the frame-level acoustic feature.

2. The method according to claim 1, wherein The acoustic parameter includes a complex frequency domain parameter and an aperiodic parameter, and the complex frequency domain parameter and the aperiodic parameter are predicted by a parameter prediction network; Wherein, the real part of the complex frequency domain parameter represents amplitude information, the imaginary part of the complex frequency domain parameter represents phase information, and the aperiodic parameter represents the ratio between the periodic harmonic component and the aperiodic noise component.

3. The method according to claim 2, wherein The prediction steps of the acoustic parameter include: Performing feature extraction on the frame-level acoustic feature to obtain a multiplexed feature for predicting the complex frequency domain parameter and the aperiodic parameter; Performing a first prediction based on the multiplexed feature to obtain the complex frequency domain parameter, and performing a second prediction based on the multiplexed feature to obtain the aperiodic parameter.

4. The method according to claim 3, wherein The parameter prediction network includes a feature extraction sub-network, a first prediction sub-network, and a second prediction sub-network. The first prediction sub-network and the second prediction sub-network are respectively connected to the feature extraction sub-network, and the feature extraction sub-network is used to perform the feature extraction, the first prediction sub-network is used to perform the first prediction, and the second prediction sub-network is used to perform the second prediction.

5. The method according to claim 1, wherein The selecting the excitation parameter or the noise parameter as the parameter to be fused based on the indication identifier of the frame-level acoustic feature includes: When the indication identifier represents that the frame-level acoustic feature is the voiceless acoustic feature, selecting the noise parameter as the parameter to be fused; And / or, when the indication identifier represents that the frame-level acoustic feature is the voiced acoustic feature, selecting the excitation parameter as the parameter to be fused.

6. The method according to claim 1, wherein The acoustic parameter includes a complex frequency domain parameter and an aperiodic parameter; the fusing based on the parameter to be fused and the acoustic parameter corresponding to the frame-level acoustic feature to obtain the spectral value corresponding to the frame-level acoustic feature includes: When the indication identifier represents that the frame-level acoustic feature is the voiceless acoustic feature, convolving the aperiodic parameter, the complex frequency domain parameter, and the parameter to be fused to obtain the spectral value corresponding to the frame-level acoustic feature; And / or, when the indication identifier indicates that the frame-level acoustic feature is the voiced acoustic feature, convolve the difference between the preset value and the aperiodic parameter, the complex frequency domain parameter, and the parameter to be fused to obtain the spectral value corresponding to the frame-level acoustic feature.

7. The method according to claim 1, wherein After obtaining the synthesized speech based on the spectral values respectively corresponding to the frame-level acoustic features, the method further includes: Performing time-domain enhancement on the synthesized speech to obtain an optimized speech.

8. The method according to claim 7, characterized in that, The acoustic parameters are predicted by a parameter prediction network, the optimized speech is processed by a time-domain enhancement network, and the parameter prediction network and the time-domain enhancement network are obtained through joint training.

9. The method according to claim 8, wherein The steps of the joint training include: Obtaining a plurality of sample frame-level acoustic features of the sample speech; Based on the parameter prediction network, respectively predicting each of the sample frame-level acoustic features to obtain the sample acoustic parameters corresponding to each of the sample frame-level acoustic features; Fusing the excitation parameter, the noise parameter, and the sample acoustic parameters corresponding to the sample frame-level acoustic features to obtain the sample spectral values corresponding to the sample frame-level acoustic features; Obtaining a sample synthesized speech based on the sample spectral values respectively corresponding to the sample frame-level acoustic features; Based on the time-domain enhancement network, performing time-domain enhancement on the sample synthesized speech to obtain a sample optimized speech; Based on the Mel-spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively, jointly adjusting the network parameters of the parameter prediction network and the time-domain enhancement network.

10. The method according to claim 9, wherein Before jointly adjusting the network parameters of the parameter prediction network and the time-domain enhancement network based on the Mel-spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively, the method further includes: Respectively performing true / false discrimination on the sample synthesized speech and the sample optimized speech to obtain a first probability of the sample synthesized speech and a second probability of the sample optimized speech; wherein, the first probability represents the possibility that the sample synthesized speech is true and / or false, and the second probability represents the possibility that the sample optimized speech is true and / or false; The jointly adjusting the network parameters of the parameter prediction network and the time-domain enhancement network based on the Mel-spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively includes: Obtaining a first loss based on the Mel-spectrum differences between the sample synthesized speech, the sample optimized speech, and the sample acoustic parameters and the sample speech respectively; and Obtaining a second loss based on the differences between the first probability of the sample synthesized speech and the second probability of the sample optimized speech and a preset probability respectively; wherein, the preset probability represents that the possibilities of being true and false are the same; Based on the first loss and the second loss, jointly adjusting the network parameters of the parameter prediction network and the time-domain enhancement network.

11. A voice synthesis device, characterized in that, Including: A frame-level feature extraction module, configured to extract a plurality of frame-level acoustic features based on the text to be synthesized; An acoustic parameter prediction module, configured to perform predictions respectively based on each of the frame-level acoustic features to obtain acoustic parameters corresponding to each of the frame-level acoustic features; A parameter fusion module, configured to fuse based on an excitation parameter, a noise parameter, and the acoustic parameter corresponding to the frame-level acoustic feature to obtain a spectral value corresponding to the frame-level acoustic feature; A voice synthesis module, configured to obtain a synthesized speech based on the spectral values respectively corresponding to each of the frame-level acoustic features; Wherein, the fusing based on the excitation parameter, the noise parameter, and the acoustic parameter corresponding to the frame-level acoustic feature to obtain the spectral value corresponding to the frame-level acoustic feature includes: Selecting the excitation parameter or the noise parameter as a parameter to be fused based on an indication identifier of the frame-level acoustic feature; wherein, the indication identifier is used to represent that the frame-level acoustic feature is a voiceless acoustic feature or a voiced acoustic feature; Fusing based on the parameter to be fused and the acoustic parameter corresponding to the frame-level acoustic feature to obtain the spectral value corresponding to the frame-level acoustic feature.

12. An electronic device, characterized in that, Comprising a memory and a processor coupled to each other, wherein program instructions are stored in the memory, and the processor is configured to execute the program instructions to implement the voice synthesis method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, Stored with program instructions capable of being run by a processor, the program instructions being used to implement the voice synthesis method according to any one of claims 1 to 10.

Images