Speech synthesis generation method, electronic equipment and storage medium

Abstract

The invention discloses a speech synthesis generation method, electronic equipment and a storage medium. The method comprises the following steps: acquiring an initial voice feature and an initial rhythm feature corresponding to initial voice data; splicing the initial speech features and the initial rhythm features to obtain an initial object to be subjected to noise addition; performing noise addition on the initial object to be subjected to noise addition to obtain a noise-added object; the noise adding object and a phoneme sequence corresponding to the initial voice data are input into a diffusion model, so that the noise adding object is denoised, a target object is obtained, and the target object comprises a combination of target voice features and target rhythm features; and obtaining target voice data corresponding to the target object. According to the invention, the naturalness and expressive force of voice generation are improved.

Description

Technical Field

[0001] The embodiments disclosed in the present application relate to the field of artificial intelligence technology, and more specifically, to a speech synthesis generation method, an electronic device and a storage medium. Background Art

[0002] In recent years, speech synthesis technology has made significant progress, and high expressiveness and high naturalness have increasingly become the focus of speech synthesis research. In speech synthesis, prosodic features are the key to speech expressiveness and naturalness. At present, various prosodic features are usually predicted based on text information, and then the acoustic features are reconstructed using prosodic features and text information. Among them, there is still a certain difference between the predicted prosodic feature distribution and the actual prosodic feature, which affects the effect of speech synthesis. Summary of the invention

[0003] According to the embodiments of the present application, the present application proposes a speech synthesis generation method, an electronic device and a storage medium to solve the above-mentioned problems.

[0004] The first aspect of the present application discloses a speech synthesis generation method, comprising: obtaining initial speech features and initial prosodic features corresponding to initial speech data; concatenating the initial speech features and the initial prosodic features to obtain an initial object to be noisy; adding noise to the initial object to be noisy to obtain a noisy object; inputting the noisy object and the phoneme sequence corresponding to the initial speech data into a diffusion model to denoise the noisy object to obtain a target object, wherein the target object includes a combination of a target speech feature and a target prosodic feature; and obtaining target speech data corresponding to the target object.

[0005] In some embodiments, the step of adding noise to the initial object to be noisied to obtain the noisied object comprises: inputting the initial object to be noisied into a noise scheduler to obtain the noisied object, wherein the noise intensity of the initial object to be noisied at each noisy time step is different and gradually increases to 1, so that the noisy object is noisied with a specific proportion of noise.

[0006] In some embodiments, the diffusion model includes a preprocessing network and a denoising network; the phoneme sequence corresponding to the noisy object and the initial speech data is input into the diffusion model to denoise the noisy object to obtain a target object, including: inputting the phoneme sequence corresponding to the noisy object and the initial speech data into the preprocessing network to obtain a feature time series, wherein the feature time series includes a combination of text features corresponding to the phoneme sequence and noisy features corresponding to the noisy object; inputting the feature time series into the denoising network to perform denoising to obtain the target object.

[0007] In some embodiments, the preprocessing network includes a phoneme encoder and a preprocessing sub-network; the phoneme sequence corresponding to the noisy object and the initial speech data is input into the preprocessing network to obtain a feature time series, including: inputting the phoneme sequence into the phoneme encoder to obtain the text feature; inputting the noisy object into the preprocessing sub-network to obtain the noisy feature; and concatenating the text feature and the noisy feature to obtain the feature time series.

[0008] In some embodiments, the phoneme encoder includes an embedding layer and a Transformer layer; the phoneme sequence is input into the phoneme encoder to obtain the text feature, including: using the embedding layer to encode the identifier of the phoneme sequence to obtain a phoneme embedding feature; using the Transformer layer to process the phoneme embedding feature to obtain the text feature.

[0009] In some embodiments, the preprocessing subnetwork includes a convolution layer and a normalization layer; the noisy object is input into the preprocessing subnetwork to obtain the noisy feature, including: using the convolution layer and the normalization layer to process the noisy object to obtain the noisy feature, wherein the noisy feature has the same dimension as the text feature.

[0010] In some embodiments, the characteristic length of the initial speech feature is obtained; wherein the time length of the initial object to be noisy is the sum of the characteristic length of the initial speech feature and the characteristic length of the initial prosodic feature.

[0011] In some embodiments, obtaining the characteristic length of the initial speech feature includes: inputting the phoneme sequence corresponding to the initial speech data into a mel-spectrogram prediction model to obtain the mel-spectrogram feature corresponding to the phoneme sequence to determine the characteristic length of the initial speech feature.

[0012] In some embodiments, the prosodic feature corresponding to the initial speech data includes at least one of an energy feature, a duration feature, and a pitch feature, and the prosodic feature is extracted based on a phoneme sequence corresponding to the initial speech data.

[0013] The second aspect of the present application discloses an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech synthesis generation method described in the first aspect.

[0014] The third aspect of the present application discloses a non-volatile computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the speech synthesis generation method described in the first aspect is implemented.

[0015] The beneficial effects of the present application are: obtaining initial speech features and initial prosodic features corresponding to initial speech data, splicing them to obtain an initial object to be noisy, obtaining a noisy object by noisy the initial object to be noisy, and inputting the phoneme sequence corresponding to the noisy object and the initial speech data into a diffusion model to denoise the noisy object to obtain a target object, wherein the target object includes a combination of target speech features and target prosodic features, and further, obtaining target speech data corresponding to the target object, thereby improving the naturalness and expressiveness of speech generation in the process of collaboratively constructing language features and prosodic features. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present application will be further described below in conjunction with the accompanying drawings and implementation methods, in which:

[0017] Figure 1 It is a flowchart of a speech synthesis generation method according to an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a speech synthesis generation method according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0020] Figure 4 It is a schematic diagram of the structure of the non-volatile computer-readable storage medium of an embodiment of the present application. DETAILED DESCRIPTION

[0021] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, and can mean including any one or more elements selected from the set consisting of A, B, and C. In addition, the terms "first", "second", and "third" in this application are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0023] Currently, many speech synthesis methods model the rhythm of speech and usually rely on text input when predicting the prosodic features of speech. The predicted prosodic features are often only strongly correlated with the text features, and the individual prosodic features and the correlation between prosodic features and speech representations are not fully learned, which in turn limits the naturalness and expressiveness of the speech synthesis effect.

[0024] Therefore, the present application provides a speech synthesis generation method, an electronic device and a storage medium to improve the naturalness and expressiveness of the speech synthesis effect.

[0025] In order to enable those skilled in the art to better understand the technical solution of the present application, the technical solution of the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0026] See also Figure 1 , Figure 1 The method can be performed by an electronic device with computing functions, such as a microcomputer, a server, and a mobile device such as a notebook computer or a tablet computer.

[0027] It should be noted that if there are substantially the same results, the method of the present application is not limited to Figure 1 The process sequence shown is limited.

[0028] In some possible implementations, the method may be implemented by a processor calling computer-readable instructions stored in a memory, such as Figure 1 As shown, the method may include the following steps:

[0029] S11: Acquire initial speech features and initial prosodic features corresponding to the initial speech data.

[0030] The initial speech data is processed to obtain initial speech features and initial prosodic features corresponding to the initial speech data. The initial speech data may be text, and the initial speech data is processed, for example, the text is converted into a phoneme sequence y, and then the corresponding initial speech feature V0 and initial prosodic feature R0 are obtained based on the phoneme sequence y. Phoneme refers to the basic sound unit in pronunciation, and the initial speech feature V0 can be used to characterize the audio signal corresponding to the phoneme sequence, and the initial prosodic feature R0 can be used to characterize the duration, pitch, and energy corresponding to each phoneme in the phoneme sequence y.

[0031] S12: concatenate the initial speech features and the initial prosodic features to obtain an initial object to be noisy.

[0032] The initial speech feature and the initial prosodic feature are concatenated, for example, the initial speech feature V0 and the initial prosodic feature R0 are concatenated in time sequence, so as to obtain an initial object to be noisy X0, and the initial object to be noisy X0 includes {V0, R0}.

[0033] S13: Adding noise to the initial object to be noisy to obtain a noisy object.

[0034] Noise is added to the initial object to be noisy. For example, at each time step, noise is added to the initial object to be noisy X0, so that the initial object to be noisy tends to be completely randomized, and the noisy object X0 is obtained. t , where the noise object X t Can include {V t , R t},V t represents the initial speech feature after adding noise, R t Represents the initial rhythmic features after adding noise.

[0035] S14: Inputting the phoneme sequence corresponding to the noisy object and the initial speech data into the diffusion model to denoise the noisy object to obtain a target object, wherein the target object includes a combination of a target speech feature and a target prosodic feature.

[0036] The phoneme sequence corresponding to the noise object and the initial speech data is input into the diffusion model, that is, the phoneme sequence corresponding to the initial speech data is used as the control condition, and the diffusion model is used to t In some examples, the target object X0′ includes a combination of a target speech feature and a target prosodic feature, for example, based on the noise-added object X0′. tand phoneme sequence y, and use the diffusion model to predict the target speech feature V′ and the target prosodic feature R′ corresponding to the initial speech data, wherein the target speech feature V′ and the target prosodic feature R′ can be predicted by referring to the joint distribution of the speech feature and the prosodic feature corresponding to the initial speech data. It can be understood that the obtained target object X0′ includes the joint distribution information of the speech feature and the prosodic feature.

[0037] S15: Obtain target voice data corresponding to the target object.

[0038] Based on the target object, target speech data including a combination of target speech features and target prosodic features is obtained, for example, the target object x′0 is speech-converted to generate a target audio corresponding to the acquired text. The obtained target speech data includes a combination of target speech features V′ and target prosodic features R′, that is, the target audio is converted from speech features associated with prosodic features.

[0039] In this embodiment, initial speech features and initial prosodic features corresponding to initial speech data are obtained, and are concatenated to obtain an initial object to be noisy. The noisy object is obtained by noisyizing the initial object to be noisy, and the phoneme sequence corresponding to the noisy object and the initial speech data is input into the diffusion model to denoise the noisy object to obtain a target object. The target object includes a combination of target speech features and target prosodic features. Furthermore, target speech data including a combination of target speech features and target prosodic features is obtained based on the target object. In the process of collaboratively constructing language features and prosodic features, the naturalness and expressiveness of speech generation are improved.

[0040] In some embodiments, the prosodic feature corresponding to the initial speech data includes at least one of an energy feature, a duration feature, and a pitch feature, and the prosodic feature is extracted based on a phoneme sequence corresponding to the initial speech data.

[0041] Obtain the initial prosodic features corresponding to the initial speech data, for example, use a text front-end tool to convert the text into a phoneme sequence y, and then obtain the corresponding initial prosodic features R0 based on the phoneme sequence y. The initial prosodic features R0 may include energy features e0, duration features d0, and pitch features p0.

[0042] The initial prosodic feature R0 is extracted based on the phoneme sequence y corresponding to the initial speech data. For example, the open source MFA (Montreal Forced Aligner) alignment tool is used to obtain the duration corresponding to the phoneme, that is, the duration feature d0; the open source RMVPE (Robust Model for Vocal Pitch Estimation) human voice pitch model is used to extract the average pitch corresponding to each phoneme, that is, the pitch feature p0; the librosa tool is used to extract the average energy corresponding to each phoneme, that is, the energy feature e0.

[0043] In some examples, initial speech features corresponding to initial speech data are obtained, for example, a phoneme sequence y is predicted using a Mel-spectrogram length predictor to obtain a Mel-spectrogram feature m0 corresponding to the phoneme sequence y as the corresponding initial speech feature, wherein the Mel-spectrogram feature m0 may include the Mel-spectrogram length corresponding to the phoneme sequence y.

[0044] In some embodiments, an initial object to be noisy is noised to obtain a noisy object, including: inputting the initial object to be noisy into a noise scheduler to obtain the noisy object, wherein the noise intensity of the initial object to be noisy at each noisy time step is different and gradually increases to 1, so that the noisy object is noisy with a specific proportion of noise.

[0045] The noise scheduler (for example, noise scheduler) can be used to control the noise intensity and define the noise intensity corresponding to each time step t. The initial object to be noisy X0 is input into the noise scheduler, and then the noisy object X is obtained. t , for example, at each time step t, according to the β preset by the noise scheduler t Add a specific proportion of Gaussian noise to the value pair X0{m0,d0,p0,e0} to obtain the noise object X t {m t ,d t ,p t ,e t}, the noise adding process can be expressed as follows:

[0046]

[0047] The noise intensity of the initial object to be noisy X0 at each noisy time step t is different and gradually increases to 1, so that the noisy object X0 t Gradually tending towards complete randomization.

[0048] In some embodiments, the diffusion model includes a preprocessing network and a denoising network, wherein the denoising network may be a DiT (Diffusion Transformer) network.

[0049] The phoneme sequence corresponding to the noisy object and the initial speech data is input into a diffusion model to denoise the noisy object to obtain a target object, including: inputting the phoneme sequence corresponding to the noisy object and the initial speech data into a preprocessing network to obtain a feature time series, wherein the feature time series includes a combination of text features corresponding to the phoneme sequence and noisy features corresponding to the noisy object; inputting the feature time series into a denoising network to denoise and obtain the target object.

[0050] The phoneme sequence corresponding to the noise object and the initial speech data is input into the preprocessing network. For example, through the preprocessing network, the noise object X t The phoneme sequence y corresponding to the initial speech data is concatenated in the time series to obtain a feature time series, where the feature time series includes a combination of the text features corresponding to the phoneme sequence y and the noise features corresponding to the noise object, for example {y,m t ,d t ,p t ,e t Furthermore, the feature time series is input into the denoising network to perform denoising to obtain the target object. For example, the feature time series and the time step are input into the DiT network together for denoising to predict the target object x′0{m′0,d′0,p′0,e0′0}.

[0051] Among them, the training process of the diffusion model can be expressed as:

[0052] p(m′0,d′0,p′0,e0′|y,m t ,d t ,p t ,e t ,t)

[0053] y represents the phoneme sequence, m t ,d t ,p t ,e t They represent the Mel spectrum feature, duration feature, pitch feature, and energy feature of the noise respectively, and m′0, d′0, p′0, and e0′ represent the Mel spectrum feature, duration feature, pitch feature, and energy feature of the denoised version respectively.

[0054] The corresponding loss function can be expressed as:

[0055]

[0056] Among them, ∈ θ represents the denoising network, and ε represents the noise-added object X t Added noise.

[0057] It can be understood that for the noise object X tNoise removal, that is, at each time step t, the denoising network can predict the current noise component and subtract it from the noisy features to restore the clean features. This process realizes the restoration of noisy speech to clean speech and the reconstruction of prosodic features. In other words, at each time step t, not only a clean Mel spectrum can be generated, but also the corresponding duration features, pitch features and energy features can be jointly generated, which improves the consistency between speech and prosodic features.

[0058] In this embodiment, by learning the relationship between speech features and various prosodic features, that is, adding noise to the speech and prosodic features to the same extent during training (all at time step t), and using the denoising network to predict clean speech and prosodic features, it is possible to learn the joint distribution of speech and prosodic features, thereby improving the naturalness and expressiveness of speech generation.

[0059] In some embodiments, the preprocessing network includes a phoneme encoder and a preprocessing sub-network. The phoneme sequence corresponding to the noised object and the initial speech data is input into the preprocessing network to obtain a feature time series, including: inputting the phoneme sequence into the phoneme encoder to obtain text features; inputting the noised object into the preprocessing sub-network to obtain noised features; and concatenating the text features and the noised features to obtain a feature time series.

[0060] Input the phoneme sequence into the phoneme encoder to obtain text features. For example, use the phoneme encoder to encode the phoneme sequence y to obtain the corresponding text features. Input the noise object into the preprocessing subnetwork to obtain the noise feature. For example, use the preNet network to encode the noise object X. t Mapping is performed to generate noise features of the target dimension. Further, the text features and the noise features are concatenated, for example, the text features and the noise features are concatenated on the realization sequence to obtain a feature time series, which is input into the denoising network together with the time step.

[0061] In some embodiments, the phoneme encoder includes an embedding layer and a Transformer layer. For example, the phoneme encoder includes an embedding layer and a 6-layer Transformer.

[0062] The phoneme sequence is input into the phoneme encoder to obtain text features, including: using the embedding layer to encode the identifier of the phoneme sequence to obtain the phoneme embedding feature; using the Transformer layer to process the phoneme embedding feature to obtain the text feature.

[0063] The phoneme sequence identifier is encoded by the embedding layer to obtain the phoneme embedding feature. For example, the phoneme sequence identifier (phoneme id) is encoded by the embedding layer to obtain a 512-dimensional phoneme embedding feature (phonemeembedding). Further, the phoneme embedding feature is processed by the Transformer layer to obtain the text feature. For example, the phoneme embedding feature is further processed by the 6-layer Transformer to obtain the corresponding text feature.

[0064] In some embodiments, the preprocessing subnetwork includes a convolution layer and a normalization layer. For example, the preprocessing subnetwork includes three one-dimensional convolution layers and a normalization layer.

[0065] The noisy object is input into the preprocessing sub-network to obtain the noisy feature, including: using a convolution layer and a normalization layer to process the noisy object to obtain the noisy feature, wherein the dimension of the noisy feature is consistent with that of the text feature.

[0066] The noisy object is input into the preprocessing sub-network, that is, the noisy object is processed by the convolution layer and the normalization layer to obtain the noisy features. For example, the noisy object X is processed by the convolution layer and the normalization layer. t Project from 80 dimensions to 512 dimensions, and then obtain 512-dimensional noisy features.

[0067] In some embodiments, the characteristic length of the initial speech feature is obtained; wherein the time length of the initial object to be noised is the sum of the characteristic length of the initial speech feature and the characteristic length of the initial prosodic feature.

[0068] The characteristic length of the initial speech feature is obtained, for example, the characteristic length of the initial speech feature is determined to be T1. Furthermore, the time length of the initial object to be noisy can be determined according to the characteristic length of the initial speech feature. The time length of the initial object to be noisy X0 is the sum of the characteristic length of the initial speech feature and the characteristic length of the initial prosodic feature. Among them, the characteristic length of the initial prosodic feature is determined based on the length T2 of the phoneme sequence y. For example, the initial prosodic feature includes energy features, duration features, and pitch features. Since the lengths of duration, pitch, and energy are the same as the length of the phoneme sequence, correspondingly, the characteristic length of the initial prosodic feature is 3 times the length of the phoneme sequence, that is, 3*T2. At this time, the time series length of the initial object to be noisy X0 is T1+3*T2.

[0069] In some embodiments, obtaining the characteristic length of the initial speech feature includes: inputting the phoneme sequence corresponding to the initial speech data into a mel-spectrogram prediction model to obtain the mel-spectrogram feature corresponding to the phoneme sequence to determine the characteristic length of the speech feature.

[0070] The mel spectrum prediction model is an encoder-decoder transformer structure. The phoneme sequence corresponding to the initial speech data is input into the mel spectrum prediction model to obtain the mel spectrum features corresponding to the phoneme sequence. For example, the text encoder in the mel spectrum prediction model is used to encode the phoneme sequence y to obtain the phoneme embedding, and the phoneme embedding is then decoded to obtain the mel spectrum length. The obtained mel spectrum length is used as the mel spectrum feature corresponding to the phoneme sequence y, and the feature length of the initial speech feature can be determined. The training of the mel spectrum prediction model can be optimized using a loss function.

[0071] In some examples, the initial speech features and initial prosodic features corresponding to the initial speech data are obtained, for example, the corresponding Mel-spectrogram feature m0, energy feature e0, duration feature d0, and pitch feature p0 are obtained based on the phoneme sequence y. For example, the dimensions of the Mel-spectrogram feature m0 and the energy feature e0 are 80 dimensions, and the dimensions of the duration feature d0 and the pitch feature p0 are 1 dimension. The duration feature d0 and the pitch feature p0 need to be copied 80 times to obtain 80-dimensional duration features d0 and pitch features p0. Further, the Mel-spectrogram feature m0 with the dimension unified into 80 dimensions, the energy feature e0, the duration feature d0, and the pitch feature p0 are spliced ​​in time series, and the initial object to be noisy X0 can be obtained. At this time, the time series length of the initial object to be noisy X0 is three times the phoneme sequence length plus the Mel-spectrogram length.

[0072] It can be understood that in some examples, the length of the mel spectrum is determined by using the phoneme sequence through the mel spectrum length predictor, and since the length of the duration, pitch, and energy is the same as the length of the phoneme sequence, the noise object X can be determined. t Further, the noise object X with full Gaussian noise t After N steps of reverse diffusion iteration (for example, set to 1000), the speech and duration, pitch, and energy features can be jointly generated.

[0073] For ease of understanding, the speech synthesis generation method of the embodiment of the present application is illustrated by way of example. Figure 2 As shown, Figure 2 It is a flowchart of a speech synthesis generation method according to an embodiment of the present application.

[0074] For example, the initial speech features and initial prosodic features corresponding to the text are obtained. The initial speech features include Mel spectrum features, and the initial prosodic features include duration features, pitch features, and energy features. The initial speech features and initial prosodic features are concatenated to obtain the initial object to be noisy X0{m0, d0, p0, e0}, and the initial object to be noisy X0 is input into the noise scheduler to obtain the noisy object X0. t {m t,d t ,p t ,e t The phoneme encoder is used to encode the phoneme sequence y to obtain a 512-dimensional text feature, and the preprocessing sub-network is used to encode the noise object X. t Projection is performed to obtain 512-dimensional noisy features, and the text features and the noisy features are concatenated to obtain a feature time series, which is input into the denoising network to obtain the target object x′0{m′0,d′0,p′0,e0′}. Then, the target speech data corresponding to the text can be obtained based on the target object x′0.

[0075] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0076] See also Figure 3 , Figure 3 3 is a schematic diagram of the structure of an electronic device of an embodiment of the present application. The electronic device 30 includes a memory 31 and a processor 32 coupled to each other, and the processor 32 is used to execute program instructions stored in the memory 31 to implement the steps of the above-mentioned speech synthesis generation method embodiment. In a specific implementation scenario, the electronic device 30 may include but is not limited to: a microcomputer, a server, which is not limited here.

[0077] Specifically, the processor 32 is used to control itself and the memory 31 to implement the steps of the above-mentioned speech synthesis generation method embodiment. The processor 32 can also be called a CPU (Central Processing Unit), and the processor 32 may be an integrated circuit chip with signal processing capabilities. The processor 32 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 32 can be implemented by an integrated circuit chip.

[0078] See also Figure 4 , Figure 44 is a schematic diagram of the structure of a non-volatile computer-readable storage medium according to an embodiment of the present application. The non-volatile computer-readable storage medium 40 is used to store a computer program 401. When the computer program 401 is executed by a processor, for example, the above-mentioned Figure 3 When the processor 32 in the embodiment is executed, it is used to implement the steps of the above-mentioned embodiment of the speech synthesis generation method.

[0079] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0080] In the several embodiments provided in the present application, it should be understood that the disclosed methods and related devices can be implemented in other ways. For example, the above-described related device implementations are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication disconnection shown or discussed can be through some interfaces, indirect coupling or communication disconnection of devices or units, which can be electrical, mechanical or other forms.

[0081] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0083] It is easy for a person skilled in the art to know that many modifications and changes can be made to the device and method while maintaining the teaching content of the present application. Therefore, the above disclosure should be considered as limited only by the scope of the appended claims.

Claims

1. A speech synthesis generation method, characterized in that: include: Acquire initial speech features and initial prosodic features corresponding to the initial speech data; The initial speech feature and the initial prosodic feature are concatenated to obtain an initial object to be noisy; Adding noise to the initial object to be noisy to obtain a noisy object; Inputting the phoneme sequence corresponding to the noisy object and the initial speech data into a diffusion model to denoise the noisy object to obtain a target object, wherein the target object includes a combination of a target speech feature and a target prosodic feature; Obtain target voice data corresponding to the target object.

2. The method according to claim 1, characterized in that The step of adding noise to the initial object to be noisy to obtain the noisy object comprises: The initial object to be noisy is input into a noise scheduler to obtain the noisy object, wherein the noise intensity of the initial object to be noisy at each noisy time step is different and gradually increases to 1, so that the noisy object is noisy with a specific proportion of noise.

3. The method according to claim 1, characterized in that The diffusion model includes a preprocessing network and a denoising network; Inputting the noisy object and the phoneme sequence corresponding to the initial speech data into a diffusion model to denoise the noisy object to obtain a target object, including: Inputting the phoneme sequence corresponding to the noise object and the initial speech data into the preprocessing network to obtain a feature time series, wherein the feature time series includes a combination of text features corresponding to the phoneme sequence and noise features corresponding to the noise object; The characteristic time series is input into a denoising network for denoising to obtain the target object.

4. The method according to claim 3, characterized in that The preprocessing network includes a phoneme encoder and a preprocessing sub-network; Inputting the phoneme sequence corresponding to the noise-added object and the initial speech data into the preprocessing network to obtain a feature time series, including: Inputting the phoneme sequence into a phoneme encoder to obtain the text feature; Inputting the noise-added object into the preprocessing subnetwork to obtain the noise-added feature; The text feature and the noise feature are concatenated to obtain the feature time series.

5. The method according to claim 4, characterized in that The phoneme encoder includes an embedding layer and a Transformer layer; Inputting the phoneme sequence into a phoneme encoder to obtain the text features includes: Encoding the identifier of the phoneme sequence using the embedding layer to obtain a phoneme embedding feature; The phoneme embedding feature is processed using the Transformer layer to obtain the text feature.

6. The method according to claim 4, characterized in that The preprocessing sub-network includes a convolutional layer and a normalization layer; Inputting the noise-added object into the preprocessing subnetwork to obtain the noise-added feature includes: The noisy object is processed by using the convolution layer and the normalization layer to obtain the noisy feature, wherein the dimension of the noisy feature is consistent with that of the text feature.

7. The method according to claim 1, characterized in that Also includes: Obtaining a feature length of the initial speech feature; The time length of the initial object to be noised is the sum of the feature length of the initial speech feature and the feature length of the initial prosodic feature.

8. The method according to claim 7, characterized in that The step of obtaining the feature length of the initial speech feature comprises: The phoneme sequence corresponding to the initial speech data is input into a mel-spectrogram prediction model to obtain a mel-spectrogram feature corresponding to the phoneme sequence, so as to determine the feature length of the initial speech feature.

9. The method according to any one of claims 1 to 8, characterized in that: The prosodic feature corresponding to the initial speech data includes at least one of an energy feature, a duration feature, and a pitch feature, and the prosodic feature is extracted based on a phoneme sequence corresponding to the initial speech data.

10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech synthesis generation method according to any one of claims 1 to 9.

11. A non-volatile computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the speech synthesis generation method according to any one of claims 1 to 9 is implemented.

Images