A speech synthesis method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a speech synthesis method and device, electronic equipment and storage medium, belonging to the technical field of speech processing. The method comprises: preprocessing the obtained sample text to obtain sample phoneme features of the sample text; performing feature conversion on the sample phoneme features based on an acoustic feature generator in a preset diffusion model, performing variable mapping on the converted sample phoneme features to obtain a hidden sequence; decoding the hidden sequence based on the acoustic feature generator to obtain a mel spectrum of the sample phoneme features; performing noise diffusion on the mel spectrum to obtain a noise diffusion sample; inputting the mel spectrum and the noise diffusion sample into the preset diffusion model for training to obtain a denoising model; and inputting the obtained target text into the denoising model for sound wave synthesis to obtain a speech waveform corresponding to the target text. The embodiments of the present application can improve the quality and stability of speech synthesis and improve the fidelity of the mel spectrum.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, electronic device and storage medium. Background Technology

[0002] With the development of speech signal processing technology, speech synthesis has gradually become an important research branch in the field of speech signal processing. Among them, the most common speech synthesis technique is based on TTS (Text To Speech). Text-to-speech synthesis is a typical multimodal generation task, which transforms a given text input sequence into speech output with different speaker identities, emotions, and styles. Currently, mainstream text-to-speech synthesis systems mainly consist of three parts: a text front-end, an acoustic model, and a vocoder. Text-to-speech synthesis systems can construct the speech waveform of a target speaker. However, in text-to-speech tasks with multiple speakers, existing models based on autoregressive methods accumulate prediction errors during inference, leading to problems such as word skipping and repetition in the generated synthesized speech, thus affecting the quality of the generated speech. Summary of the Invention

[0003] The main objective of this application is to provide a speech synthesis method, apparatus, electronic device, and storage medium that can improve the quality and stability of speech synthesis and enhance the fidelity of the Mel spectrum.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0005] The obtained sample text is preprocessed to obtain the sample phoneme features of the sample text;

[0006] Based on the acoustic feature generator in the preset diffusion model, the sample phoneme features are transformed, and the transformed sample phoneme features are variable-mapped to obtain a hidden sequence.

[0007] The hidden sequence is decoded based on the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features;

[0008] The Mel spectrum is subjected to noise diffusion to obtain a noise diffusion sample;

[0009] The Mel spectrum and the noise diffusion sample are input into the preset diffusion model for training to obtain the denoising model;

[0010] The acquired target text is input into the denoising model for sound wave synthesis to obtain the speech waveform corresponding to the target text.

[0011] In some embodiments, the preset diffusion model includes a text front-end module, wherein preprocessing the acquired sample text to obtain the sample phoneme features of the sample text includes:

[0012] The sample text is input into the text front-end module for homograph-heterogeneous character replacement to obtain the replaced text;

[0013] The replaced text is text normalized to obtain a phoneme sequence;

[0014] The sample phoneme features are generated based on the phoneme sequence.

[0015] In some embodiments, the acoustic feature generator includes an encoder and an adapter; the acoustic feature generator based on the preset diffusion model performs feature transformation on the sample phoneme features, and performs variable mapping on the transformed sample phoneme features to obtain a hidden sequence, including:

[0016] The phoneme sequence in the sample phoneme features is input into the encoder in the acoustic feature generator for sequence conversion to obtain the conversion result;

[0017] The sample phoneme features are labeled to obtain the audio energy label, pitch label, and duration label of the sample phoneme features;

[0018] The conversion result, the audio energy label, the pitch label, and the duration label are input into the adapter for variable mapping to obtain the hidden sequence.

[0019] In some embodiments, the adapter includes a duration predictor, an energy predictor, and a pitch predictor; the step of inputting the conversion result, the audio energy label, the pitch label, and the duration label into the adapter for variable mapping to obtain the hidden sequence includes:

[0020] The duration tag is input into the duration predictor for prediction to obtain duration information;

[0021] The audio energy label is input into the energy predictor for prediction to obtain audio energy information;

[0022] The pitch label is input into the pitch predictor for prediction to obtain pitch information;

[0023] The duration information, the audio energy information, and the pitch information are added to the conversion result to generate the hidden sequence.

[0024] In some embodiments, the acoustic feature generator includes a decoder; the decoding of the hidden sequence based on the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features includes:

[0025] The hidden sequence is input into the decoder for causal convolution to obtain the convolution result;

[0026] Probabilistic prediction is performed on the convolution result to obtain the time step of the hidden sequence;

[0027] The hidden sequence is decoded according to the time step to obtain the Mel spectrum.

[0028] In some embodiments, the step of performing noise diffusion on the Mel spectrum to obtain a noise diffusion sample includes:

[0029] Noise is added to the Mel spectrum according to the time step to obtain the added result;

[0030] The added result is denoised to obtain the noise diffusion sample.

[0031] In some embodiments, the preset diffusion model includes a first preset loss function and a second preset loss function; the step of inputting the Mel spectrum and the noise diffusion samples into the preset diffusion model for training to obtain a denoising model includes:

[0032] The acoustic feature generator is calculated based on the Mel spectrum and the first preset loss function to obtain the reconstruction loss value;

[0033] Calculate the Mel spectrum and the distance between the noise diffusion samples to obtain the discrimination loss value;

[0034] The matching loss value is calculated based on the noise diffusion sample and the second preset loss function;

[0035] The preset diffusion model is updated based on the reconstruction loss value, the discrimination loss value, and the matching loss value to obtain the denoising model.

[0036] To achieve the above objectives, a second aspect of this application provides a speech synthesis apparatus, the apparatus comprising:

[0037] The phoneme extraction module is used to preprocess the acquired sample text to obtain the sample phoneme features of the sample text.

[0038] The feature conversion module is used to perform feature conversion on the sample phoneme features based on the acoustic feature generator in the preset diffusion model, and to perform variable mapping on the converted sample phoneme features to obtain a hidden sequence.

[0039] The spectrum acquisition module is used to decode the hidden sequence based on the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features;

[0040] A noise diffusion module is used to diffuse noise from the Mel spectrum to obtain a noise diffusion sample.

[0041] The model training module is used to input the Mel spectrum and the noise diffusion sample into the preset diffusion model for training to obtain a denoising model.

[0042] The speech determination module is used to input the acquired target text into the denoising model for sound wave synthesis to obtain a speech waveform corresponding to the target text.

[0043] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speech synthesis method as described in the first aspect.

[0044] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method as described in the first aspect.

[0045] The speech synthesis method, apparatus, electronic device, and storage medium proposed in this application first preprocess the acquired sample text to obtain the sample phoneme features of the sample text. Then, based on the acoustic feature generator in the preset diffusion model, the sample phoneme features are transformed, and the transformed sample phoneme features are variable-mapped to obtain a hidden sequence. Then, the hidden sequence is decoded by the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features. Then, noise diffusion is performed on the Mel spectrum to obtain noise-diffused samples, thereby avoiding the influence of noise in the speech conversion process. Finally, the Mel spectrum and noise-diffused samples are input into the preset diffusion model for training to obtain a trained denoising model, which enhances the robustness and stability of the preset diffusion model. The acquired target text is input into the denoising model for sound wave synthesis to obtain the speech waveform corresponding to the target text, thereby achieving denoising of the synthesized speech and improving the quality of the synthesized speech waveform. Attached Figure Description

[0046] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0047] Figure 2 yes Figure 1 The flowchart of step S101 in the text;

[0048] Figure 3 yes Figure 1 The flowchart of step S102 in the document;

[0049] Figure 4 yes Figure 3 The flowchart of step S303 in the process;

[0050] Figure 5 yes Figure 1 The flowchart of step S103 in the process;

[0051] Figure 6 yes Figure 1 The flowchart of step S104 in the process;

[0052] Figure 7 yes Figure 1 The flowchart of step S105 in the process;

[0053] Figure 8 This is a schematic diagram of the speech synthesis device provided in the embodiments of this application;

[0054] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0058] First, let's analyze some of the terms used in this application:

[0059] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0060] Markov Chain (MC): A Markov chain is a stochastic process in probability theory and mathematical statistics that possesses the Markov property and exists within a discrete exponent set and state space. Markov chains applicable to continuous exponent sets are called Markov processes, but they are sometimes considered subsets of Markov chains, namely continuous-time Markov chains, in contrast to discrete-time Markov chains. Therefore, the concept of a Markov chain is quite broad. A Markov chain can be defined using a transition matrix and a transition graph. Besides the Markov property, Markov chains may possess irreducibility, recurrence, periodicity, and ergodicity. An irreducible and recurrenced Markov chain is a strictly stationary Markov chain with a unique stationary distribution. The limiting distribution of an ergodic Markov chain converges to its stationary distribution.

[0061] Gaussian noise is a type of noise whose probability density function follows a Gaussian distribution (i.e., a normal distribution). Common types of Gaussian noise include fluctuation noise, cosmic noise, thermal noise, and shot noise. Besides commonly used noise suppression methods, mathematical statistics methods are frequently employed to suppress Gaussian noise.

[0062] Grapheme-to-Phoneme (G2P): G2P uses Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks to convert English words into phonemes. The sequence-to-sequence model of LSTM networks has been successfully applied to many projects, including machine translation and grapheme-to-phoneme conversion.

[0063] Corrected Linear Unit (ReLU): The ReLU activation function is a commonly used neural activation function. The ReLU function is actually a piecewise linear function that sets all negative values ​​to 0 while leaving positive values ​​unchanged; this operation is known as unilateral inhibition.

[0064] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.

[0065] Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis (HiFi-GAN): Comprising a generator and two discriminators, HiFi-GAN can convert low-quality recordings into high-quality studio-quality audio. Previous AI audio processing methods typically only improved one aspect of the recording quality, such as filtering background noise or removing reverb. This new method, however, is a versatile tool.

[0066] Dropout: Dropout is a method for optimizing artificial neural networks with deep structures. During the learning process, some weights or outputs of the hidden layers are randomly reset to zero, thereby reducing the interdependence between nodes, which in turn regularizes the neural network and reduces its structural risk.

[0067] Timesteps: In solving nonlinear problems, the entire solution process is divided into smaller segments. For structural problems, this segmentation is equivalent to dividing the loading process into multiple steps, with the structural loading changing slightly in each step, until the entire loading process is completed. For dynamic problems, this loading step can be understood as a true time interval. For static problems, this loading step is what many solvers call a pseudo-time step.

[0068] Unicode: Unicode, also known as Universal Character Set or Unique Character Set, is an industry standard in computer science that includes character sets and encoding schemes. Unicode was created to address the limitations of traditional character encoding schemes. It assigns a unified and unique binary code to each character in every language to meet the requirements of cross-language and cross-platform text conversion and processing.

[0069] Based on this, embodiments of this application provide a speech synthesis method and apparatus, electronic device and storage medium, which aim to achieve noise reduction of synthesized speech and improve the quality of synthesized speech waveforms.

[0070] The speech synthesis method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech synthesis method in this application is described.

[0071] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0072] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech synthesis, natural language processing, and machine learning / deep learning.

[0073] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0076] Step S101: Preprocess the obtained sample text to obtain the sample phoneme features of the sample text;

[0077] In step S101 of some embodiments, the obtained sample text is preprocessed to extract the sample phoneme features of the sample text, which facilitates the subsequent extraction of Mel spectrum based on the phoneme features.

[0078] Step S102: Based on the acoustic feature generator in the preset diffusion model, the sample phoneme features are transformed, and the transformed sample phoneme features are variable-mapped to obtain the hidden sequence.

[0079] In step S102 of some embodiments, the sample phoneme features are input into the acoustic feature generator in the preset diffusion model for feature transformation, different variable information is added to the sample phoneme features, and then the transformed sample phoneme features are variable-mapped to obtain the hidden sequence corresponding to the sample phoneme features, thereby providing sufficient information to predict changing speech and solving the one-to-many mapping problem in speech synthesis.

[0080] It should be noted that, in addition to the sample text, speech or recordings typically contain a large amount of other variable information, such as phoneme duration, pitch, or energy. Phoneme duration represents the speech rate; pitch represents the emotional changes in speech, which has a significant impact on perception; and energy represents the frame-level amplitude of the Mel spectrum, directly affecting the loss in Mel spectrum calculation. Therefore, the variable information is not entirely determined by the sample text, and a one-to-many mapping problem may exist. This one-to-many mapping problem in speech synthesis can be solved by using the acoustic feature generator in the pre-defined diffusion model to perform feature transformation on the sample phoneme features.

[0081] Step S103: Decode the hidden sequence based on the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features;

[0082] In step S103 of some embodiments, the hidden sequence obtained in step S102 is decoded to obtain the Mel spectrum corresponding to the sample phoneme features, thereby improving the quality of speech synthesis.

[0083] Step S104: Perform noise diffusion on the Mel spectrum to obtain a noise diffusion sample;

[0084] In step S104 of some embodiments, noise diffusion is performed on the Mel spectrum obtained in step S103 to obtain a noise diffusion sample, thereby improving the fidelity of the Mel spectrum and avoiding the influence of noise.

[0085] It should be noted that noise diffusion of the Mel spectrum involves inputting the Mel spectrum into a diffusion model for noise diffusion. The diffusion model includes a parameterless Markov chain and a parameterized Markov chain. The parameterless Markov chain is used to diffuse the acoustic features in the Mel spectrum, while the parameterized Markov chain is used to denoise the Mel spectrum. During the diffusion process, Gaussian noise is added to the Mel spectrum data structure until the data result is completely destroyed. During the denoising process, a denoising function is used to remove the added noise and restore the data structure, thereby obtaining the noise diffusion sample.

[0086] Step S105: Input the Mel spectrum and noise diffusion samples into the preset diffusion model for training to obtain the denoising model;

[0087] In step S105 of some embodiments, the Mel spectrum and noise diffusion samples are input into a preset diffusion model for training to obtain a denoising model, thereby enhancing the denoising ability of the denoising model and improving the quality of speech synthesis by the denoising model.

[0088] It should be noted that the denoising model is trained on the preset diffusion model using Mel spectrum and noise diffusion samples. Therefore, the denoising model has the ability to extract Mel spectrum and diffuse noise.

[0089] Step S106: The obtained target text input denoising model is used for sound wave synthesis to obtain the speech waveform corresponding to the target text.

[0090] In step S106 of some embodiments, the obtained target text input denoising model is subjected to sound wave synthesis to obtain a speech waveform corresponding to the target text, thereby enhancing the quality of the speech waveform and effectively generating a high-quality speech waveform.

[0091] It should be noted that the acquired target text is input into the denoising model. First, the text front-end module in the denoising model preprocesses the target text to obtain the target phoneme features. Then, based on the acoustic feature generator in the denoising model, variables are added and mapped to the target phoneme features. Next, the target phoneme features after variable mapping are decoded to obtain the target Mel spectrum corresponding to the target phoneme features. Then, based on the vocoder in the denoising model, a time-domain waveform corresponding to the target Mel spectrum is generated, and the time-domain waveform is reconstructed. Finally, the speech waveform corresponding to the target text is obtained, realizing the waveform reconstruction of the Mel spectrum, generating high-quality synthesized speech, and avoiding the influence of noise.

[0092] It is understood that the vocoder can be a HiFi-GAN, and waveform reconstruction includes, but is not limited to, operations such as deconvolution and multi-receptive field fusion. This embodiment does not impose specific limitations.

[0093] It should be noted that the discriminator in the vocoder includes a multi-scale discriminator and a multi-period discriminator. Therefore, this application can identify speech waveforms from two different perspectives. This embodiment does not impose any specific limitations.

[0094] Steps S101 to S106 as shown in the embodiments of this application involve preprocessing the acquired sample text to obtain the sample phoneme features of the sample text, then performing feature transformation on the sample phoneme features based on the acoustic feature generator in the preset diffusion model, and performing variable mapping on the transformed sample phoneme features to obtain a hidden sequence. The hidden sequence is then decoded by the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features. Noise diffusion is then performed on the Mel spectrum to obtain a noise-diffused sample, thereby avoiding the influence of noise during the speech conversion process. Finally, the Mel spectrum and the noise-diffused sample are input into the preset diffusion model for training to obtain a trained denoising model, enhancing the robustness and stability of the preset diffusion model. The acquired target text is then input into the denoising model for sound wave synthesis to obtain a speech waveform corresponding to the target text, thereby achieving denoising of the synthesized speech and improving the quality of the synthesized speech waveform.

[0095] Please see Figure 2In some embodiments, step S101 may include, but is not limited to, steps S201 to S203:

[0096] It should be noted that the preset diffusion model includes a text front-end module.

[0097] Step S201: Input the sample text into the text front-end module for homograph-to-code replacement to obtain the replacement text;

[0098] Step S202: Perform text normalization on the replacement text to obtain a phoneme sequence;

[0099] Step S203: Generate sample phoneme features based on the phoneme sequence.

[0100] In some embodiments, the sample text is first input into the text front-end module for recognition to determine the language of the sample text, such as Chinese, English, or Japanese. The text front-end module processes illegal input of the sample text, and then performs homograph replacement on the sample text to remove invisible characters, unify punctuation marks, and convert full-width input to half-width input, thereby obtaining the replaced text. Then, text standardization is performed on physical quantities, currencies, dates, etc. of different formats to obtain phoneme sequences. Finally, sample phoneme features are generated based on the phoneme sequences, thereby achieving the standardization of the sample text and reducing interference in the process of extracting sample phoneme features.

[0101] In step S201 of some embodiments, the sample text is input into the text front-end module for homograph-heterogeneous code replacement to obtain the replacement text. Unicode encoding can be used for homograph-heterogeneous code replacement to avoid interference from texts with the same spelling and reduce conversion errors.

[0102] In step S202 of some embodiments, the replacement text is standardized to obtain a phoneme sequence, and the text format is unified to facilitate the generation of standardized sample phoneme features.

[0103] It should be noted that text standardization includes, but is not limited to, unifying mathematical symbols, physical symbols, and date information in different formats; this embodiment does not impose specific limitations.

[0104] In step S203 of some embodiments, sample phoneme features are generated based on the phoneme sequence. The method for generating sample phoneme features can be an open-source text-to-phoneme model or a G2P model trained using a basic dictionary.

[0105] Please see Figure 3 In some embodiments, step S102 may include, but is not limited to, steps S301 to S303:

[0106] It should be noted that the acoustic feature generator includes an encoder and an adapter.

[0107] It is understandable that the encoder in the acoustic feature generator is a Transformer encoder, and the adapter is a variable adapter.

[0108] Step S301: Input the phoneme sequence in the sample phoneme features into the encoder in the acoustic feature generator for sequence conversion to obtain the conversion result;

[0109] In step S301 of some embodiments, the phoneme sequence in the sample phoneme features is input into the encoder in the acoustic feature generator for sequence conversion. The encoder of the acoustic feature generator includes four feedforward Transformer modules. Each module is composed of self-attention layers and convolution layers. The Transformer module includes hidden nodes, attention heads, convolution kernels, and filters. The number of hidden nodes is 256, the number of attention heads is 2, the size of the convolution kernel is 9, and the size of the filter is 1024. The phoneme sequence is convolved and filtered to obtain the conversion result, which improves the accuracy of sequence conversion and enhances the fitting ability of complex processes.

[0110] Step S302: Label the sample phoneme features to obtain the audio energy label, pitch label and duration label of the sample phoneme features;

[0111] Step S303: Input the conversion result, audio energy label, pitch label, and duration label into the adapter and perform variable mapping to obtain the hidden sequence.

[0112] In some embodiments, the phoneme sequence in the sample phoneme features is first input into the encoder of the acoustic feature generator for sequence transformation. After convolution, filtering, and superposition, the transformation result is obtained. Then, the sample phoneme features are labeled to obtain the audio energy label, pitch label, and duration label of the sample phoneme features, thereby realizing the prediction of the sample phoneme features. Finally, the transformation result, audio energy label, pitch label, and duration label are input into the adapter for variable mapping, which facilitates variable mapping through various labels to obtain the hidden sequence, thereby solving the one-to-many mapping problem in the speech synthesis process and improving the accuracy of mapping.

[0113] Please see Figure 4 In some embodiments, step S303 may include, but is not limited to, steps S401 to S404:

[0114] It should be noted that the adapter includes a duration predictor, an energy predictor, and a pitch predictor.

[0115] Understandably, the adapter consists of a 2-layer convolutional network with modified linear unit activation, each layer followed by a normalized and randomly deactivated layer, and an additional linear layer to project the hidden state to the output sequence. The duration predictor, energy predictor, and pitch predictor have the same model structure but different parameters.

[0116] Step S401: Input the duration tag into the duration predictor for prediction to obtain duration information;

[0117] In step S401 of some embodiments, the duration label is input into the duration predictor for prediction to obtain duration information corresponding to the sample phoneme features, thereby improving the segmentation accuracy of the hidden sequence and reducing the information gap between outputs.

[0118] It should be noted that, in order to improve the accuracy of duration information segmentation and reduce the information gap between input and output, in this embodiment, duration labels are input into the duration predictor for prediction. The duration predictor is a high-performance open-source speech-to-text segmentation system that does not require any manual alignment annotations, thereby obtaining duration information corresponding to the sample phoneme features.

[0119] Step S402: Input the audio energy tag into the energy predictor for prediction to obtain audio energy information;

[0120] In step S402 of some embodiments, the audio energy label is input into the energy predictor for prediction to obtain audio energy information, thereby obtaining the audio energy information of the sample phoneme features, which facilitates the subsequent generation of hidden sequences, thereby obtaining the Mel spectrum.

[0121] Step S403: Input the pitch label into the pitch predictor for prediction to obtain pitch information;

[0122] In step S403 of some embodiments, the pitch label is input into the pitch predictor for prediction to obtain pitch information corresponding to the sample phoneme features, thereby directly setting the sequence frame without sound to zero and avoiding interference from the phoneme features without sound.

[0123] Step S404: Add duration information, audio energy information, and pitch information to the conversion result to generate a hidden sequence.

[0124] In step S404 of some embodiments, duration information, audio energy information, and pitch information are added as variables to the conversion result to generate a hidden sequence, reduce the gap in phoneme information output, solve the one-to-many mapping problem in speech synthesis, and facilitate the subsequent obtaining of high-quality Mel spectrum.

[0125] Please see Figure 5In some embodiments, step S103 may also include, but is not limited to, steps S501 to S503:

[0126] It should be noted that the acoustic feature generator includes a decoder.

[0127] Understandably, the decoder is a deep neural network capable of generating the original audio waveform. In step S501, the hidden sequence is input into the decoder and subjected to causal convolution to obtain the convolution result.

[0128] In step S501 of some embodiments, the hidden sequence input decoder is subjected to causal convolution to obtain the convolution result, and the causal convolution avoids the occurrence of data order violations.

[0129] It should be noted that occlusion convolution can also be used to convolve the hidden sequence. When using occlusion convolution to process the hidden sequence, it can be achieved by constructing an occlusion tensor and performing a dot product with the convolution kernel before use.

[0130] Understandably, since there are no recurrent connections in the model using causal convolution, the dilation rate is set to 1 during the causal convolution process to make the receptive field large enough and reduce computational cost.

[0131] Step S502: Perform probability prediction on the convolution result to obtain the time step of the hidden sequence;

[0132] In step S502 of some embodiments, the convolution result obtained in step S501 is input into a parameterless Markov chain and a parameterized Markov chain for probability prediction to obtain the time step of the hidden sequence, which facilitates the classification of the hidden sequence.

[0133] Step S503: Decode the hidden sequence according to the time step to obtain the Mel spectrum.

[0134] In step S503 of some embodiments, the decoding of the hidden sequence is restricted according to the time step to obtain the Mel spectrum. For example, the time step of both the parameterless Markov chain and the parameterized Markov chain is 5 steps. Therefore, during the noise diffusion process, the data structure is completely destroyed in the fifth step.

[0135] Please see Figure 6 In some embodiments, step S104 includes, but is not limited to, steps S601 to S602:

[0136] Step S601: Add noise to the Mel spectrum according to the time step to obtain the added result;

[0137] Step S602: Perform a denoising operation on the added result to obtain a noise diffusion sample.

[0138] In some embodiments, the Mel spectrum is input into the discriminator, which can add speaker noise to the Mel spectrum according to the time step, obtain the added result, and then denoise the added result to remove the time step and speaker noise, and restore the added result to obtain a noise diffusion sample.

[0139] Please see Figure 7 In some embodiments, step S105 may include, but is not limited to, steps S701 to S704:

[0140] It should be noted that the preset diffusion model includes a first preset loss function and a second preset loss function.

[0141] Step S701: Calculate the reconstruction loss value by using the Mel spectrum and the first preset loss function on the acoustic feature generator.

[0142] Step S702: Calculate the Mel spectrum and the distance between noise diffusion samples to obtain the discrimination loss value;

[0143] Step S703: Calculate the matching loss value based on the noise diffusion sample and the second preset loss function;

[0144] Step S704: Update the preset diffusion model based on the reconstruction loss value, the discrimination loss value, and the matching loss value to obtain the denoising model.

[0145] In some embodiments, firstly based on the Mel spectrum and a first preset loss function L recon The acoustic feature generator is used to calculate the reconstruction loss value. Then, the Mel spectrum and the distance between noise diffusion samples are calculated, and the loss function L of the discriminator is applied. adv Calculations are performed to obtain the discrimination loss value, and then based on the noise diffusion samples and the second preset loss function L... fm The matching loss value is calculated, and finally, a preset diffusion model is trained based on the reconstruction loss value, the discrimination loss value, and the matching loss value to obtain the denoising model.

[0146] It should be noted that the first preset loss function is the loss function of the acoustic feature generator, and the second preset loss function is the feature matching loss function, which distinguishes between real and fake data through similarity measurement.

[0147] Understandably, the denoising model is trained by minimizing the following loss function, as shown in Equation (1):

[0148] L G =L adv +L recon +λL fm (1)

[0149] It should be noted that the parameter λ is a dynamic scaling parameter, set to λ = L. recon / L fm .

[0150] Please see Figure 8 This application also provides a speech synthesis apparatus that can implement the above-described speech synthesis method. The apparatus includes:

[0151] The phoneme extraction module 801 is used to preprocess the acquired sample text to obtain the sample phoneme features of the sample text.

[0152] The feature conversion module 802 is used to perform feature conversion on the sample phoneme features based on the acoustic feature generator in the preset diffusion model, and to perform variable mapping on the converted sample phoneme features to obtain the hidden sequence.

[0153] The spectrum acquisition module 803 is used to decode the hidden sequence based on the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features, wherein the Mel spectrum carries the speech feature information of the target speaker;

[0154] The noise diffusion module 804 is used to diffuse noise from the Mel spectrum to obtain a noise diffusion sample.

[0155] The model training module 805 is used to input the Mel spectrum and noise diffusion samples into the preset diffusion model for training to obtain the denoising model.

[0156] The speech determination module 806 is used to input the acquired target text into the denoising model and perform sound wave synthesis to obtain the speech waveform corresponding to the target text.

[0157] The specific implementation of this speech synthesis device is basically the same as the specific implementation of the speech synthesis method described above, and will not be repeated here.

[0158] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned speech synthesis method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0159] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0160] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0161] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speech synthesis method of the embodiments of this application.

[0162] The input / output interface 903 is used to implement information input and output;

[0163] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0164] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0165] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0166] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described speech synthesis method.

[0167] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0168] The speech synthesis method, speech synthesis device, electronic device, and storage medium provided in this application first preprocess the acquired sample text to obtain the sample phoneme features of the sample text. Then, based on the acoustic feature generator in the preset diffusion model, the sample phoneme features are transformed, and the transformed sample phoneme features are variable-mapped to obtain a hidden sequence. Then, the hidden sequence is decoded by the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features. Then, noise diffusion is performed on the Mel spectrum to obtain noise-diffused samples, thereby avoiding the influence of noise in the speech conversion process. Finally, the Mel spectrum and noise-diffused samples are input into the preset diffusion model for training to obtain a trained denoising model, which enhances the robustness and stability of the preset diffusion model. The acquired target text is input into the denoising model for sound wave synthesis to obtain a speech waveform corresponding to the target text, thereby achieving denoising of the synthesized speech and improving the quality of the synthesized speech waveform.

[0169] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0170] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0173] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0174] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: The obtained sample text is preprocessed to obtain the sample phoneme features of the sample text; The acoustic feature generator in the preset diffusion model performs feature transformation on the sample phoneme features, and performs variable mapping on the transformed sample phoneme features to obtain a hidden sequence; wherein, the acoustic feature generator includes an encoder, a decoder and an adapter, the encoder in the acoustic feature generator is a Transformer encoder, the decoder is a deep neural network, and the adapter is a variable adapter; The decoder based on the acoustic feature generator decodes the hidden sequence to obtain the Mel spectrum of the sample phoneme features; The Mel spectrum is subjected to noise diffusion to obtain a noise diffusion sample; The Mel spectrum and the noise diffusion sample are input into the preset diffusion model for training to obtain the denoising model; The acquired target text is input into the denoising model for sound wave synthesis to obtain the speech waveform corresponding to the target text; The preset diffusion model includes a first preset loss function and a second preset loss function; the step of inputting the Mel spectrum and the noise diffusion samples into the preset diffusion model for training to obtain a denoising model includes: The acoustic feature generator is calculated based on the Mel spectrum and the first preset loss function to obtain the reconstruction loss value; Calculate the Mel spectrum and the distance between the noise diffusion samples to obtain the discrimination loss value; The matching loss value is calculated based on the noise diffusion sample and the second preset loss function; The preset diffusion model is updated based on the reconstruction loss value, the discrimination loss value, and the matching loss value to obtain the denoising model.

2. The speech synthesis method according to claim 1, characterized in that, The preset diffusion model includes a text front-end module; the preprocessing of the acquired sample text to obtain the sample phoneme features of the sample text includes: The sample text is input into the text front-end module for homograph-heterogeneous character replacement to obtain the replaced text; The replaced text is text normalized to obtain a phoneme sequence; The sample phoneme features are generated based on the phoneme sequence.

3. The speech synthesis method according to claim 2, characterized in that, The acoustic feature generator based on the preset diffusion model performs feature transformation on the sample phoneme features, and performs variable mapping on the transformed sample phoneme features to obtain a hidden sequence, including: The phoneme sequence in the sample phoneme features is input into the encoder in the acoustic feature generator for sequence conversion to obtain the conversion result; The sample phoneme features are labeled to obtain the audio energy label, pitch label, and duration label of the sample phoneme features; The conversion result, the audio energy label, the pitch label, and the duration label are input into the adapter for variable mapping to obtain the hidden sequence.

4. The speech synthesis method according to claim 3, characterized in that, The adapter includes a duration predictor, an energy predictor, and a pitch predictor; the process of inputting the conversion result, the audio energy label, the pitch label, and the duration label into the adapter for variable mapping to obtain the hidden sequence includes: The duration tag is input into the duration predictor for prediction to obtain duration information; The audio energy label is input into the energy predictor for prediction to obtain audio energy information; The pitch label is input into the pitch predictor for prediction to obtain pitch information; The duration information, the audio energy information, and the pitch information are added to the conversion result to generate the hidden sequence.

5. The speech synthesis method according to claim 1, characterized in that, The decoder based on the acoustic feature generator decodes the hidden sequence to obtain the Mel spectrum of the sample phoneme features, including: The hidden sequence is input into the decoder for causal convolution to obtain the convolution result; Probabilistic prediction is performed on the convolution result to obtain the time step of the hidden sequence; The hidden sequence is decoded according to the time step to obtain the Mel spectrum.

6. The speech synthesis method according to claim 5, characterized in that, The step of performing noise diffusion on the Mel spectrum to obtain a noise diffusion sample includes: Noise is added to the Mel spectrum according to the time step to obtain the added result; The added result is denoised to obtain the noise diffusion sample.

7. A speech synthesis device, characterized in that, The device includes: The phoneme extraction module is used to preprocess the acquired sample text to obtain the sample phoneme features of the sample text. The feature transformation module is used to perform feature transformation on the sample phoneme features based on the acoustic feature generator in the preset diffusion model, and to perform variable mapping on the transformed sample phoneme features to obtain a hidden sequence; wherein, the acoustic feature generator includes an encoder, a decoder and an adapter, the encoder in the acoustic feature generator is a Transformer encoder, the decoder is a deep neural network, and the adapter is a variable adapter; The spectrum acquisition module is used to decode the hidden sequence based on the decoder of the acoustic feature generator to obtain the Mel spectrum of the sample phoneme features; A noise diffusion module is used to diffuse noise from the Mel spectrum to obtain a noise diffusion sample. The model training module is used to input the Mel spectrum and the noise diffusion sample into the preset diffusion model for training to obtain a denoising model. The speech determination module is used to input the acquired target text into the denoising model for sound wave synthesis to obtain a speech waveform corresponding to the target text; The preset diffusion model includes a first preset loss function and a second preset loss function; the step of inputting the Mel spectrum and the noise diffusion samples into the preset diffusion model for training to obtain a denoising model includes: The acoustic feature generator is calculated based on the Mel spectrum and the first preset loss function to obtain the reconstruction loss value; Calculate the Mel spectrum and the distance between the noise diffusion samples to obtain the discrimination loss value; The matching loss value is calculated based on the noise diffusion sample and the second preset loss function; The preset diffusion model is updated based on the reconstruction loss value, the discrimination loss value, and the matching loss value to obtain the denoising model.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech synthesis method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method according to any one of claims 1 to 6.

Images