Speech synthesis method, speech synthesis system, electronic device, and storage medium

Abstract

The application provides a speech synthesis method, a speech synthesis system, an electronic device and a storage medium, and belongs to the technical field of financial technologies. The method comprises the following steps: obtaining sample phoneme data and sample speech, inputting the sample phoneme data into an original synthesis model, performing phoneme coding on the sample phoneme data through a phoneme coding sub-model to obtain phoneme hidden data; performing phoneme adaptation on the phoneme hidden data through a variance adaptation sub-model to obtain phoneme alignment data and phoneme feature data; performing spectral prediction on the phoneme alignment data and the phoneme feature data through a noise reduction sub-model to obtain predicted mel spectrum; adjusting parameters of the original synthesis model according to the sample speech and the predicted mel spectrum to obtain a speech synthesis model; and inputting target text data into the speech synthesis model for speech synthesis processing to obtain target synthesized speech. The embodiment of the application can improve the generation quality and generation efficiency of synthesized speech, and effectively simplify the computational complexity of the speech synthesis process.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech synthesis method, a speech synthesis system, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of fintech and the socio-economic landscape, people have increasingly higher demands for banking services. In scenarios such as intelligent customer service, multi-turn dialogues, and robotic outbound calls, speech synthesis technology can be applied to specific areas including daily business processing, business consultation, business recommendations, marketing, and debt collection. Therefore, being able to convey relevant information to the target audience more realistically and accurately through voice is one of the most effective and direct methods to improve customer experience and service levels. Text-to-Speech (TTS) technology is a technique that synthesizes given text into audio that simulates the pronunciation of a target audience. Related TTS methods generate corresponding Mel spectrograms from the text using an autoregressive approach and then use a pre-trained vocoder to synthesize speech from the generated Mel spectrograms. However, while this method can generate high-fidelity audio, it is computationally intensive and has low synthesis efficiency. Therefore, how to provide a speech synthesis method that can improve the quality and efficiency of synthesized speech while effectively simplifying the computational workload of the speech synthesis process has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to propose a speech synthesis method, a speech synthesis system, an electronic device, and a storage medium, which can improve the generation quality and efficiency of synthesized speech and effectively simplify the computational workload of the speech synthesis process.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech synthesis method, the method comprising:

[0005] Acquire sample data, which includes sample phoneme data and sample speech, wherein the sample phoneme data is used to characterize the text content of the sample speech;

[0006] The sample phoneme data is input into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model.

[0007] The sample phoneme data is processed by phoneme encoding sub-model to obtain phoneme hidden data;

[0008] The phoneme hidden data is processed by the variance adaptor sub-model to obtain phoneme alignment data and phoneme feature data.

[0009] The noise reduction sub-model is used to perform spectral prediction processing on the phoneme alignment data and the phoneme feature data to obtain the predicted Mel spectrum.

[0010] The parameters of the original synthesis model are adjusted based on the sample speech and the predicted Mel spectrum to obtain a speech synthesis model.

[0011] The acquired target text data is input into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0012] In some embodiments, the step of performing spectral prediction processing on the phoneme alignment data and the phoneme feature data through the noise reduction sub-model to obtain the predicted Mel spectrum includes:

[0013] The phoneme alignment data is input into the noise reduction sub-model, and the phoneme alignment data is sampled to obtain candidate adaptation data and the position information of the candidate adaptation data.

[0014] The candidate adaptation data is subjected to spectral diffusion processing according to a preset time step to obtain spectral diffusion data.

[0015] Based on the preset time step, the candidate adaptation data, and the phoneme feature data, the spectrum diffusion data is subjected to spectrum inverse sampling to obtain the predicted spectrum data.

[0016] The predicted Mel spectrum is obtained by generating a spectrum from the predicted spectrum data based on the location information.

[0017] In some embodiments, the step of performing spectral diffusion processing on the candidate adaptation data according to a preset time step to obtain spectral diffusion data includes:

[0018] Obtain the noise scheduling parameters for the preset time step;

[0019] The candidate adaptation data is sampled to obtain the first adaptation data;

[0020] The first adaptation data is noise-added according to the preset time step and the noise scheduling parameters to obtain the second adaptation data;

[0021] The spectral diffusion data is obtained based on the first adaptation data and the second adaptation data.

[0022] In some embodiments, adjusting the parameters of the original synthesis model based on the sample speech and the predicted Mel spectrum to obtain a speech synthesis model includes:

[0023] The diffusion parameters are calculated based on the noise scheduling parameters and the preset time step to obtain the diffusion process parameters;

[0024] Noise distribution data is acquired, and prediction loss is calculated based on the noise distribution data, the candidate adaptation data, the predicted spectrum data, the preset time step, the diffusion process parameters, and the phoneme feature data to obtain prediction loss data.

[0025] The original synthesis model is adjusted based on the predicted loss data to obtain the speech synthesis model.

[0026] In some embodiments, the variance-adaptive submodel includes a duration predictor, a pitch predictor, and an energy predictor;

[0027] The process of performing phoneme adaptation on the phoneme hidden data using the variance adaptor sub-model to obtain phoneme alignment data and phoneme feature data includes:

[0028] The phoneme hidden data is processed by the duration predictor to obtain the phoneme aligned data.

[0029] The phoneme alignment data is processed by the pitch predictor to obtain the first conditional data.

[0030] The energy prediction data of the phoneme alignment data is processed by the energy predictor to obtain the second conditional data.

[0031] The phoneme feature data is obtained by combining the first conditional data and the second conditional data.

[0032] In some embodiments, the pitch predictor includes a pitch activation layer, a normalization layer, and a pitch projection layer;

[0033] The step of performing pitch prediction processing on the phoneme alignment data based on the pitch predictor to obtain first conditional data includes:

[0034] The phoneme alignment data is processed non-linearly based on the pitch activation layer to obtain pitch activation data;

[0035] The pitch activation data is normalized according to the normalization layer to obtain normalized hidden data;

[0036] The normalized hidden data is linearly projected based on the pitch projection layer to obtain the first conditional data.

[0037] In some embodiments, the phoneme coding sub-model includes a phoneme convolutional layer, a phoneme self-attention layer, and a phoneme projection layer;

[0038] The step of performing phoneme encoding processing on the sample phoneme data through the phoneme encoding sub-model to obtain phoneme hidden data includes:

[0039] The sample phoneme data is processed by phoneme convolution layer to obtain phoneme encoded data.

[0040] The phoneme encoding data is processed by the phoneme self-attention layer to obtain phoneme attention data.

[0041] The phoneme attention data is linearly projected onto the phoneme projection layer to obtain the phoneme hidden data.

[0042] To achieve the above objectives, a second aspect of this application provides a speech synthesis system, the system comprising:

[0043] A sample acquisition module is used to acquire sample data, which includes sample phoneme data and sample speech. The sample phoneme data is used to characterize the text content of the sample speech.

[0044] The model input module is used to input the sample phoneme data into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model.

[0045] The phoneme encoding module is used to perform phoneme encoding processing on the sample phoneme data through the phoneme encoding sub-model to obtain phoneme hidden data;

[0046] The phoneme adaptation module is used to perform phoneme adaptation processing on the phoneme hidden data through the variance adaptation sub-model to obtain phoneme alignment data and phoneme feature data.

[0047] The spectrum prediction module is used to perform spectrum prediction processing on the phoneme alignment data and the phoneme feature data through the noise reduction sub-model to obtain the predicted Mel spectrum.

[0048] The parameter adjustment module is used to adjust the parameters of the original synthesis model according to the sample speech and the predicted Mel spectrum to obtain the speech synthesis model.

[0049] The speech synthesis module is used to input the acquired target text data into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0050] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any of the first aspects of the present application.

[0051] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects of the present application.

[0052] The speech synthesis method, speech synthesis system, electronic device, and storage medium proposed in this application first acquire sample data, which includes sample phoneme data and sample speech. The sample phoneme data is used to characterize the text content of the sample speech. Then, the sample phoneme data is input into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model. The phoneme encoding sub-model performs phoneme encoding processing on the sample phoneme data to obtain phoneme hidden data. The variance adaptation sub-model performs phoneme adaptation processing on the phoneme hidden data to obtain phoneme alignment data and phoneme feature data. The noise reduction sub-model performs spectral prediction processing on the phoneme alignment data and phoneme feature data to obtain a predicted Mel spectrum. Then, the parameters of the original synthesis model are adjusted based on the sample speech and the predicted Mel spectrum to obtain a speech synthesis model. The acquired target text data is input into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech. This application embodiment can improve the generation quality and efficiency of synthesized speech and effectively simplify the computational workload of the speech synthesis process. Attached Figure Description

[0053] Figure 1 This is a first flowchart of the speech synthesis method provided in the embodiments of this application;

[0054] Figure 2 yes Figure 1 A flowchart illustrating the specific method of step S130;

[0055] Figure 3 This is a schematic diagram of the structure of the phoneme coding sub-model provided in the embodiments of this application;

[0056] Figure 4 yes Figure 1 A flowchart illustrating the specific method of step S140;

[0057] Figure 5 This is a schematic diagram of the structure of the variance aptamer model provided in the embodiments of this application;

[0058] Figure 6 yes Figure 4 A flowchart illustrating the specific method of step S420;

[0059] Figure 7 yes Figure 1 A flowchart illustrating the specific method of step S150;

[0060] Figure 8 yes Figure 7 A flowchart illustrating the specific method of step S720;

[0061] Figure 9 yes Figure 1 A flowchart illustrating the specific method of step S160;

[0062] Figure 10 This is a block diagram of the module structure of the speech synthesis system provided in the embodiments of this application;

[0063] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0067] First, let's analyze some of the terms used in this application:

[0068] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0069] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. NLP involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0070] Text-to-Speech (TTS) is a technology that converts text into speech. TTS generally includes two steps: the first step is text processing, which mainly converts the text into a phoneme sequence and marks the start and end times, frequency changes, and other information for each phoneme; the second step is speech synthesis, which mainly generates speech based on the phoneme sequence.

[0071] Phoneme: The smallest unit of speech based on the natural properties of speech. It is analyzed based on the articulation of a syllable, and one articulation constitutes one phoneme.

[0072] With the rapid development of fintech and the socio-economic landscape, people have increasingly higher demands for banking services. In scenarios such as intelligent customer service, multi-turn dialogues, and robotic outbound calls, speech synthesis technology can be applied to specific areas including daily business processing, business consultation, business recommendations, marketing, and debt collection. Therefore, being able to convey relevant information to the target audience more realistically and accurately through voice is one of the most effective and direct methods to improve customer experience and service levels. Text-to-Speech (TTS) technology is a technique that synthesizes given text into audio that simulates the pronunciation of a target audience. Related TTS methods generate corresponding Mel spectrograms from the text using an autoregressive approach and then use a pre-trained vocoder to synthesize speech from the generated Mel spectrograms. However, while this method can generate high-fidelity audio, it is computationally intensive and has low synthesis efficiency. Therefore, how to provide a speech synthesis method that can improve the quality and efficiency of synthesized speech while effectively simplifying the computational workload of the speech synthesis process has become an urgent technical problem to be solved.

[0073] Based on this, the speech synthesis method, speech synthesis system, electronic device and storage medium provided in the embodiments of this application can improve the generation quality and efficiency of synthesized speech, and effectively simplify the computational workload of the speech synthesis process.

[0074] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0075] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0076] The speech synthesis method provided in this application relates to the field of artificial intelligence technology. The speech synthesis method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the speech synthesis method, but is not limited to the above forms.

[0077] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0078] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user voice data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0079] Please refer to Figure 1 , Figure 1 This is an optional flowchart of the speech synthesis method provided in the embodiments of this application. In some embodiments of this application, the speech synthesis method includes, but is not limited to, steps S110 to S170. The following is a detailed explanation... Figure 1 These seven steps will be explained in detail.

[0080] Step S110: Obtain sample data, which includes sample phoneme data and sample speech. The sample phoneme data is used to characterize the text content of the sample speech.

[0081] Step S120: Input the sample phoneme data into the preset original synthesis model. The original synthesis model includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model.

[0082] Step S130: The sample phoneme data is processed by phoneme encoding sub-model to obtain phoneme hidden data;

[0083] Step S140: Phoneme adaptation processing is performed on the phoneme hidden data through the variance adaptor sub-model to obtain phoneme alignment data and phoneme feature data.

[0084] Step S150: Perform spectral prediction processing on the phoneme alignment data and phoneme feature data using a noise reduction sub-model to obtain the predicted Mel spectrum;

[0085] Step S160: Adjust the parameters of the original synthesis model based on the sample speech and the predicted Mel spectrum to obtain the speech synthesis model;

[0086] Step S170: Input the acquired target text data into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0087] In steps S110 to S170 of some embodiments, firstly, sample data is acquired, including sample phoneme data and sample speech. The sample phoneme data is used to characterize the text content of the sample speech. Then, the sample phoneme data is input into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model. The phoneme encoding sub-model performs phoneme encoding processing on the sample phoneme data to obtain phoneme hidden data. The variance adaptation sub-model performs phoneme adaptation processing on the phoneme hidden data to obtain phoneme alignment data and phoneme feature data. The noise reduction sub-model performs spectral prediction processing on the phoneme alignment data and phoneme feature data to obtain a predicted Mel spectrum. Afterwards, the parameters of the original synthesis model are adjusted based on the sample speech and the predicted Mel spectrum to obtain a speech synthesis model. The acquired target text data is input into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech. The embodiments of this application predict the Mel spectrum based on phoneme alignment data and phoneme feature data, which can improve the generation quality and efficiency of synthesized speech and effectively simplify the computational load of the speech synthesis process.

[0088] In step S110 of some embodiments, in order to train the speech synthesis model, this application embodiment first obtains a training sample set, which includes at least one sample data, and each sample data includes sample phoneme data and sample speech. The sample speech is used to represent a reference speech for the sample phoneme data during model training. The sample phoneme data can be a text phoneme sequence obtained by converting the acquired initial text into text phonemes using a pre-set phoneme conversion model. To improve the generation efficiency of the speech synthesis model, sample phoneme data is used to characterize the text content of the sample speech; that is, the text content of the sample phoneme data and the speech content of the sample speech are the same. This application embodiment uses the sample speech as a contrast label for model prediction to guide the training process of the sample phoneme data in the model.

[0089] It should be noted that the phoneme conversion model used in the embodiments of this application can be constructed using model structures such as Deep Voice3 model and grapheme to phoneme (G2P), and no specific limitation is made here.

[0090] It should be noted that the storage format of the sample audio in this application may be MP3, CDA, WAV, WMA, RA, MIDI, OGG, APE or AAC, etc., and this application does not limit it.

[0091] It should be noted that the speech synthesis method of this application can be used to assist applications such as car radio and announcements, car navigation, electronic dictionaries, consumer electronics, smartphones, smart speakers, voice assistants, and e-book reading. For example, in the daily business consultation scenario of intelligent customer service, when the target person dials the phone, the intelligent customer service can generate the target synthesized speech based on the preset script text, and guide the target person to perform the corresponding operation by playing the synthesized speech, thereby obtaining the required information.

[0092] In step S120 of some embodiments, the related speech synthesis system includes an acoustic model and a vocoder. The acoustic model can map input text to speech features, and the vocoder can synthesize speech based on the speech features. After the text to be synthesized is input into the speech synthesis system, the acoustic model predicts speech features from the input text and inputs the predicted speech features into the vocoder. The vocoder is used to synthesize speech based on the obtained predicted speech features. However, due to the prediction loss of the acoustic model, there is a large mismatch between the predicted speech features received by the vocoder from the acoustic model and the real speech features. This results in the synthesized speech generated by the vocoder being less than ideal, often exhibiting obvious hoarseness or background noise. In order to predict high-quality synthesized speech and make the synthesized speech have diverse characteristics, the original synthesis model preset in this application includes a phoneme coding sub-model, a variance fitting sub-model, and a noise reduction sub-model.

[0093] It should be noted that the original synthesis model also includes a vocoder, which is used to synthesize speech based on the speech features of its input. The input of the vocoder can be the predicted Mel spectrum output by the noise reduction sub-model.

[0094] In step S130 of some embodiments, the phoneme coding sub-model is used to perform phoneme coding processing on the input sample phoneme data to obtain phoneme hidden data. This phoneme hidden data is used to characterize deeper feature information of the sample phoneme data.

[0095] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the specific method of step S130 provided in the embodiments of this application. In some embodiments of this application, the phoneme coding sub-model includes a phoneme convolutional layer, a phoneme self-attention layer, and a phoneme projection layer. Step S130 may specifically include, but is not limited to, steps S210 to S230. The following describes the process in conjunction with... Figure 2 These three steps will be explained in detail.

[0096] Step S210: Perform phoneme convolution processing on the sample phoneme data according to the phoneme convolution layer to obtain phoneme encoded data;

[0097] Step S220: Perform self-attention processing on the phoneme encoding data according to the phoneme self-attention layer to obtain phoneme attention data;

[0098] Step S230: Perform linear projection processing on the phoneme attention data according to the phoneme projection layer to obtain phoneme hidden data.

[0099] In some embodiments, steps S210 to S230, such as Figure 3 As shown, the phoneme coding sub-model 310 proposed in this embodiment consists of a model structure based on a feedforward converter architecture. The phoneme coding sub-model 310 includes a phoneme convolutional layer 311, a phoneme self-attention layer 312, and a phoneme projection layer 313. The phoneme convolutional layer 311 is used to extract key features from the sample phoneme data to eliminate the impact of noise or redundant features on speech synthesis efficiency. Specifically, the phoneme self-attention layer 312 applies different weights (attention scores) to different information that needs to be considered in solving a problem in a specific scenario. Higher weights are applied to information that is more helpful to the problem, and lower weights are applied to information that is less helpful, thereby better utilizing this information in the problem-solving process. When phoneme-encoded data undergoes self-attention processing in the phoneme self-attention layer 312, phoneme-encoded data that contributes significantly to recognizing emotional changes in the target object are assigned higher attention scores, while those that contribute less are assigned lower attention scores. Phoneme-encoded data with high attention scores correspond to more important sentences, and the emotions they convey are more representative of the target object's true emotions compared to other sentences. Therefore, this effectively improves the accuracy and efficiency of speech synthesis. Subsequently, the input phoneme attention data undergoes linear projection processing in the phoneme projection layer 313, and the resulting phoneme hidden data is sent to the variance adaptor sub-model 320. This embodiment integrates these feature data through a modality fusion network, effectively improving the diversity of synthesized speech.

[0100] It should be noted that, in order to focus more attention on the position of the phoneme encoding data itself, the phoneme self-attention layer proposed in this application embodiment can be constructed using a multi-head attention mechanism.

[0101] In step S140 of some embodiments, the variance adaptor model proposed in this application is used to predict the duration of each phoneme, so as to adjust the length of the phoneme hidden features to the length and dimension of the predicted speech. For example, if the feature data obtained after the phoneme coding sub-model is 80*70, and assuming that the two-dimensional matrix size of the Mel spectrum corresponding to the final output predicted speech is 80×140, then the variance adaptor model can also adjust the matrix of the phoneme hidden data obtained after processing by the phoneme coding sub-model to 80×140 to obtain the synthesized speech with the required matrix size.

[0102] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the specific method of step S140 provided in the embodiments of this application. In some embodiments of this application, the variance fitter sub-model includes a duration predictor, a pitch predictor, and an energy predictor. Therefore, step S140 may specifically include, but is not limited to, steps S410 to S440. The following describes the process in conjunction with... Figure 4 These four steps will be explained in detail.

[0103] Step S410: Perform phoneme alignment processing on the phoneme hidden data according to the duration predictor to obtain phoneme aligned data;

[0104] Step S420: Perform pitch prediction processing on the phoneme alignment data according to the pitch predictor to obtain the first conditional data;

[0105] Step S430: Perform energy prediction processing on the phoneme alignment data according to the energy predictor to obtain the second conditional data;

[0106] Step S440: Combine the first conditional data and the second conditional data to obtain phoneme feature data.

[0107] In steps S410 to S440 of some embodiments, in order to ensure that the synthesized speech has full energy and accurate pitch, i.e., to effectively improve the quality of speech synthesis, embodiments of this application employ a variance adaptor model to indicate different variances in the speech, such as energy variance and pitch variance. Specifically, as... Figure 5As shown, the phoneme hidden data is input into the variance adaptation sub-model 510. The phoneme hidden data is then processed by the duration predictor 511 to obtain phoneme aligned data. This phoneme aligned data is in matrix form, adjusting the length of the phoneme hidden features to match the length and dimension of the predicted speech. The phoneme aligned data is then processed by the pitch predictor 512 to obtain first conditional data, which characterizes the pitch conditional variance of the output predicted speech. The phoneme aligned data is then processed by the energy predictor 513 to obtain second conditional data, which characterizes the energy conditional variance of the output predicted speech. The first and second conditional data are combined to obtain phoneme feature data, which is used as a conditional parameter in the noise reduction sub-model 520 to synthesize a more accurate and higher-quality predicted Mel spectrum.

[0108] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating the specific method of step S420 provided in an embodiment of this application. In some embodiments of this application, the pitch predictor includes a pitch activation layer, a normalization layer, and a pitch projection layer. Therefore, step S420 may specifically include, but is not limited to, steps S610 to S630. The following describes the method in conjunction with... Figure 6 These three steps will be explained in detail.

[0109] Step S610: Perform non-linear processing on the phoneme alignment data according to the pitch activation layer to obtain pitch activation data;

[0110] Step S620: Normalize the pitch activation data according to the normalization layer to obtain normalized hidden data;

[0111] Step S630: Perform linear projection processing on the normalized hidden data according to the pitch projection layer to obtain the first conditional data.

[0112] In steps S610 to S630 of some embodiments, pitch is used to characterize the frequency of speech, and energy is used to characterize the intensity of speech. To more accurately estimate the pitch of phoneme-aligned data, the pitch predictor proposed in this application includes a pitch activation layer, a normalization layer, and a pitch projection layer. Specifically, the pitch activation layer consists of a Rectified Linear Unit (ReLU) and two layers of one-dimensional convolutional units. ReLU is a piecewise linear function; if the input is positive, the function outputs directly; otherwise, it outputs zero. ReLU makes the model easier to train and achieves better performance. The one-dimensional convolutional unit can be equivalent to a fully connected network, allowing the number of channels to be changed without altering the feature map size, thereby effectively enhancing the abstract representation ability of local network modules and improving the accuracy of pitch prediction. After the pitch activation layer, the pitch predictor also includes a normalization layer. The normalization layer is used to ensure that the result after convolution again satisfies a normal distribution, preventing gradient vanishing when input to the pitch projection layer. The pitch projection layer is used to project the normalized hidden data of the hidden state into the output sequence.

[0113] It should be noted that when a complex feedforward neural network is trained on a small dataset, or when it performs too well for certain data classifications, exhibiting a bias towards the training set, it is prone to overfitting, resulting in significant errors in the testing cycle. Therefore, to prevent overfitting, after the normalization layer, the pitch predictor proposed in this embodiment also includes a random dropout layer. This random dropout layer discards random data during training and sets the discarded random data to zero. By repeatedly doing this during training, overfitting during the training phase can be effectively prevented.

[0114] It should be noted that the duration predictor and energy predictor proposed in this application embodiment have similar model structures to the pitch predictor. However, in the specific training process, the duration predictor mainly trains to predict the duration features in the phoneme hidden features, and the energy predictor mainly trains to predict the phoneme energy features in the phoneme alignment data. This application embodiment uses a variance adaptor sub-model to perform phoneme adaptation processing on the phoneme hidden data, obtaining two matrices: phoneme alignment data and phoneme feature data. The phoneme alignment data is used to represent data with the same alignment form as the phoneme hidden data. The phoneme feature data is used to represent the conditional information required to generate the predicted Mel spectrum.

[0115] In step S150 of some embodiments, in order to iteratively refine the length-adjusted hidden data into predicted Mel spectra, the data output by the variance-fitting submodel is processed by the denoising submodel for spectrum prediction to obtain the predicted Mel spectra. In this embodiment, the sample phoneme data is transformed using the scaling of the Mel spectrum, and the resulting predicted Mel spectra can learn the nonlinear transformation of the spectrum.

[0116] Please refer to Figure 7 , Figure 7 This is a flowchart illustrating the specific method of step S150 provided in an embodiment of this application. In some embodiments of this application, step S150 may specifically include, but is not limited to, steps S710 to S740, as described below. Figure 7 These four steps will be explained in detail.

[0117] Step S710: Input the phoneme alignment data into the noise reduction sub-model, sample the phoneme alignment data, and obtain the candidate adaptation data and the position information of the candidate adaptation data.

[0118] Step S720: Perform spectrum diffusion processing on the candidate adaptation data according to the preset time step to obtain spectrum diffusion data;

[0119] Step S730: Perform spectrum inverse sampling on the spectrum diffusion data according to the preset time step, candidate adaptation data and phoneme feature data to obtain the predicted spectrum data;

[0120] Step S740: Generate a spectrum from the predicted spectrum data based on the location information to obtain the predicted Mel spectrum.

[0121] In steps S710 to S740 of some embodiments, a parameterized denoising sub-model is trained to directly predict clean data, thus avoiding the problems of significant data quality degradation and model convergence with fewer diffusion iterations during the accelerated sampling process. This application's embodiments are based on an improved version of the Progressive Fast Diffusion Model for High-Quality Text-to-Speech (ProDiff) to obtain the denoising sub-model, that is, by reducing the data variance of the predicted Mel spectrum through knowledge extraction. Specifically, the denoising sub-model of this application includes a model sampling layer and a model inverse sampling layer. In the model sampling layer, phoneme alignment data is first input into the denoising sub-model, and data sampling is performed on the phoneme alignment data to obtain candidate fitting data and the position information of the candidate fitting data. The denoising sub-model includes T preset time steps, which also represent the number of iterations of the model. The spectral diffusion process employs a Markov chain-like forward diffusion process. Over T time steps, a small amount of Gaussian noise is gradually added to the candidate fitting data, generating a series of noise samples, which constitute the spectral diffusion data. This spectral diffusion data follows a Gaussian distribution. During the spectral diffusion process, as the time step increases, the candidate fitting data gradually loses its distinguishable features. When T approaches infinity, the spectral diffusion data is equivalent to isotropic Gaussian distributed data. In the model inverse sampling layer, to accurately obtain the predicted Mel spectrum, this embodiment performs spectral inverse sampling on the spectral diffusion data based on a preset time step, candidate fitting data, and phoneme feature data. This accurately predicts the added noise data based on the candidate fitting data and performs a denoising process on the candidate fitting data using the preset time step and phoneme feature data to accurately derive the predicted spectral data. Finally, based on the location information of the candidate fitting data, the corresponding predicted spectral data is synthesized to obtain the predicted Mel spectrum. The embodiments of this application use a parameterized approach to sample candidate adaptation data according to a preset time step. By directly predicting clean data to parameterize the denoising model, the problem of significant data quality degradation during the accelerated sampling process is avoided.

[0122] Please refer to Figure 8 , Figure 8 This is a flowchart illustrating the specific method of step S720 provided in the embodiments of this application. In some embodiments of this application, step S720 may specifically include, but is not limited to, steps S810 to S840, as described below. Figure 8 These four steps will be explained in detail.

[0123] Step S810: Obtain the noise scheduling parameters for the preset time step;

[0124] Step S820: Sample the candidate adaptation data to obtain the first adaptation data;

[0125] Step S830: Noise is added to the first adaptation data according to the preset time step and noise scheduling parameters to obtain the second adaptation data;

[0126] Step S840: Obtain spectral diffusion data based on the first adaptation data and the second adaptation data.

[0127] In some embodiments, during steps S810 to S840, when performing spectrum spreading processing, noise scheduling parameters for a preset time step are first obtained, which can be denoted as β. t t represents the time step in the specific diffusion step, t∈[0,T]. This noise scheduling parameter is used to characterize the hyperparameter of the forward diffusion process. Data sampling is performed on the candidate adaptation data to obtain the first adaptation data, denoted as x0. As shown in the following formulas (1) and (2), the first adaptation data is noise-added according to the preset time step and noise scheduling parameter to obtain the second adaptation data, denoted as x1; and so on, the second adaptation data is used as the new first adaptation data to derive the data of the next time step, so as to derive the latent variable x. T .

[0128]

[0129]

[0130] Where Ι represents the identity matrix, x t This represents the adapted data after adding noise at time step t, x t-1 This represents the adapted data after adding noise at time step t-1, where q represents the data distribution at time step t, and N represents the preset spectral spread function, which can be a Gaussian distribution function.

[0131] It should be noted that, in the spectral inverse sampling process of this application, the pre-sampled first adaptation data x0 is used as a condition for the denoising process. This avoids the problem in existing technologies that use distillation learning as input for the denoising process, thereby reducing the synthesis efficiency of the model by calculating the predicted spectral data and loss data based on the obtained data. This application combines the spectral sampling process and the spectral inverse sampling process into one stage; that is, both processes can be performed based on the sampled candidate adaptation data, which can effectively improve the synthesis efficiency of speech synthesis.

[0132] It should be noted that for any time step t, the noise scheduling parameters used in this application can be calculated using a cosine scheduling function, as shown in the following formula (3).

[0133] β t=cos(0.5πt) (3)

[0134] In step S160 of some embodiments, this application embodiment obtains better quality synthesized speech by directly predicting the loss data of clean data. Specifically, when adjusting the parameters of the original synthesis model based on sample speech and predicted Mel spectrum, this application uses the initially extracted candidate adaptation data as conditional parameters. This avoids the problem in the prior art of using distillation learning as input for the denoising process, thereby reducing the synthesis efficiency of the model by calculating the predicted spectrum data and loss data based on the obtained data.

[0135] Please refer to Figure 9 , Figure 9 This is a flowchart illustrating the specific method of step S160 provided in an embodiment of this application. In some embodiments of this application, step S160 may specifically include, but is not limited to, steps S910 to S930, as described below. Figure 9 These three steps will be explained in detail.

[0136] Step S910: Calculate the diffusion parameters based on the noise scheduling parameters and the preset time step to obtain the diffusion process parameters;

[0137] Step S920: Obtain noise distribution data, and calculate the prediction loss based on the noise distribution data, candidate adaptation data, predicted spectrum data, preset time step, diffusion process parameters and phoneme feature data to obtain the prediction loss data;

[0138] Step S930: Adjust the parameters of the original synthesis model based on the predicted loss data to obtain the speech synthesis model.

[0139] In steps S910 to S930 of some embodiments, in order to directly predict the loss of the initial clean data, diffusion parameters are calculated based on noise scheduling parameters and preset time steps to obtain diffusion process parameters, denoted as α. t And the diffusion process parameters satisfy the following formula (4). Obtain the noise distribution data, denoted as ∈, where the noise distribution data ∈ is noise that satisfies a normal distribution. According to the following formula (5), the prediction loss is calculated based on the noise distribution data, candidate adaptation data, prediction spectrum data, preset time step, diffusion process parameters and phoneme feature data to obtain the prediction loss data L.

[0140]

[0141]

[0142] Where θ represents a shared parameter, which indicates that the spectral inverse sampling process is a Markov chain with shared parameters; x θThe time step parameter is θ; con represents the phoneme feature data, that is, the data of con includes the first conditional data obtained from pitch prediction and the second conditional data obtained from energy prediction.

[0143] It should be noted that a speech synthesis model is obtained when the adjusted parameters of the original synthesis model meet a preset termination condition. This preset termination condition can be achieved by calculating the similarity between sample speech and predicted Mel spectra. Specifically, the Mel spectra of the sample speech are obtained, and a similarity calculation is performed between these sample Mel spectra and the predicted Mel spectra to obtain spectral similarity data. When the spectral similarity data is greater than or equal to a preset similarity threshold, the current model training is considered complete.

[0144] It should be noted that the function used for similarity calculation can be selected according to actual needs, such as cosine similarity calculation, time axis comparison method, etc., and no specific limitation is made here.

[0145] In step S170 of some embodiments, in practical applications, this application is applied to a terminal. When a target object needs to perform speech synthesis on the terminal, it can select the text content to be synthesized on the terminal page, and a pop-up box can be displayed on the terminal page. The target object touches the synthesized speech button in the pop-up box, and at this time, the target text is sent to the speech synthesis system for speech synthesis processing through a speech synthesis service request. Afterwards, the speech is played through the terminal's speaker, so that the target object can hear the target synthesized speech that is the same as the target text content. For example, a text-to-speech speech synthesis system can be installed on a smartphone, and the speech synthesis model trained by this application is deployed in the speech synthesis system. When the operation of converting text to speech is detected, the smartphone generates a speech synthesis service request and sends the speech synthesis service request to the speech synthesis system. In response to the speech synthesis service request, the smartphone uses the speech synthesis system to extract the target text data from the speech synthesis service request, so as to synthesize the target synthesized speech from the target text data through the trained speech synthesis model.

[0146] It should be noted that, for example, in the business processing under fintech, the speech synthesis method provided in this application can be used to synthesize speech from pre-set process texts for different business processing procedures, generating multiple target synthesized voices for different business processing procedures. Specifically, after recognizing that the target object's need is "Apply for A-card", the target synthesized voice can be matched and selected to guide the processing of this business based on this need. Specifically, after recognizing that the target object has completed one step, the guiding voice for the next step of processing the business is played, thereby guiding the target object to complete the business processing by playing the target synthesized voice.

[0147] Please refer to Figure 10 , Figure 10This is a schematic diagram of the module structure of a speech synthesis system provided in an embodiment of this application. In some embodiments of this application, the speech synthesis system includes a sample acquisition module 1010, a model input module 1020, a phoneme encoding module 1030, a phoneme adaptation module 1040, a spectrum prediction module 1050, a parameter adjustment module 1060, and a speech synthesis module 1070.

[0148] The sample acquisition module 1010 is used to acquire sample data, which includes sample phoneme data and sample speech. The sample phoneme data is used to characterize the text content of the sample speech.

[0149] The model input module 1020 is used to input sample phoneme data into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model.

[0150] The phoneme encoding module 1030 is used to perform phoneme encoding processing on the sample phoneme data through the phoneme encoding sub-model to obtain phoneme hidden data.

[0151] The phoneme adaptation module 1040 is used to perform phoneme adaptation processing on the phoneme hidden data through the variance adaptation sub-model to obtain phoneme alignment data and phoneme feature data.

[0152] The spectrum prediction module 1050 is used to perform spectrum prediction processing on phoneme alignment data and phoneme feature data through a noise reduction sub-model to obtain the predicted Mel spectrum.

[0153] The parameter adjustment module 1060 is used to adjust the parameters of the original synthesis model based on the sample speech and the predicted Mel spectrum to obtain the speech synthesis model.

[0154] The speech synthesis module 1070 is used to input the acquired target text data into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

[0155] It should be noted that the speech synthesis system in this application embodiment is used to execute the above-described speech synthesis method, and the speech synthesis system in this application embodiment corresponds to the aforementioned speech synthesis method. For the specific training process, please refer to the aforementioned speech synthesis method, which will not be described in detail here.

[0156] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speech synthesis method described in the embodiments of this application.

[0157] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0158] The following is combined Figure 11 The electronic devices described in the embodiments of this application will be described in detail.

[0159] Please refer to Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0160] The processor 1110 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0161] The memory 1120 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1120 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1120 and is called and executed by the processor 1110 using the speech synthesis method of the embodiments of this application.

[0162] The input / output interface 1130 is used to implement information input and output;

[0163] The communication interface 1140 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0164] Bus 1150 transmits information between various components of the device (e.g., processor 1110, memory 1120, input / output interface 1130, and communication interface 1140);

[0165] The processor 1110, memory 1120, input / output interface 1130 and communication interface 1140 are connected to each other within the device via bus 1150.

[0166] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech synthesis method described in the embodiments of this application.

[0167] This application provides a speech synthesis method, speech synthesis system, electronic device, and storage medium. By improving the ProDiff model, specifically by directly predicting the loss based on initial data, the quality of synthesized speech can be effectively improved. Furthermore, this application employs a parameterized approach to sample candidate adaptation data according to a preset time step and parameterizes the denoising model by directly predicting clean data, avoiding the significant degradation of data quality during accelerated sampling. This application also improves the diversity of speech synthesis by predicting Mel-spectrum data based on phoneme alignment data and phoneme feature data. By reducing the number of sampling steps to single digits, resource consumption is greatly reduced, effectively simplifying the computational load of the speech synthesis process and thus significantly improving the efficiency of synthesized speech generation.

[0168] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0169] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0170] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0173] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0174] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Acquire sample data, which includes sample phoneme data and sample speech, wherein the sample phoneme data is used to characterize the text content of the sample speech; The sample phoneme data is input into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model. The sample phoneme data is processed by phoneme encoding sub-model to obtain phoneme hidden data; The phoneme hidden data is processed by the variance adaptor sub-model to obtain phoneme alignment data and phoneme feature data. The phoneme alignment data is input into the noise reduction sub-model, and the phoneme alignment data is sampled to obtain candidate adaptation data and the position information of the candidate adaptation data. The candidate adaptation data is then subjected to spectral diffusion processing according to a preset time step to obtain spectral diffusion data. The spectral diffusion data is then subjected to spectral inverse sampling processing according to the preset time step, the candidate adaptation data, and the phoneme feature data to obtain predicted spectral data. Finally, the predicted spectral data is used to generate a spectrum based on the position information to obtain a predicted Mel spectrum. The diffusion process parameters are calculated based on the noise scheduling parameters and the preset time step to obtain the diffusion process parameters; noise distribution data is acquired, and prediction loss is calculated based on the noise distribution data, the candidate adaptation data, the predicted spectrum data, the preset time step, the diffusion process parameters, the phoneme feature data, and the shared parameters to obtain prediction loss data. The shared parameters are used to represent that the spectrum inverse sampling process is a Markov chain with shared parameters; the parameters of the original synthesis model are adjusted based on the prediction loss data to obtain the speech synthesis model. The acquired target text data is input into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

2. The method according to claim 1, characterized in that, The step of performing spectral diffusion processing on the candidate adaptation data according to a preset time step to obtain spectral diffusion data includes: Obtain the noise scheduling parameters for the preset time step; The candidate adaptation data is sampled to obtain the first adaptation data; The first adaptation data is noise-added according to the preset time step and the noise scheduling parameters to obtain the second adaptation data; The spectral diffusion data is obtained based on the first adaptation data and the second adaptation data.

3. The method according to any one of claims 1 to 2, characterized in that, The variance-adaptive sub-model includes a duration predictor, a pitch predictor, and an energy predictor. The process of performing phoneme adaptation on the phoneme hidden data using the variance adaptor sub-model to obtain phoneme alignment data and phoneme feature data includes: The phoneme hidden data is processed by the duration predictor to obtain the phoneme aligned data. The phoneme alignment data is processed by the pitch predictor to obtain the first conditional data. The energy prediction data of the phoneme alignment data is processed by the energy predictor to obtain the second conditional data. The phoneme feature data is obtained by combining the first conditional data and the second conditional data.

4. The method according to claim 3, characterized in that, The pitch predictor includes a pitch activation layer, a normalization layer, and a pitch projection layer; The step of performing pitch prediction processing on the phoneme alignment data based on the pitch predictor to obtain first conditional data includes: The phoneme alignment data is processed non-linearly based on the pitch activation layer to obtain pitch activation data; The pitch activation data is normalized according to the normalization layer to obtain normalized hidden data; The normalized hidden data is linearly projected based on the pitch projection layer to obtain the first conditional data.

5. The method according to any one of claims 1 to 2, characterized in that, The phoneme encoding sub-model includes a phoneme convolutional layer, a phoneme self-attention layer, and a phoneme projection layer; The step of performing phoneme encoding processing on the sample phoneme data through the phoneme encoding sub-model to obtain phoneme hidden data includes: The sample phoneme data is processed by phoneme convolution layer to obtain phoneme encoded data. The phoneme encoding data is processed by the phoneme self-attention layer to obtain phoneme attention data. The phoneme attention data is linearly projected onto the phoneme projection layer to obtain the phoneme hidden data.

6. A speech synthesis system, characterized in that, The system includes: A sample acquisition module is used to acquire sample data, which includes sample phoneme data and sample speech. The sample phoneme data is used to characterize the text content of the sample speech. The model input module is used to input the sample phoneme data into a preset original synthesis model, which includes a phoneme encoding sub-model, a variance adaptation sub-model, and a noise reduction sub-model. The phoneme encoding module is used to perform phoneme encoding processing on the sample phoneme data through the phoneme encoding sub-model to obtain phoneme hidden data; The phoneme adaptation module is used to perform phoneme adaptation processing on the phoneme hidden data through the variance adaptation sub-model to obtain phoneme alignment data and phoneme feature data. The spectrum prediction module is used to input the phoneme alignment data into the noise reduction sub-model, sample the phoneme alignment data to obtain candidate adaptation data and the position information of the candidate adaptation data; perform spectrum diffusion processing on the candidate adaptation data according to a preset time step to obtain spectrum diffusion data; perform spectrum inverse sampling processing on the spectrum diffusion data according to the preset time step, the candidate adaptation data and the phoneme feature data to obtain predicted spectrum data; and generate a spectrum from the predicted spectrum data according to the position information to obtain a predicted Mel spectrum. The parameter adjustment module is used to calculate diffusion parameters based on the noise scheduling parameters and the preset time step to obtain diffusion process parameters; acquire noise distribution data, and calculate prediction loss based on the noise distribution data, the candidate adaptation data, the predicted spectrum data, the preset time step, the diffusion process parameters, the phoneme feature data, and shared parameters to obtain prediction loss data, wherein the shared parameters are used to represent that the spectrum inverse sampling process is a Markov chain with shared parameters; and adjust the parameters of the original synthesis model based on the prediction loss data to obtain a speech synthesis model. The speech synthesis module is used to input the acquired target text data into the speech synthesis model for speech synthesis processing to obtain the target synthesized speech.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images