Speech synthesis method, device, equipment, medium and product

Abstract

The invention provides a speech synthesis method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text; performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is obtained by performing multi-stage prediction task training according to a phoneme sequence and a discrete acoustic feature sequence corresponding to the training text, and a multi-stage prediction task comprises a discrete acoustic feature sequence prediction task and a Mel spectrum feature prediction task. The quality of synthesized speech can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech data processing, and in particular to a speech synthesis method, device, equipment, medium and product. Background Art

[0002] Speech synthesis, also known as text-to-speech (TTS), aims to convert input text into fluent and natural output speech, and is a key technology for realizing intelligent human-computer speech interaction. The current mature TTS system can generate high-quality audio for a single speaker or multiple speakers in a set, but the effect is poor when processing unknown speakers (i.e., speakers outside the set).

[0003] Although existing TTS systems perform well in specific scenarios, their performance significantly decreases when faced with diverse speech requirements, especially when dealing with unknown speakers. The large speech model based on the codec structure surpasses the traditional baseline effect in zero-sample synthesis, demonstrating its potential for dealing with unknown speakers. However, its synthesized speech may have problems such as muteness and unnatural rhythm, resulting in poor quality of synthesized speech. Summary of the invention

[0004] Based on the above technical status, the present application provides a speech synthesis method, device, equipment, medium and product, which can improve the quality of synthesized speech.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a speech synthesis method is provided, comprising: obtaining a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text; performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is trained by performing a multi-stage prediction task according to the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction task includes a discrete acoustic feature sequence prediction task and a Mel-spectrogram feature prediction task.

[0007] In some implementations, the phoneme sequence includes individual phoneme units, and the discrete acoustic feature sequence includes individual discrete acoustic feature units; wherein, performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence using a pre-trained acoustic model to obtain a speech signal corresponding to the target text includes: inputting the phoneme sequence and the discrete acoustic feature sequence into the pre-trained acoustic model, and determining the correspondence between the individual phoneme units and the individual discrete acoustic feature units through the acoustic model; and generating a speech signal corresponding to the target text according to the correspondence between the individual phoneme units and the individual discrete acoustic feature units.

[0008] In some implementations, the acoustic model includes an encoder and a decoder; wherein, determining the correspondence between each phoneme unit and each discrete acoustic feature unit through the acoustic model includes: inputting the phoneme sequence into the encoder, and the encoder generating a phoneme hidden layer representation sequence according to the phoneme sequence, and the phoneme hidden layer representation sequence includes the phoneme hidden layer representation corresponding to each phoneme unit; inputting the phoneme hidden layer representation corresponding to each phoneme unit and the discrete acoustic feature sequence into the decoder, and the decoder aligning the phoneme hidden layer representation corresponding to each phoneme unit and the discrete acoustic feature unit to determine the correspondence between each phoneme unit and the discrete acoustic feature unit.

[0009] In some implementations, the training process of the acoustic model includes the following steps: obtaining a phoneme sequence and a discrete acoustic feature sequence corresponding to a training text; using the phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the previous t frames as training samples, and using the discrete acoustic feature prediction task of the t+1th frame as a training target, training the acoustic model to be trained to obtain an initial acoustic model, where t is a positive integer; using the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text as training samples, and using the mel-spectrogram feature prediction task as a training target, continuing to train the initial acoustic model to obtain the acoustic model.

[0010] In some implementations, the acoustic model to be trained is trained using the phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the previous t frames as training samples, and the discrete acoustic feature prediction task of the t+1th frame as a training target to obtain an initial acoustic model, including: generating predicted discrete acoustic features of the t+1th frame according to the phoneme sequence and the discrete acoustic feature sequence of the previous t frames through the acoustic model to be trained; determining the acoustic feature prediction loss according to the difference between the predicted discrete acoustic features of the t+1th frame and the actual discrete acoustic features of the t+1th frame, and converging the acoustic model to be trained based on the acoustic feature prediction loss to obtain an initial acoustic model.

[0011] In some implementations, the initial acoustic model is continuously trained with the phoneme sequence and discrete acoustic feature sequence corresponding to the training text as training samples and the mel-spectrogram feature prediction task as a training target to obtain the acoustic model, including: generating a predicted mel-spectrogram feature sequence according to the phoneme sequence and the discrete acoustic feature sequence through the acoustic model to be trained; determining a mel-spectrogram feature prediction loss according to the difference between the predicted mel-spectrogram feature sequence and a true mel-spectrogram feature sequence, and continuing to train the initial acoustic model based on the mel-spectrogram feature prediction loss to obtain the acoustic model.

[0012] In some implementations, the method further includes: generating a predicted phoneme sequence according to the phoneme sequence and the discrete acoustic feature sequence through the acoustic model to be trained; wherein, continuing to train the initial acoustic model based on the Mel-spectrogram feature prediction loss to obtain the acoustic model includes: determining the phoneme prediction loss according to the difference between the predicted phoneme sequence and the phoneme sequence; determining the comprehensive loss based on the phoneme prediction loss and the Mel-spectrogram feature prediction loss; and continuing to train the initial acoustic model according to the comprehensive loss to obtain the acoustic model.

[0013] According to a second aspect of an embodiment of the present application, a speech synthesis device is provided, comprising: an acquisition unit, for acquiring a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text; a speech synthesis unit, for performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is trained by performing a multi-stage prediction task based on the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction task includes a discrete acoustic feature sequence prediction task and a Mel-spectrogram feature prediction task.

[0014] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the speech synthesis method as described in the first aspect by running the program in the memory.

[0015] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method as described in the first aspect is implemented.

[0016] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, cause the processor to execute: the speech synthesis method as described in the first aspect.

[0017] The embodiments of the present application provide a method, apparatus, device, medium and product for speech synthesis, which obtains a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text, and uses a pre-trained acoustic model to perform speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence to obtain a speech signal corresponding to the target text. The acoustic model is trained by performing multi-stage prediction tasks according to the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction tasks include discrete acoustic feature sequence prediction tasks and Mel-spectrogram feature prediction tasks. Since the present application uses multi-stage prediction tasks to train the acoustic model, the complex task is decomposed into multiple gradually optimized sub-tasks, which significantly reduces the difficulty of modeling at each stage, thereby improving the stability of speech synthesis. When speech synthesis is performed using the trained acoustic model, the quality of the synthesized speech can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of a large speech model provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the framework of the speech synthesis model provided in the embodiment of the present application;

[0022] Figure 4 A flowchart of determining a speech signal corresponding to a target text provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the framework of the acoustic model provided in the embodiment of the present application;

[0024] Figure 6 A flowchart of the acoustic model training process provided in the embodiment of the present application;

[0025] Figure 7 A schematic diagram of the first stage training process of the acoustic model provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of the second stage training process of the acoustic model provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application;

[0028] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solution of the embodiment of the present application is applicable to a wide range of speech synthesis task scenarios to improve the speech synthesis quality and user experience in these scenarios. The specific scenarios are as follows:

[0030] Intelligent voice assistants convert text information into voice signals through speech synthesis technology, so that users can interact through voice to obtain information or perform operations; screen readers convert text content on the screen into voice signals through speech synthesis technology to help visually impaired people read the text content on the screen and provide barrier-free access; audio books are produced through speech synthesis technology to facilitate users to listen to books while driving, exercising, etc., and improve reading comprehension ability; voice mail and voice messages convert text mail or SMS into audio through speech synthesis technology to facilitate users to listen; voice passwords convert user-entered passwords into audio through speech synthesis technology to improve the security of password storage and transmission; virtual human anchors, automated customer service systems based on speech synthesis interact with users by generating natural language voice, replacing manual customer service and providing efficient services; car navigation systems convert text information into voice broadcasts to ensure driving safety; voice games and entertainment enhance user experience by providing various speech synthesis effects.

[0031] By adopting the technical solution of the embodiments of the present application, the quality of speech synthesis in the above-mentioned application scenarios is significantly improved, thereby providing users with a smoother, more natural and safer interactive experience.

[0032] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] Before introducing this application solution, the relevant technologies are first introduced:

[0035] At present, mature text-to-speech (TTS) systems are able to generate high-quality audio for a single speaker or multiple speakers in a set in specific scenarios. However, for unknown speakers (i.e., speakers outside the set), the synthesis effect is significantly reduced. High-quality speech synthesis requires high training data, and the recording and collection of high-quality data is not only time-consuming and labor-intensive, but may also be subject to resource constraints. Although the use of techniques such as speaker adaptation can improve the synthesis effect of speakers outside the set to a certain extent, this usually requires a complex model architecture, additional model fine-tuning, and carefully designed feature engineering.

[0036] In recent years, large speech models based on codec structures have made significant progress in zero-shot synthesis, surpassing traditional baselines and demonstrating their great potential in processing unknown speakers. These models draw on the design ideas of large language models (LLMs) and effectively improve the generalization and stability of the models by using discrete acoustic features as intermediate representations. Specifically, large speech models perform well in zero-shot synthesis tasks with the help of intermediate representations of discrete acoustic features, and can better capture the subtle differences in speech and generate more natural and smooth audio.

[0037] The current mainstream speech model solution adopts the neural codec language model (VALL-E) structure, such as Figure 1As shown in Figure 2. The model concatenates the phoneme sequence and the discrete acoustic feature sequence as the input data of the encoder, and uses the discrete acoustic features as the training target. The encoder uses a combination of autoregressive and non-autoregressive methods to make predictions to improve the accuracy and efficiency of the predictions. Subsequently, the decoder converts the predicted discrete acoustic features into the final speech waveform, ensuring that the generated audio is both high-quality and natural.

[0038] Although large speech models based on codec structures perform well in zero-shot synthesis, the task of directly recovering speech waveforms from discrete acoustic feature sequences remains extremely challenging. This process is difficult, resulting in difficulties in model convergence, which in turn affects the stability and reliability of the synthesis system. Common problems include muteness (some parts have no sound at all), unnatural rhythm (intonation, rhythm, etc. do not sound very coherent), etc. These problems not only reduce the user experience, but also limit the widespread promotion of the model in practical applications.

[0039] In view of this, the embodiments of the present application are dedicated to providing a speech synthesis method, device, equipment, medium and product, which trains an acoustic model through multi-stage training tasks, reduces the difficulty of model convergence, improves the stability of speech synthesis, and then performs speech synthesis through the trained acoustic model, thereby improving the quality of synthesized speech. The following embodiments will be described in detail one by one.

[0040] Exemplary Methods

[0041] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of the present application. Figure 2 As shown, the speech synthesis method provided in this embodiment includes step S201 and step S202:

[0042] S201. Obtain a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text.

[0043] In the speech synthesis system, step S201 is the key starting point of the entire process, which aims to extract two important components from the target text: phoneme sequence and discrete acoustic feature sequence. These two sequences are the basis of speech synthesis and can ensure that the generated speech is both accurate and natural.

[0044] Phoneme sequence refers to converting input text into the basic sound units (i.e., phoneme units) that constitute a language. Phoneme sequence includes various phoneme units, each of which can be a single initial consonant or a single final vowel. For Chinese text, obtaining the phoneme sequence corresponding to the target text includes: converting the target text into pinyin; further splitting the obtained pinyin into initial consonants and finals, and mapping the split pinyin to specific phonemes.

[0045] Specifically, the target text can be converted into pinyin through a pinyin conversion tool or API to ensure that each Chinese character is correctly mapped to the corresponding pinyin. Each initial or final sound in the split pinyin is a phoneme unit, and these phoneme units form the final phoneme sequence, which is used to guide the subsequent speech synthesis process. For example, "你好" (nǐ hǎo) can be converted to "nǐhǎo" and then split into "nǐ" and "hǎo". Next, "nǐ" is mapped to the phoneme "n i3", and "hǎo" is mapped to "h ao3", obtaining the phoneme sequence "n i3 h ao3" corresponding to the target text.

[0046] Discrete acoustic features are quantitative information that describes the sound characteristics of the speech data corresponding to the target text. To obtain a high-quality discrete acoustic feature sequence, it can be extracted by the acoustic feature prediction module in the speech synthesis large model for the target text.

[0047] Figure 3 This is a schematic framework diagram of the speech synthesis large model provided by the embodiments of this application. As Figure 3 shown, the speech synthesis large model includes: an acoustic feature prediction module 31 and a speech synthesis module 32; among them, the acoustic feature prediction module 31 is used to predict a discrete acoustic feature sequence for the target text; the speech synthesis module 32 is used to perform speech synthesis based on the phoneme sequence and the predicted discrete acoustic feature sequence to obtain the speech signal corresponding to the target text.

[0048] Among them, the acoustic feature prediction module 31 can be a Generative Pre-trained Transformer (GPT). By using the training text and the discrete acoustic feature sequence of the first t frames as input data, and taking the discrete acoustic feature of the (t + 1)-th frame corresponding to the training text as the label, the acoustic feature prediction module 31 to be trained is pre-trained.

[0049] The label can be obtained by extracting acoustic features from the real speech data corresponding to the training text through acoustic feature extraction models such as VQ-VAE (Vector Quantized Variational Autoencoder) and RVQ (Residual Vector Quantization).

[0050] S202. Perform speech synthesis based on the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain the speech signal corresponding to the target text.

[0051] Among them, the acoustic model is trained by performing multi-stage prediction tasks based on the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text. The multi-stage prediction tasks include a discrete acoustic feature sequence prediction task and a Mel spectrum feature prediction task.

[0052] Figure 4 A flowchart for determining a speech signal corresponding to a target text provided in an embodiment of the present application.

[0053] like Figure 4 As shown, step S202 includes the following steps S401 and S402:

[0054] S401: Input the phoneme sequence and the discrete acoustic feature sequence into a pre-trained acoustic model, and determine the correspondence between each phoneme unit and each discrete acoustic feature unit through the acoustic model.

[0055] In step S401, the acoustic model receives a phoneme sequence and a discrete acoustic feature sequence as input, and determines the correspondence between each phoneme unit in the phoneme sequence and each discrete acoustic feature unit in the discrete acoustic feature sequence. Each phoneme unit can be understood as a token. Each discrete acoustic feature unit can also be understood as a token. This step is explained in detail below in conjunction with the framework diagram of the acoustic model:

[0056] Figure 5 The framework diagram of the speech synthesis module provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, the speech synthesis module 32 includes: an acoustic model 321 and a vocoder 322; wherein the acoustic model 321 includes an encoder 3211 and a decoder 3212; the encoder 3211 is used to generate a phoneme hidden layer representation sequence according to a phoneme sequence, and the phoneme hidden layer representation sequence includes a phoneme hidden layer representation corresponding to each phoneme unit; the decoder 3212 is used to align each phoneme hidden layer representation and each discrete acoustic feature unit, determine the correspondence between each phoneme hidden layer representation and each discrete acoustic feature unit, and predict the Mel-spectrogram feature sequence corresponding to the target text based on the correspondence; the vocoder 322 is used to generate a speech signal according to the predicted Mel-spectrogram feature sequence.

[0057] Specifically, step S401 includes the following steps a1 and a2:

[0058] Step a1: input the phoneme sequence into the encoder, and the encoder generates a phoneme hidden layer representation sequence according to the phoneme sequence. The phoneme hidden layer representation sequence includes the phoneme hidden layer representation corresponding to each phoneme unit.

[0059] Specifically, the encoder 3211 receives the phoneme sequence corresponding to the target text as input, and processes the phoneme sequence to generate the phoneme hidden layer representation corresponding to each phoneme unit. These phoneme hidden layer representations capture the semantic information and contextual dependencies of the phonemes, providing a basis for subsequent alignment and prediction. The phoneme hidden layer representation sequence includes the phoneme hidden layer representation corresponding to each phoneme unit, for example:

[0060] {phoneme hidden layer representation sequence} = {h1, h2, ..., h t};

[0061] Among them, ht represents the phoneme hidden layer representation corresponding to the t-th phoneme unit.

[0062] Step a2: input the phoneme hidden layer representation and discrete acoustic feature sequence corresponding to each phoneme unit into the decoder, and the decoder aligns the phoneme hidden layer representation and each discrete acoustic feature unit corresponding to each phoneme unit to determine the corresponding relationship between each phoneme unit and each discrete acoustic feature unit.

[0063] The decoder 3212 can align the phoneme hidden layer representation sequence with the discrete acoustic feature sequence through techniques such as the cross attention mechanism or the duration hard alignment method, thereby ensuring that each phoneme unit can accurately correspond to the corresponding discrete acoustic feature unit, establishing a mapping relationship between the two, and providing a basis for predicting the Mel-spectrogram features.

[0064] Through the above steps, step S401 achieves accurate alignment between the phoneme unit and the discrete acoustic feature unit, and provides a high-quality intermediate representation for subsequent speech waveform generation. This not only improves the quality of the synthesized speech, but also enhances the stability and naturalness of the system.

[0065] Continue reading Figure 4 , after step S401, step S402 may also be included.

[0066] S402: Generate a speech signal corresponding to the target text according to the correspondence between each phoneme unit and each discrete acoustic feature unit.

[0067] In step S402, a mel-spectrogram feature sequence corresponding to the target text can be further generated according to the correspondence between each phoneme unit and each discrete acoustic feature unit determined in the previous step S401. This process is implemented by the decoder 3212, which predicts the mel-spectrogram feature sequence in a non-autoregressive manner based on the phoneme hidden layer representation sequence and the discrete acoustic feature sequence.

[0068] Specifically, the decoder 3212 receives the phoneme hidden layer representation sequence generated by the encoder 3211 as one of the input data. These representations capture the semantic information of the phonemes and their contextual dependencies, providing rich background information for generating high-quality mel-spectrogram features. At the same time, the decoder 3212 also receives a discrete acoustic feature sequence as input data, and predicts the mel-spectrogram features of the entire sequence based on the phoneme hidden layer representation sequence and the discrete acoustic feature sequence. The discrete acoustic feature sequence can help the model understand the continuity and dependency in time, ensuring that the generated mel-spectrogram feature sequence has good coherence and naturalness.

[0069] After obtaining the Mel-spectrogram feature sequence, it can then be converted into a final audio waveform signal through the vocoder 322 using waveform generation technology, such as the Griffin-Lim algorithm, WaveNet, HiFi-GAN, etc., thereby restoring the spectrum information into an audio waveform in the time domain, ensuring that the generated speech is both fidelity and natural.

[0070] In this embodiment, an encoder and decoder structure is adopted. Since the encoder's task is simplified to only process the phoneme sequence, and the decoder focuses on the prediction of discrete acoustic features, the burden of a single module can be reduced, so that each module can complete its specific task more efficiently, thereby improving the speed of speech synthesis and achieving real-time speech synthesis.

[0071] The above embodiment introduces a method for implementing speech synthesis by using an acoustic model. Before the acoustic model is applied to the speech synthesis method, it needs to be trained. The following will introduce a method for training the acoustic model in conjunction with the accompanying drawings.

[0072] Figure 6 The following is a flow chart of the acoustic model training process provided in the embodiment of the present application. Figure 6 As shown, the acoustic model is trained through the following steps S601 to S603:

[0073] S601: Obtain a phoneme sequence and a discrete acoustic feature sequence corresponding to a training text.

[0074] In step S601, a data set for training an acoustic model needs to be prepared, specifically including a phoneme sequence and a discrete acoustic feature sequence corresponding to a training text.

[0075] The phoneme sequence corresponding to the training text can be obtained by converting the training text into pinyin, further splitting the obtained pinyin into initial consonants and finals, and mapping the split pinyin to specific phonemes.

[0076] The discrete acoustic feature sequence corresponding to the training text can be obtained from the above-mentioned trained acoustic feature prediction module 31, or can be obtained by extracting acoustic features from real speech data corresponding to the training text through an acoustic feature extraction model.

[0077] S602. The phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the first t frames are used as training samples, and the discrete acoustic feature prediction task of the t+1th frame is used as the training target to train the acoustic model to obtain an initial acoustic model, where t is a positive integer.

[0078] In step S602, the phoneme sequence corresponding to the training text and the discrete acoustic features of the first t frames are used as input, and the acoustic model is trained with the goal of predicting the discrete acoustic features of the t+1th frame. Specifically, the following steps b1 and b2 are included:

[0079] Step b1: Generate predicted discrete acoustic features of the t+1th frame according to the phoneme sequence and the discrete acoustic feature sequence of the previous t frames through the acoustic model to be trained.

[0080] Input the phoneme sequence and the historical discrete acoustic feature sequence of the previous t frames into the acoustic model to be trained. These input data can provide contextual information before the current moment, helping the model understand temporal dependencies. The model uses autoregression to predict subsequent discrete acoustic features frame by frame based on the input phoneme sequence and historical discrete acoustic feature sequence. For example, for the prediction of the t+1th frame, the specific formula is as follows:

[0081] p(c t+1 ∣c< t+1 ; h; θ);

[0082] In the formula, c t+1 represents the discrete acoustic features of the t+1th frame; c< t+1 represents all historical discrete acoustic feature sequences before the t+1th frame; h represents the phoneme sequence; θ represents the model parameters.

[0083] The model outputs the predicted discrete acoustic features c of the t+1th frame t+1 , which represents the model's estimate of the discrete acoustic features of the next frame.

[0084] Figure 7 Schematic diagram of the first stage training process of the acoustic model provided in the embodiment of the present application. Figure 7 As shown, the encoder 3211 generates a phoneme hidden layer representation sequence for the phoneme sequence, and inputs it together with the historical discrete acoustic feature sequence of the previous t frames into the decoder 3212. The decoder 3212 predicts the discrete acoustic features of the t+1th frame based on the phoneme hidden layer representation sequence and the historical discrete acoustic feature sequence of the previous t frames.

[0085] Step b2: determine the acoustic feature prediction loss based on the difference between the predicted discrete acoustic features of the t+1th frame and the actual discrete acoustic features of the t+1th frame, and converge the acoustic model to be trained based on the acoustic feature prediction loss to obtain an initial acoustic model.

[0086] In step b2, the acoustic feature prediction loss may be calculated based on a loss function such as cross entropy loss, mean square error (MSE), etc., and the difference between the predicted discrete acoustic features of the t+1th frame and the true discrete acoustic features of the t+1th frame.

[0087] After that, the model parameter θ is adjusted according to the calculated acoustic feature prediction loss to minimize the acoustic feature prediction error. This process is usually implemented through the backpropagation algorithm (Backpropagation) and optimizer (such as Adam, SGD). The above process is iterated continuously until the model converges, that is, the loss function no longer decreases significantly. At this point, a preliminarily trained acoustic model is obtained, which is called the initial acoustic model.

[0088] The above embodiment introduces that step S602 uses an autoregressive method to predict a discrete acoustic feature sequence. In some embodiments, a non-autoregressive method can also be used to predict a discrete acoustic feature sequence, that is: the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text are used as training samples, and the discrete acoustic feature prediction task is used as the training target, the acoustic model to be trained is trained, and an initial acoustic model is obtained.

[0089] After the first stage of training is completed through steps b1 and b2 included in S602, an initial acoustic model that can effectively predict discrete acoustic feature sequences can be obtained. In order to improve the performance of the model in a specific application scenario, the initial acoustic model can be trained in the second stage, i.e., step S603, to further fine-tune and optimize the model through more sophisticated tasks.

[0090] S603: Using the phoneme sequence and discrete acoustic feature sequence corresponding to the training text as training samples and the mel spectrum feature prediction task as a training target, the initial acoustic model is continuously trained to obtain an acoustic model.

[0091] Specifically, step S603 includes the following steps c1 and c2:

[0092] Step c1: Generate a predicted Mel-spectrogram feature sequence according to the phoneme sequence and the discrete acoustic feature sequence through the acoustic model to be trained.

[0093] The goal of step c1 is to use the existing initial acoustic model to generate a more realistic speech signal. It involves taking the phoneme sequence and discrete acoustic feature sequence as input to predict the Mel-spectrogram feature sequence. Specifically, it is necessary to prepare the phoneme sequence and discrete acoustic feature sequence corresponding to the training text. These data can provide the model with necessary context information, which helps it make more accurate predictions. Afterwards, the above input data is processed using the initial acoustic model to predict the Mel-spectrogram feature sequence y t ', and synthesizes an audio waveform through the Mel spectrum feature sequence to imitate real human speech as much as possible.

[0094] Figure 8 Schematic diagram of the second stage training process of the acoustic model provided in the embodiment of the present application. Figure 8 As shown, the encoder 3211 receives a phoneme sequence as input data and generates a phoneme hidden layer representation sequence, which is then input into the decoder 3212 together with a discrete acoustic feature sequence. The decoder 3212 predicts a mel-spectrogram feature sequence based on the phoneme hidden layer representation sequence and the discrete acoustic feature sequence.

[0095] Step c2: Determine the mel-spectrogram feature prediction loss based on the difference between the predicted mel-spectrogram feature sequence and the true mel-spectrogram feature sequence, and continue to train the initial acoustic model based on the mel-spectrogram feature prediction loss to obtain the acoustic model.

[0096] After getting the predicted Mel spectrum feature sequence y t ', then we need to evaluate its accuracy and adjust the model parameters according to the evaluation results to improve the model performance. Specifically, we can use loss functions such as mean square error (MSE), L1 distance or L2 distance to calculate the predicted Mel spectrum feature sequence y t ' and the corresponding true Mel spectrum feature sequence y t * is used to define the Mel spectrum feature prediction loss The details are as follows:

[0097]

[0098] In the formula, y t * is the target of model learning, i.e., the true Mel spectrum feature sequence, which can be obtained by the Mel spectrum feature extractor; y t ' is the Mel spectrum feature sequence predicted by the model, that is, the predicted Mel spectrum feature sequence.

[0099] Finally, according to the calculated loss The model parameters are adjusted through the back-propagation algorithm to reduce the prediction error. This process may involve the gradient descent method or its variants, such as Adam, RMSprop and other optimization algorithms.

[0100] By repeating the above process, as the number of training rounds increases, the model gradually converges, that is, the loss no longer decreases significantly, and a fully optimized acoustic model can be obtained, which can generate high-quality and realistic speech signals given a phoneme sequence and discrete acoustic features.

[0101] In order to further improve the quality of the synthesized speech signal and enable the model to better capture the relationship between speech and text, the phoneme sequence prediction task can also be added to the second stage of training, that is, the phoneme sequence and discrete acoustic feature sequence corresponding to the training text are used as training samples, and the Mel spectrum feature prediction task and phoneme prediction task are used as training targets, and the initial acoustic model is continuously trained to obtain an acoustic model. The specific training process includes the following steps d1 to d4:

[0102] Step d1: Generate a predicted phoneme sequence according to the phoneme sequence and the discrete acoustic feature sequence through the acoustic model to be trained.

[0103] In step d1, the complete phoneme sequence and the discrete acoustic feature sequence are used as input to generate a predicted phoneme sequence through the acoustic model to be trained. These data provide the model with necessary context information to help it understand the speech and text features before the current moment.

[0104] Step d2: Determine the phoneme prediction loss according to the difference between the predicted phoneme sequence and the true phoneme sequence.

[0105] After obtaining the predicted phoneme sequence, its accuracy needs to be evaluated and the phoneme prediction loss is calculated based on the evaluation results. Specifically, the difference between the predicted phoneme sequence and the corresponding true phoneme sequence can be calculated to define the phoneme prediction loss. Commonly used loss functions include cross-entropy loss, which can effectively measure the error in classification tasks. Cross-entropy loss measures the difference between the predicted probability distribution and the true probability distribution. The smaller the value, the more accurate the prediction.

[0106] Step d3: Determine a comprehensive loss based on the phoneme prediction loss and the speech signal prediction loss.

[0107] In order to fully evaluate the performance of the model, it is necessary to consider not only the phoneme prediction loss, but also the speech signal prediction loss to determine a comprehensive loss. Therefore, the phoneme prediction loss and the speech signal prediction loss can be combined to form a comprehensive loss. The comprehensive loss can be calculated by weighted summation, and the weights can be adjusted according to actual needs to ensure that the importance of the two tasks is balanced.

[0108] Step d4: Continue training the initial acoustic model according to the comprehensive loss to obtain an acoustic model.

[0109] Finally, the initial acoustic model can continue to be trained based on the comprehensive loss to optimize the model parameters and improve its performance.

[0110] Specifically, the comprehensive loss is propagated backwards through the backpropagation algorithm, and the gradients of each parameter are calculated. The model parameters are updated using optimizers (such as Adam, SGD, etc.) to minimize the comprehensive loss. This step usually involves multiple training rounds until the model converges, that is, the loss no longer decreases significantly. By continuously iterating the above process, the loss changes and performance indicators of the model (such as the quality and stability of the synthesized speech) are monitored. When the model converges, a fully optimized acoustic model can be obtained, which can generate high-quality and realistic speech signals given a phoneme sequence and historical discrete acoustic features.

[0111] In some embodiments, in order to obtain a speech signal that meets the characteristics of the target speaker, an auxiliary speech signal of the target speaker may also be obtained, and the speech characteristics of the target speaker may be determined based on the auxiliary speech signal through a pre-trained speech synthesis model, the speech characteristics including speech habit characteristics and / or timbre characteristics. Subsequently, the speech characteristics of the target speaker, the phoneme sequence corresponding to the target text, and the discrete acoustic feature sequence are combined to synthesize a speech signal consistent with the target speaker.

[0112] The auxiliary speech samples can be natural conversations, readings or other forms of speech data of the speaker in different situations. These auxiliary speech signals provide rich reference information for the model to help it capture the specific characteristics of the speaker.

[0113] Speaking habit features include speaking speed, pause patterns, stress distribution, etc., which reflect the speaker's habitual behavior when expressing.

[0114] Timbre characteristics include fundamental frequency (F0), formant position, brightness of the sound, etc., which determine the unique voice quality of the speaker.

[0115] During the synthesis process, the above-mentioned determined speech features are combined with the phoneme sequence and discrete acoustic feature sequence corresponding to the target text. The model can fully consider the personalized characteristics of the target speaker when generating speech signals, so that the synthesized speech is closer to the natural performance of the target speaker in terms of intonation, rhythm and timbre, providing a more realistic and natural auditory experience, thereby ensuring that the final output speech not only accurately conveys the text content, but also realistically reproduces the voice characteristics of the target speaker.

[0116] In summary, this application uses multi-stage prediction tasks to train acoustic models, decomposing complex tasks into multiple gradually optimized subtasks, significantly reducing the difficulty of modeling at each stage. Specifically, the model is first trained through discrete acoustic feature sequence prediction tasks to ensure that it can accurately capture subtle changes in speech; then, on this basis, further speech signal prediction tasks are performed to enable the model to generate high-quality speech waveforms. This phased training strategy not only improves the convergence speed of the model, but also enhances its stability.

[0117] Secondly, through the training of multi-stage prediction tasks, the acoustic model can gradually learn and optimize at each stage, thereby reducing the possibility of error accumulation. In particular, when dealing with long sequences or variable speech features, the model can maintain high stability and consistency. For example, when predicting the discrete acoustic features of each frame, the model can make full use of historical information and contextual dependencies to ensure that the generated speech signal is coherent and natural, avoiding problems such as muteness or unnatural rhythm. This makes the entire synthesis process more reliable and suitable for a wider range of application scenarios.

[0118] In addition, the multi-stage training strategy of this application not only improves the convergence and stability of the model, but also significantly improves the quality of synthesized speech. Through the initial discrete acoustic feature prediction task, the model can better understand the basic structure and characteristics of speech; the subsequent speech signal prediction task further refines these features and converts them into realistic audio waveforms. In addition, technical means such as the cross attention mechanism and the duration hard alignment method introduced during the training process ensure the precise alignment between phonemes and acoustic features, making the generated speech more natural and fluent in intonation, rhythm, etc. In the end, users can experience higher fidelity and closer to real human speech synthesis results.

[0119] Exemplary Devices

[0120] Corresponding to the above-mentioned speech synthesis method, an embodiment of the present application also provides a speech synthesis device. Fig. 9 Schematic diagram of a speech synthesis device provided in an embodiment of the present application. Fig. 9As shown, the speech synthesis device provided in the embodiment of the present application includes: an acquisition unit 901 and a speech synthesis unit 902; wherein the acquisition unit 901 is used to acquire the phoneme sequence and the discrete acoustic feature sequence corresponding to the target text; the speech synthesis unit 902 is used to perform speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is trained by performing a multi-stage prediction task based on the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction task includes a discrete acoustic feature sequence prediction task and a speech signal prediction task.

[0121] In some embodiments, the phoneme sequence includes individual phoneme units, and the discrete acoustic feature sequence includes individual discrete acoustic feature units; wherein the speech synthesis unit 902 performs speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text, including: inputting the phoneme sequence and the discrete acoustic feature sequence into the pre-trained acoustic model, and determining the correspondence between the individual phoneme units and the individual discrete acoustic feature units through the acoustic model; and generating the speech signal corresponding to the target text according to the correspondence between the individual phoneme units and the individual discrete acoustic feature units.

[0122] In some embodiments, the acoustic model includes an encoder and a decoder; wherein the speech synthesis unit 902 determines the correspondence between each phoneme unit and each discrete acoustic feature unit through the acoustic model, including: inputting the phoneme sequence into the encoder, and the encoder generates a phoneme hidden layer representation sequence according to the phoneme sequence, and the phoneme hidden layer representation sequence includes the phoneme hidden layer representation corresponding to each phoneme unit; inputting the phoneme hidden layer representation corresponding to each phoneme unit and the discrete acoustic feature sequence into the decoder, and the decoder aligns the phoneme hidden layer representation corresponding to each phoneme unit and the discrete acoustic feature unit to determine the correspondence between each phoneme unit and the discrete acoustic feature unit.

[0123] In some embodiments, the training process of the acoustic model includes the following steps: obtaining the phoneme sequence and discrete acoustic feature sequence corresponding to the training text; using the phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the previous t frames as training samples, and using the discrete acoustic feature prediction task of the t+1th frame as the training target, training the acoustic model to be trained to obtain an initial acoustic model, where t is a positive integer; using the phoneme sequence and discrete acoustic feature sequence corresponding to the training text as training samples, and using the mel-spectrogram feature prediction task as the training target, continuing to train the initial acoustic model to obtain the acoustic model.

[0124] In some embodiments, the acoustic model to be trained is trained with the phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the previous t frames as training samples, and the discrete acoustic feature prediction task of the t+1th frame as the training target to obtain the initial acoustic model, including: generating predicted discrete acoustic features of the t+1th frame according to the phoneme sequence and the discrete acoustic feature sequence of the previous t frames through the acoustic model to be trained; determining the acoustic feature prediction loss according to the difference between the predicted discrete acoustic features of the t+1th frame and the actual discrete acoustic features of the t+1th frame, and converging the acoustic model to be trained based on the acoustic feature prediction loss to obtain the initial acoustic model.

[0125] In some embodiments, the initial acoustic model is continuously trained with the phoneme sequence and discrete acoustic feature sequence corresponding to the training text as training samples and the Mel-spectrogram feature prediction task as a training target to obtain the acoustic model, including: generating a predicted Mel-spectrogram feature sequence according to the phoneme sequence and the discrete acoustic feature sequence through the acoustic model to be trained; determining a Mel-spectrogram feature prediction loss according to the difference between the predicted Mel-spectrogram feature sequence and the true Mel-spectrogram feature sequence, and continuing to train the initial acoustic model based on the Mel-spectrogram feature prediction loss to obtain the acoustic model.

[0126] In some embodiments, the model training process also includes: generating a predicted phoneme sequence according to the phoneme sequence and the discrete acoustic feature sequence through the acoustic model to be trained; wherein, continuing to train the initial acoustic model based on the Mel-spectrogram feature prediction loss to obtain the acoustic model includes: determining the phoneme prediction loss based on the difference between the predicted phoneme sequence and the phoneme sequence; determining the comprehensive loss based on the phoneme prediction loss and the Mel-spectrogram feature prediction loss; continuing to train the initial acoustic model based on the comprehensive loss to obtain the acoustic model.

[0127] The speech synthesis device provided in this embodiment belongs to the same application concept as the speech synthesis method provided in the above embodiments of this application, can execute the speech synthesis method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the speech synthesis method. For technical details not fully described in this embodiment, please refer to the specific processing content of the speech synthesis method provided in the above embodiments of this application, which will not be repeated here.

[0128] It should be noted that the processing process of the speech synthesis unit 902 corresponds to Figure 3 The acquisition unit 901 can obtain the discrete acoustic feature sequence corresponding to the target text from the acoustic feature prediction module 31.

[0129] The functions implemented by the above acquisition unit 901 and speech synthesis unit 902 can be implemented by the same or different processors respectively, and the embodiment of the present application is not limited thereto.

[0130] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0131] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0132] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0133] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0134] Exemplary Electronic Devices

[0135] The present application embodiment provides an electronic device, see Fig.10 As shown, the electronic device includes:

[0136] Memory 200 and processor 210;

[0137] The memory 200 is connected to the processor 210 and is used to store programs;

[0138] The processor 210 is used to implement the speech synthesis method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0139] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0140] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0141] A bus may include a pathway that transfers information between components of a computer system.

[0142] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0143] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0144] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0145] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0146] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0147] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0148] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any speech synthesis method provided in the above embodiments of the present application.

[0149] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the speech synthesis method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the above-mentioned embodiment of the speech synthesis method.

[0150] Exemplary computer program products and storage media

[0151] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech synthesis method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0152] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the speech synthesis method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0154] Step S101: Obtain a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text.

[0155] Step S201, performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; wherein the acoustic model is trained by performing multi-stage prediction tasks according to the phoneme sequence and the discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction tasks include a discrete acoustic feature sequence prediction task and a speech signal prediction task.

[0156] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0157] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0159] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0161] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0163] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0165] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0166] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining the phoneme sequence and discrete acoustic feature sequence corresponding to the target text; Performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; The acoustic model is obtained by performing multi-stage prediction task training based on the phoneme sequence and discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction task includes a discrete acoustic feature sequence prediction task and a Mel-spectrogram feature prediction task.

2. The method according to claim 1, characterized in that The phoneme sequence includes various phoneme units, and the discrete acoustic feature sequence includes various discrete acoustic feature units; The method of performing speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence by using a pre-trained acoustic model to obtain a speech signal corresponding to the target text includes: Inputting the phoneme sequence and the discrete acoustic feature sequence into the pre-trained acoustic model, and determining the correspondence between each phoneme unit and each discrete acoustic feature unit through the acoustic model; According to the correspondence between the various phoneme units and the various discrete acoustic feature units, a speech signal corresponding to the target text is generated.

3. The method according to claim 2, characterized in that The acoustic model includes an encoder and a decoder; The determining of the correspondence between each of the phoneme units and each of the discrete acoustic feature units by using the acoustic model includes: Inputting the phoneme sequence into the encoder, the encoder generates a phoneme hidden layer representation sequence according to the phoneme sequence, wherein the phoneme hidden layer representation sequence includes the phoneme hidden layer representation corresponding to each of the phoneme units; The phoneme hidden layer representation corresponding to each phoneme unit and the discrete acoustic feature sequence are input into the decoder, and the decoder aligns the phoneme hidden layer representation corresponding to each phoneme unit and the discrete acoustic feature units to determine the correspondence between each phoneme unit and each discrete acoustic feature unit.

4. The method according to any one of claims 1 to 3, characterized in that The training process of the acoustic model includes the following steps: Obtaining the phoneme sequence and discrete acoustic feature sequence corresponding to the training text; Taking the phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the first t frames as training samples, and taking the discrete acoustic feature prediction task of the t+1th frame as the training target, the acoustic model to be trained is trained to obtain an initial acoustic model, where t is a positive integer; The initial acoustic model is further trained by taking the phoneme sequence and discrete acoustic feature sequence corresponding to the training text as training samples and the mel spectrum feature prediction task as a training target to obtain the acoustic model.

5. The method according to claim 4, characterized in that The method uses the phoneme sequence corresponding to the training text and the discrete acoustic feature sequence of the first t frames as training samples, and uses the discrete acoustic feature prediction task of the t+1th frame as a training target to train the acoustic model to be trained to obtain an initial acoustic model, including: Generate predicted discrete acoustic features of the t+1th frame according to the phoneme sequence and the discrete acoustic feature sequence of the previous t frames through the acoustic model to be trained; According to the difference between the predicted discrete acoustic features of the t+1th frame and the actual discrete acoustic features of the t+1th frame, the acoustic feature prediction loss is determined, and the acoustic model to be trained is converged based on the acoustic feature prediction loss to obtain an initial acoustic model.

6. The method according to claim 4, characterized in that The method further trains the initial acoustic model by taking the phoneme sequence and discrete acoustic feature sequence corresponding to the training text as training samples and the mel spectrum feature prediction task as a training target to obtain the acoustic model, including: Generate a predicted Mel-spectrogram feature sequence according to the phoneme sequence and the discrete acoustic feature sequence by using the acoustic model to be trained; According to the difference between the predicted Mel-spectrogram feature sequence and the true Mel-spectrogram feature sequence, a Mel-spectrogram feature prediction loss is determined, and the initial acoustic model is further trained based on the Mel-spectrogram feature prediction loss to obtain the acoustic model.

7. The method according to claim 6, characterized in that The method further comprises: Generate a predicted phoneme sequence according to the phoneme sequence and the discrete acoustic feature sequence by using an acoustic model to be trained; The step of continuing to train the initial acoustic model based on the mel spectrum feature prediction loss to obtain the acoustic model includes: determining a phoneme prediction loss based on a difference between the predicted phoneme sequence and the phoneme sequence; Determine a comprehensive loss based on the phoneme prediction loss and the Mel-spectrogram feature prediction loss; The initial acoustic model is further trained according to the comprehensive loss to obtain the acoustic model.

8. A speech synthesis device, characterized in that: include: An acquisition unit, used for acquiring a phoneme sequence and a discrete acoustic feature sequence corresponding to a target text; A speech synthesis unit, configured to perform speech synthesis according to the phoneme sequence and the discrete acoustic feature sequence through a pre-trained acoustic model to obtain a speech signal corresponding to the target text; The acoustic model is obtained by performing multi-stage prediction task training based on the phoneme sequence and discrete acoustic feature sequence corresponding to the training text, and the multi-stage prediction task includes a discrete acoustic feature sequence prediction task and a Mel-spectrogram feature prediction task.

9. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 7 by running the program in the memory.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 7.

Images