Speech synthesis method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech synthesis method and device, electronic equipment and storage medium, and relates to the technical field of speech synthesis. The method comprises the following steps: obtaining a target text; calling a multi-level prosody prediction model to perform prosody prediction on the target text, so as to obtain prosody features of the target text at at least two levels; and generating speech corresponding to the target text according to the target text and the prosody features at the at least two levels. Different levels represent different time scales of the prosody features. The multi-level prosody prediction model is trained based on a first text sample and multi-level prosody feature samples. The technical scheme provided by the application can improve the prosody expressiveness of speech synthesis.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of computer technology and artificial intelligence, speech synthesis technology, also known as text-to-speech (TTS), has been widely applied. The diverse applications of speech synthesis have greatly facilitated daily life and enriched the viewing experience brought by multimedia technology. Addressing the needs of different scenarios, improving the prosodic expressiveness of speech synthesis is crucial for enhancing the anthropomorphism and style matching of the synthesized speech, thereby improving the listening experience in corresponding scenarios.

[0003] In related technologies, latent variable learning methods such as Variational Autoencoders (VAEs) can be used for prosodic representation in speech synthesis. This method learns global style labels for speech and then uses a reference speech to achieve style transfer during speech synthesis, or uses attention mechanisms and other techniques to specify the style of the synthesized speech. However, this method can only characterize speech style information at the sentence level and cannot fully control the prosodic characteristics of speech. This results in synthesized speech with monotonous tone and intonation, lacking diversity and variability, which can easily lead to listening fatigue for users. Summary of the Invention

[0004] This invention provides a speech synthesis method, apparatus, electronic device, and storage medium to solve the problem of monotonous speech effects in existing speech synthesis technologies and improve the prosodic expressiveness of speech synthesis.

[0005] This invention provides a speech synthesis method, comprising:

[0006] Obtain the target text;

[0007] A multi-level prosodic prediction model is invoked to predict the prosodic features of the target text at at least two levels; wherein, different levels represent different time scales of the prosodic features; the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples.

[0008] The speech corresponding to the target text is generated based on the target text and the prosodic features of at least two levels.

[0009] According to a speech synthesis method provided by the present invention, the step of calling a multi-level prosodic prediction model to predict the prosodic features of the target text and obtaining the prosodic features of the target text at at least two levels includes:

[0010] The target text is input into the text encoding layer of the multi-level prosody prediction model to obtain the text features of the target text;

[0011] For each of the at least two levels, the text features and the prosodic features of the previous level are input into the feature processing layer of the multi-level prosodic prediction model to obtain the temporal fusion features output by the feature processing layer; the feature processing layer is used to perform temporal alignment and splicing of the text features and the prosodic features of the previous level.

[0012] The temporal fusion features are input into the decoding layer of the multi-level prosodic prediction model to obtain the prosodic features corresponding to each level output by the decoding layer.

[0013] The higher-level prosodic features are determined based on the prosodic features of the target text at the sentence level.

[0014] According to a speech synthesis method provided by the present invention, the multi-level prosodic prediction model is trained based on the following steps:

[0015] Obtain the first text sample and obtain the multi-level prosodic feature samples, wherein the multi-level prosodic feature samples include a first prosodic feature sample at the sentence level and at least one second prosodic feature sample; the second prosodic feature sample is a prosodic feature sample at a level other than the sentence level.

[0016] Based on the first text sample and the prosodic feature sample of the previous level, the initial multi-level prosodic prediction model is trained for each level to obtain the multi-level prosodic prediction model; wherein, the sentence level is the initial level.

[0017] According to a speech synthesis method provided by the present invention, obtaining the multi-level prosodic feature samples includes:

[0018] Obtain the target acoustic features of the sample speech;

[0019] Each of the sentence level and other levels is sequentially taken as the target level. The target acoustic features are input into the hierarchical prosodic prediction model corresponding to the target level to obtain the prosodic feature sample corresponding to the target level output by the hierarchical prosodic prediction model. The hierarchical prosodic prediction model is used to downsample and quantize the target acoustic features.

[0020] The downsampling scale is different for each of the target levels.

[0021] According to a speech synthesis method provided by the present invention, the step of obtaining the target acoustic features of sample speech includes:

[0022] The original acoustic features of the sample speech are obtained and input into a dimensionality reduction coding network to obtain the target acoustic features of the sample speech output by the dimensionality reduction coding network; wherein, the dimensionality reduction coding network is obtained by jointly training an initial coding network and an initial decoding network based on the sample acoustic features; the dimensionality reduction coding network is used to perform dimensionality reduction and compression on the original acoustic features.

[0023] According to a speech synthesis method provided by the present invention, the hierarchical prosodic prediction model corresponding to the target level is trained based on the following steps:

[0024] The target acoustic features are input into the initial hierarchical prosodic prediction model corresponding to the target level to obtain the first prosodic feature output by the initial hierarchical prosodic prediction model.

[0025] The second prosodic feature is obtained, and the first and second prosodic features are temporally aligned and concatenated to obtain the third prosodic feature; the second prosodic feature is the prosodic feature output by the hierarchical prosodic prediction model of the next level above the target level based on the target acoustic features.

[0026] A second text sample is obtained and input into a text encoding network to obtain sample text features output by the text encoding network; the text encoding network is used to extract the text features of the second text sample and upsample the text features of the second text sample to the same time scale as the target acoustic features;

[0027] The third prosodic feature and the sample text feature are time-aligned and concatenated to obtain the processing result;

[0028] The processing result and the target acoustic features are input into the decoding network to obtain the output result of the decoding network;

[0029] Based on the output results and loss function, the parameters of the initial hierarchical prosody prediction model, the text encoding network, and the decoding network are adjusted to obtain the hierarchical prosody prediction model corresponding to the target level.

[0030] According to a speech synthesis method provided by the present invention, generating speech corresponding to the target text based on the target text and the prosodic features of at least two levels includes:

[0031] The target text and the prosodic features at least two levels are input into the speech synthesis model to obtain the target speech acoustic features output by the speech synthesis model.

[0032] The vocoder is invoked to generate the speech corresponding to the target text from the acoustic features of the target speech.

[0033] The speech synthesis model is obtained by training an initial speech synthesis model based on the first text sample, the multi-level prosodic feature samples, and the target acoustic features of the sample speech.

[0034] The present invention also provides a speech synthesis device, comprising:

[0035] The acquisition module is used to acquire the target text;

[0036] The prosody prediction module is used to call a multi-level prosody prediction model to predict the prosody of the target text and obtain the prosodic features of the target text at at least two levels; wherein, different levels represent different time scales of speech acoustic features; the multi-level prosody prediction model is trained based on a first text sample and multi-level prosodic feature samples.

[0037] The generation module is used to generate the speech corresponding to the target text based on the target text and the prosodic features of at least two levels.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described speech synthesis methods.

[0039] The present invention also provides an electronic device, including a processor and a speaker connected to the processor;

[0040] The processor is used to acquire target text, call a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels, and generate speech corresponding to the target text based on the target text and the prosodic features at the at least two levels; wherein, different levels represent different time scales of the acoustic features of the speech; the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples;

[0041] The speaker is used to play the voice.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0043] The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention acquire target text, invoke a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels, and then generate speech corresponding to the target text based on the target text and the prosodic features at at least two levels, thus realizing text-to-speech conversion. The multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples. Different levels can represent different time scales of prosodic features, and the representation of prosodic features at different time scales can reflect different prosodic attributes of speech. Therefore, at least two levels of prosodic features can reflect the prosodic attributes of speech at at least two time scales. Different prosodic attributes can represent the characteristics of speech at different levels. Thus, the speech generated using at least two levels of prosodic features contains prosodic information at multiple time scales, exhibiting diverse performance effects, thereby improving the prosodic expressiveness of speech synthesis. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the speech synthesis method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the principle of using a multi-level prosodic prediction model to predict the prosodic rhythm of a target text in an embodiment of the present invention.

[0047] Figure 3 This is a flowchart illustrating the method of calling a multi-level prosodic prediction model to predict the prosodicity of a target text in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram illustrating the training principle of the speech synthesis model in this embodiment of the invention;

[0049] Figure 5 This is a flowchart illustrating the training method of the multi-level prosody prediction model in an embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram illustrating the training principle of the multi-level prosody prediction model in an embodiment of the present invention;

[0051] Figure 7 This is a flowchart illustrating the training method of the hierarchical prosody prediction model corresponding to the target level in this embodiment of the invention.

[0052] Figure 8 This is a schematic diagram illustrating the training principle of the hierarchical prosody prediction model at the sentence level in an embodiment of the present invention;

[0053] Figure 9 This is a schematic diagram illustrating the training principle of the hierarchical prosody prediction model at the second time scale in an embodiment of the present invention.

[0054] Figure 10 This is a schematic diagram illustrating the training principle of the hierarchical prosody prediction model at the third time scale in an embodiment of the present invention.

[0055] Figure 11 This is a schematic diagram illustrating the training principle of the dimensionality reduction coding network in an embodiment of the present invention;

[0056] Figure 12 This is a schematic diagram of the speech synthesis device provided in an embodiment of the present invention;

[0057] Figure 13 This is one of the structural schematic diagrams of the electronic device provided in the embodiments of the present invention;

[0058] Figure 14 This is the second structural schematic diagram of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] It should be noted that the serial numbers assigned to the objects described in this invention, such as "first" and "second", are only used to distinguish the objects being described and do not have any sequential or technical meaning.

[0061] Speech synthesis technology transforms computer-generated or externally input text information into near-human speech output, and is a crucial component of human-computer voice communication. In different application scenarios, the style matching, prosodic controllability, and expressiveness of the output speech will provide users with varying auditory experiences. Therefore, improving the prosodic expressiveness of synthesized speech, achieving a certain degree of controllability, and enhancing its human-like quality are issues of great interest in the industry, contributing to a better auditory experience in relevant scenarios.

[0062] In related technologies, latent variable learning methods based on VAEs can be used for prosodic representation in speech synthesis. For example, Vector Quantization Variational Autoencoders (VQ-VAEs) can be used to learn global style labels for speech. During speech synthesis, a reference speech can be used to achieve style transfer, or attention mechanisms and other techniques can be used to specify the style of the synthesized speech. However, this method can only characterize style information at the sentence level and cannot fully control the prosodic characteristics of speech. Moreover, this synthesis method depends on the reference speech and is not flexible enough in application.

[0063] In related technologies, additional fundamental frequency and energy prediction units can be introduced into the acoustic model. During model training, fundamental frequency and energy values ​​are extracted from the recording data as supervision signals, making the model more accurate in characterizing the distribution of variables such as fundamental frequency and energy in the data. This increases the prosodic controllability of speech synthesis to some extent and improves the expressiveness of the synthesized style. However, in scenarios with low data volume, it is difficult for the model to achieve accurate fundamental frequency and energy prediction, which may actually degrade the overall speech synthesis effect.

[0064] Based on this, embodiments of the present invention provide a speech synthesis method that can acquire target text, invoke a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels, and then generate speech corresponding to the target text based on the target text and the prosodic features at at least two levels, thus realizing multi-level prosodic speech synthesis. The multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples; different levels represent different time scales of prosodic features. In this way, it can fully utilize the prosodic information at multiple scales in the speech signal to synthesize speech, and can be applied to various speech synthesis systems, achieving a significant improvement in the expressiveness of speech synthesis.

[0065] The following is combined Figures 1-11 The speech synthesis method of the present invention is described below. This speech synthesis method can be applied to electronic devices such as servers or smart terminals. The terminal devices may include mobile phones, computers, in-vehicle devices, tablet computers, wearable devices, smart robots, smart home devices, etc.; the servers may include independent servers, cluster servers, or cloud servers, etc. This speech synthesis method can also be applied to speech synthesis devices installed in terminal devices or servers and other electronic devices, which can be implemented through software, hardware, or a combination of both.

[0066] Figure 1 An exemplary flowchart of the speech synthesis method provided in an embodiment of the present invention is shown below, with reference to... Figure 1 As shown, the speech synthesis method may include the following steps 110 to 130.

[0067] Step 110: Obtain the target text.

[0068] The target text refers to the text to be converted from text to speech, and it is the input information for the entire speech synthesis process. For example, the target text can be text at the document level, paragraph level, or sentence level.

[0069] For example, the text format of the target text can be represented as characters of a language, or as a visual encoding or computer-readable encoding of a language.

[0070] For example, an electronic device can obtain target text from locally stored text data or from other electronic devices that are communicatively connected to the electronic device.

[0071] Step 120: Call the multi-level prosody prediction model to predict the prosody of the target text and obtain the prosodic features of the target text at at least two levels.

[0072] Different levels represent different time scales of prosodic features. The multi-level prosodic prediction model is trained based on the first text sample and prosodic feature samples from multiple levels, and can be used to predict the prosodic features of the target text at at least two time scales.

[0073] The prosodic attributes of speech are hierarchical. For example, changes in prosodic features over a larger time scale are strongly correlated with speaker style and role, while changes over a finer time scale are closely related to timbre and intonation. Based on this, prosodic prediction of target text can be performed at different levels to obtain prosodic features at at least two levels. These time scales can include at least two of the following: sentence duration, paragraph duration, phrase duration, and syllable duration.

[0074] Multi-level prosodic feature samples include prosodic feature samples from at least two levels. For example, a multi-level prosodic feature sample may include a first prosodic feature sample at the sentence level and at least one second prosodic feature sample, which is a prosodic feature sample from a level other than the sentence level. For example, the second prosodic feature sample may include prosodic feature samples with a time scale smaller than the time scale of the sentence level.

[0075] For example, a multi-level prosodic prediction model can be invoked to predict the prosodicity of the target text at each level; alternatively, the multi-level prosodic prediction model can be invoked to predict from the initial level according to the dependencies between levels, with each subsequent level's prediction depending on the prediction result of the previous level. The initial level could be, for example, the sentence level.

[0076] Based on this, in one example embodiment, Figure 2 This example illustrates the principle of using a multi-level prosodic prediction model to predict the prosodic rhythm of a target text. (Refer to...) Figure 2 As shown, a multi-level prosodic prediction model can include a text encoding layer, a feature processing layer, and a decoding layer. The model structure of this multi-level prosodic prediction model can be a Transformer model structure. Accordingly, the text encoding layer can serve as the encoder of the Transformer model structure, the feature processing layer as the hidden encoding layer, and the decoding layer as the decoder of the Transformer model structure. For example, the feature processing layer can include a first sub-processing layer and a second sub-processing layer. The first sub-processing layer can perform temporal alignment and concatenation of the input features, and the second sub-processing layer can further process the output of the first sub-processing layer, such as performing feature fusion based on temporal information. The decoding layer can decode the output of the feature processing layer. The second sub-processing layer can be a Long Short-Term Memory (LSTM) network.

[0077] For example, the decoding network of the decoding layer can be based on a normalizing-flow generative model.

[0078] Combination Figure 2 , Figure 3 An exemplary diagram illustrates a method for prosodic prediction of target text using a multi-level prosodic prediction model, as illustrated in an embodiment of the present invention. (Refer to...) Figure 3 As shown, the method may include the following steps 310 to 330.

[0079] Step 310: Input the target text into the text encoding layer of the multi-level prosody prediction model to obtain the text features of the target text.

[0080] For example, text features could be the phoneme features of the target text.

[0081] Step 320: For each of at least two levels, input the text features and the prosodic features of the previous level into the feature processing layer of the multi-level prosodic prediction model to obtain the temporal fusion features output by the feature processing layer.

[0082] The feature processing layer is used to perform temporal alignment and concatenation of text features and the prosodic features of the previous level. The prosodic features of the previous level are determined based on the prosodic features of the target text at the sentence level. For example, the sentence level can be used as the initial level, and the prosodic features of the second level can be determined based on the prosodic features of the sentence level, the prosodic features of the third level can be determined based on the prosodic features of the second level, and so on.

[0083] For example, the feature processing layer may include a first sub-processing layer and a second sub-processing layer. The feature processing layer can perform temporal alignment and splicing of text features and the prosodic features of the previous level through the first sub-processing layer, and then input the processing result into the second sub-processing layer for processing, such as temporal fusion, to obtain temporal fusion features.

[0084] Step 330: Input the temporal fusion features into the decoding layer of the multi-level prosody prediction model to obtain the prosody features corresponding to each level output by the decoding layer.

[0085] Combination Figure 2 Taking the prediction of prosodic features of the i-th layer as an example, the target text is encoded by the text encoding layer to obtain text features. After the text features are input into the feature processing layer, the feature processing layer compares the text features with the prosodic features V of the (i-1)-th layer. i-1 After performing temporal alignment and concatenation to obtain temporal fusion features, these features are input into the decoding layer for decoding to obtain the prosodic features V of the i-th layer. i The prosodic feature V i It can be used to predict the prosodic features of the (i+1)th layer.

[0086] For the prosodic feature V of the (i-1)th layer i-1 It can be used with V i The processing flow is similar, the only difference being that the prosodic features V in the feature processing layer are the same as those in the (i-2)th layer. i-2 Temporal alignment and concatenation are then performed. This process continues, tracing back to the prosodic feature V1 of layer 1 (the initial layer). It's understandable that the initial layer prediction is unaffected by other layers; that is, the feature processing layer only receives the output of the text encoding layer. In this case, the multi-level prosodic prediction model can directly predict the prosodic feature V1 of the initial layer for the target text.

[0087] In this way, prosodic prediction through a hierarchical approach can take into account the relationships between levels, making the prosodic features predicted at each level more accurate and better representing the characteristics of speech, thereby further improving the effect and expressiveness of the synthesized speech. Moreover, the prosodic prediction process is the same at each level, making the prediction of speech prosody very simple.

[0088] Step 130: Generate the speech corresponding to the target text based on the target text and at least two levels of prosodic features.

[0089] After obtaining at least two levels of prosodic features, speech can be generated based on speech synthesis technology, using the target text and these prosodic features to produce the corresponding speech, thus achieving text-to-speech conversion. Through this text-to-speech conversion, anthropomorphic speech can be obtained.

[0090] For example, the acquired speech can be a speech signal that can be directly recognized by the human ear or a machine, and can be played through a speaker or other speech playback device. Alternatively, the acquired speech can be stored in a storage medium in the form of a speech file, which may include audio files in formats such as wav, MP3, and MP4, or audio tracks of video files in formats such as mkv and avi. This speech file can be accessed by an electronic device and played through a speaker or other speech playback device.

[0091] For example, speech corresponding to a target text can be synthesized based on a speech synthesis model. Specifically, generating speech corresponding to a target text based on the target text and at least two levels of prosodic features may include: inputting the target text and at least two levels of prosodic features into a speech synthesis model to obtain the target speech acoustic features output by the speech synthesis model; calling a vocoder to generate speech corresponding to the target text from the target speech acoustic features; wherein, the speech synthesis model is obtained by training an initial speech synthesis model based on a first text sample, multi-level prosodic feature samples, and the target acoustic features of the sample speech.

[0092] For example, a vocoder may include a vocoder based on Generative Adversarial Networks (GANs), a vocoder based on an Autoregressive Neural Network (WaveNet), or a vocoder based on a non-autoregressive neural network.

[0093] The sample speech includes the speech data corresponding to the first text sample. For example, the sample speech can be obtained by recording the reading of the first text sample. For instance, the sample speech can be the original recording data, or the recording data after noise reduction, cropping, and other cleaning processes. The target acoustic features can be the original acoustic features extracted from the sample speech, or the acoustic features after dimensionality reduction and compression of the original acoustic features. These acoustic features can include Mel spectral features or perceptual linear predictive (PLP) features, etc.

[0094] For example, the initial speech synthesis model may include a Transformer-based deep neural network model or a Normalizing-Flow-based deep neural network model, etc.

[0095] For example, taking a deep neural network model based on Normalizing-Flow as an example, Figure 4 An exemplary diagram illustrating the training principle of the speech synthesis model in an embodiment of the present invention is shown below. Figure 4As shown, multi-level prosodic feature samples are utilized during the training process of the Normalizing-Flow-based speech synthesis model. Specifically, during the training phase, multi-level prosodic feature samples, such as N levels of prosodic feature samples, are temporally aligned with the text features of the first text sample after text encoding at the feature processing layer. After concatenation along the feature dimension, further processing is performed at the feature processing layer, such as using LSTM for sequence information transfer and fusion. The processing result is then used as the conditional input of the Normalizing-Flow modeling target acoustic features (such as mel spectral features) and input to the decoding layer. The model parameters of the initial speech synthesis model are adjusted based on the output of the decoding layer and the loss function until the model converges. The final trained initial speech synthesis model can then be determined as the speech synthesis model. The method for obtaining multi-level prosodic feature samples is described below.

[0096] For example, the model can be updated during training by calculating the maximum likelihood criterion.

[0097] The speech synthesis method provided in this invention acquires target text, calls a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels, and then generates speech corresponding to the target text based on the target text and the prosodic features at at least two levels, thus realizing text-to-speech conversion. The multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples. Different levels can represent different time scales of prosodic features, and the representation of prosodic features at different time scales can reflect different prosodic attributes of speech. Therefore, at least two levels of prosodic features can reflect the prosodic attributes of speech at at least two time scales. Different prosodic attributes will exhibit different speech effects. In this way, the speech generated using at least two levels of prosodic features contains prosodic information at multiple time scales, has diverse performance effects, and thus improves the expressiveness of speech synthesis.

[0098] Based on the methods of the above embodiments, Figure 5 An exemplary flowchart illustrates the training method of a multi-level prosody prediction model according to an embodiment of the present invention. (Refer to...) Figure 5 As shown, the multi-level prosody prediction model can be trained based on the following steps 510 to 530.

[0099] Step 510: Obtain the first text sample.

[0100] The first text sample may include at least one of the following: passage text, paragraph text, and sentence text.

[0101] Step 520: Obtain multi-level prosodic feature samples.

[0102] Multi-level prosodic feature samples can be prosodic features at different time scales extracted from the target acoustic features of the sample speech. The sample speech includes the speech data corresponding to the first text sample. The target acoustic features can be the original acoustic features extracted from the sample speech, or acoustic features after dimensionality reduction and compression of the original acoustic features.

[0103] The multi-level prosodic feature samples include a first prosodic feature sample at the sentence level and at least one second prosodic feature sample. The second prosodic feature sample is a prosodic feature sample from a level other than the sentence level.

[0104] The extraction of multi-level prosodic features can be divided into multiple stages according to time scale. For example, prosodic feature extraction can be performed at the sentence-level time scale, the second time scale (smaller than the sentence-level time scale), and the third time scale (smaller than the sentence-level time scale), where the second and third time scales are different. For instance, in the staged extraction, prosodic feature extraction can be performed first at the sentence-level time scale, then based on the extracted prosodic features, prosodic feature extraction can be performed at the second time scale, and then based on the prosodic features extracted at the second time scale, prosodic feature extraction can be performed at the third time scale, thus obtaining prosodic feature samples at each time scale.

[0105] Step 530: Based on the first text sample and the prosodic feature sample of the previous level of each level, train the initial multi-level prosodic prediction model for each level to obtain the multi-level prosodic prediction model.

[0106] For example, a multi-level prosodic prediction model can use a multi-level autoregressive network to predict prosodic features. Figure 6 An exemplary diagram illustrating the training principle of a multi-level prosody prediction model is shown below. Figure 6 As shown, taking the training of the prosodic representation at level i as an example, the first text sample is input into the text encoding layer of the initial multi-level prosodic prediction model for encoding to obtain text encoding features. Then, the text encoding features and the input prosodic features at level i-1 are temporally aligned and processed by LSTM in the feature processing layer. The resulting processing is then used as the conditional input to the decoding layer, which combines the sequence data V from the previous time step. i [t-1] represents the sequence data V at the current time. i [t] is used to make predictions, and then the model parameters of the initial multi-level prosody prediction model are adjusted in reverse based on the loss function until the loss function value converges. The final adjusted initial multi-level prosody prediction model is then determined as the multi-level prosody prediction model.

[0107] In this training process, the prosodic representations at each level are trained using a teacher-forcing approach, with the ground-truth prosodic representations from one or more previous levels serving as additional input. This accelerates the model's convergence while enhancing its robustness. Using an autoregressive network for prosodic prediction allows for consideration of the dependencies between prosodic representations at different time scales during model training, thereby improving the prosodic representation performance of the trained multi-level prosodic prediction model.

[0108] The text encoding layer can be a Transformer-based encoding layer, and the feature processing layer can represent a Transformer-based encoding hidden layer.

[0109] based on Figure 5 In one example embodiment, the method for obtaining multi-level prosodic feature samples may include: obtaining target acoustic features of the sample speech; sequentially using each level from the sentence level and other levels as the target level; inputting the target acoustic features into the hierarchical prosodic prediction model corresponding to the target level; and obtaining the prosodic feature samples corresponding to the target level output by the hierarchical prosodic prediction model. The hierarchical prosodic prediction model is used to downsample and quantize the target acoustic features; each target level has a different downsampling scale.

[0110] Figure 7 An exemplary schematic diagram illustrates the training method of the hierarchical prosody prediction model corresponding to the target level in an embodiment of the present invention. (Refer to...) Figure 7 As shown, the hierarchical prosody prediction model corresponding to the target level can be trained based on the following steps 710 to 760.

[0111] Step 710: Input the target acoustic features into the initial level prosodic prediction model corresponding to the target level to obtain the first prosodic feature output by the initial level prosodic prediction model.

[0112] Each level in the sentence hierarchy and other hierarchical levels is sequentially used as the target level. For example, when training the hierarchical prosodic prediction model corresponding to the i-th level, an initial hierarchical prosodic prediction model is first established for that level, which is the hierarchical prosodic prediction model to be trained. Then, the target acoustic features are input into this initial hierarchical prosodic prediction model. This initial hierarchical prosodic prediction model can perform encoding pooling on the target acoustic features, and then perform vector quantization encoding on the obtained encoding pooling results to obtain the first prosodic feature.

[0113] For example, the initial hierarchical prosody prediction model may include a first encoding layer and a second encoding layer. The initial hierarchical prosody prediction model may encode and average pool the target acoustic features through the first encoding layer, and then quantize and encode the pooling results through the second encoding layer.

[0114] Step 720: Obtain the second prosodic feature, and perform temporal alignment and splicing on the first and second prosodic features to obtain the third prosodic feature.

[0115] Among them, the second prosodic feature is the prosodic feature output by the hierarchical prosodic prediction model of the next level above the target level, based on the target acoustic features.

[0116] For example, when training the hierarchical prosodic prediction model corresponding to the i-th level, the second prosodic feature can be obtained after the hierarchical prosodic prediction model corresponding to the (i-1)-th level extracts the prosodic features of the target acoustic features. Then, the first and second prosodic features are temporally aligned and concatenated to obtain the third prosodic feature.

[0117] Step 730: Obtain the second text sample and input the second text sample into the text encoding network to obtain the sample text features output by the text encoding network.

[0118] The text encoding network is used to extract the text features of the second text sample and upsample the text features of the second text sample to the same time scale as the target acoustic features.

[0119] The second text sample can be the same as or different from the first text sample.

[0120] Step 740: Perform temporal alignment and splicing on the third prosodic feature and the sample text feature to obtain the processing result.

[0121] Step 750: Input the processing results and target acoustic features into the decoding network to obtain the output of the decoding network.

[0122] Step 760: Adjust the parameters of the initial hierarchical prosody prediction model, text encoding network, and decoding network based on the output results and loss function to obtain the hierarchical prosody prediction model corresponding to the target level.

[0123] After obtaining the output of the decoding network, the loss value can be calculated based on the output and the loss function. Based on the loss value, the parameters of the initial hierarchical prosody prediction model, the text encoding network, and the decoding network are adjusted through the backpropagation algorithm. The final adjusted initial hierarchical prosody prediction model is then determined as the hierarchical prosody prediction model corresponding to the target level.

[0124] For example, the loss function could be the Maximum Likelihood Estimation (MLE) loss function.

[0125] The following is combined Figures 8-10Taking sentence-level time scale, second time scale smaller than sentence-level time scale, and third time scale smaller than sentence-level time scale as examples, the training principle of the hierarchical prosody prediction model corresponding to the target level in the embodiment of the present invention is explained, wherein the second time scale is larger than the third time scale.

[0126] Figure 8 An exemplary diagram illustrates the training principle of a hierarchical prosodic prediction model at the sentence level in an embodiment of the present invention. (Refer to...) Figure 8 As shown, in this stage, we attempt to represent the global prosodic style of a sentence using a vector of length 1. The initial hierarchical prosodic prediction model 1 can be trained using the Vector Quantization Autoencoder (VQ-AutoEncoder) network framework to obtain the hierarchical prosodic prediction model at the sentence level.

[0127] Specifically, the target acoustic features are input into the first encoding layer of the initial hierarchical prosodic prediction model 1. This first encoding layer can be an encoder network. After downsampling by the average pooling sublayer in the first encoding layer, the average value of the target acoustic features in the temporal dimension is obtained. Then, the average value is vector quantized (VQ) by the second encoding layer of the initial hierarchical prosodic prediction model 1 to output the discretized prosodic features V1 at the sentence level. The second text sample is encoded by a Transformer-based text encoding network. The resulting text features are upsampled to the same time scale as the target acoustic features according to the phoneme duration. Then, they are temporally aligned with the prosodic features V1 obtained by vector quantization and concatenated in the feature dimension. The concatenated result is processed by an LSTM layer and used as the input to the decoding network. The LSTM layer can transfer and fuse sequence information.

[0128] Among them, the decoding network can be implemented using a generative model based on normalizing-flow, which can achieve good prosody learning results. Normalizing-flow can use an invertible distribution transformation to normalize the target distribution into a standard Gaussian distribution with independent dimensions. Using the properties of the distribution transformation, the log-likelihood of the target distribution can be expressed as the following formula (1):

[0129]

[0130] Where c represents the conditional input, which in this embodiment of the invention is the input to the decoding network; x represents the target distribution; z represents the standard Gaussian distribution; logP θ (x|c) represents the log-likelihood of the target distribution, logP θ (z|c) represents the log-likelihood of the standard Gaussian distribution. Let represent the determinant of the Jacobian matrix of z with respect to x. Thus, using the maximum likelihood criterion, the network of this VQ-AutoEncoder structure can be trained. After the network training converges, the model trained from the initial hierarchical prosody prediction model 1 is the sentence-level hierarchical prosody prediction model, which can be denoted as hierarchical prosody prediction model 1 in this embodiment of the invention.

[0131] In use Figure 8 After obtaining the sentence-level hierarchical prosodic prediction model 1 through the training principle, this model can be used to extract the sentence-level discrete speech prosodic features V1. Based on the prosodic features V1, prosodic feature extraction at a second time scale smaller than the sentence level can be further achieved. V1 can then be used to further train the hierarchical prosodic prediction model at the second time scale.

[0132] Figure 9 An exemplary diagram illustrating the training principle of the hierarchical prosody prediction model at the second time scale in an embodiment of the present invention is shown below. Figure 9 As shown, the structure of the initial hierarchical prosodic prediction model 2 at the second time scale can be the same as that of the hierarchical prosodic prediction model 1. During model training, the target acoustic features are input into the first encoding layer of the initial hierarchical prosodic prediction model 2. After performing the same processing procedure as the initial hierarchical prosodic prediction model 1 in the initial hierarchical prosodic prediction model 2, the discretized prosodic features V2 at the second time scale are output. At the same time, the prosodic features V1 obtained by sentence hierarchical quantization encoding are also used as additional input in this training stage. They are temporally aligned and concatenated with the prosodic features V2 to obtain the prosodic features V. 21 In the initial hierarchical prosody prediction model 2, the downsampling scale of the first coding layer during average pooling is smaller than that of the first coding layer in the initial hierarchical prosody prediction model 1.

[0133] The second text sample can be compared with... Figure 8 A similar processing flow involves upsampling text features and combining them with prosodic features V during the alignment and splicing processes. 21 Align them temporally and stitch them together along the feature dimension.

[0134] according to Figure 9 As shown, it can be adopted with Figure 8 The same model training method is used for model training. During model training, the gradient of V1 is not backpropagated to the hierarchical prosody prediction model 1. That is, during training, only the parameters of the network structure other than the hierarchical prosody prediction model 1 are adjusted, and the model parameters of the hierarchical prosody prediction model 1 are not changed. Under this architecture, because of the existence of V1, quantization encoding of V2 can be achieved with only a small codebook, and there is no need to group the codewords, which reduces the difficulty of quantization encoding.

[0135] Thus, a hierarchical prosody prediction model at the second time scale can be trained, which can be referred to as hierarchical prosody prediction model 2 in this embodiment of the invention.

[0136] In use Figure 9 After obtaining the hierarchical prosodic prediction model 2 at the second time scale through the training principle, this model can be used to extract discrete speech prosodic features V2 at the second time scale. Based on prosodic features V2, prosodic feature extraction at a third time scale smaller than the sentence level can be further achieved. The hierarchical prosodic prediction model at the third time scale can be further trained using V1 and V2.

[0137] Figure 10 An exemplary diagram illustrating the training principle of the hierarchical prosody prediction model at the third time scale in an embodiment of the present invention is shown below. Figure 10 As shown, it is possible to Figure 9 The schematic diagram shows the addition of an initial hierarchical prosodic prediction model 3 at a third time scale. The structure of this initial hierarchical prosodic prediction model 3 at the third time scale can be the same as that of the hierarchical prosodic prediction model 1. Based on this, a method similar to... Figure 9 A similar method is used to train the initial hierarchical prosody prediction model 3.

[0138] In this training phase, both sentence-level prosodic features V1 and second-time-scale prosodic features V2 are incorporated into the training. The prosodic features V1 obtained by aligning and concatenating the two are then combined. 21 It also serves as an additional input for this training phase, and is temporally aligned and concatenated with prosodic feature V3 to obtain prosodic feature V. 321 In the initial hierarchical prosody prediction model 3, the downsampling scale of the first coding layer during average pooling is smaller than that of the first coding layer in the initial hierarchical prosody prediction model 2.

[0139] according to Figure 10 As shown, it can be adopted with Figure 8 and Figure 9 The same model training method is used for model training. During model training, the gradients of V1 and V2 are not backpropagated to their respective hierarchical prosody prediction models; that is, the model parameters of hierarchical prosody prediction model 1 and hierarchical prosody prediction model 2 are not adjusted. In this way, a hierarchical prosody prediction model at the third time scale can be trained.

[0140] Through the aforementioned training stages, the VQ-AutoEncoder network can sequentially obtain discrete prosodic representations at the sentence level, second time scale, and third time scale from a single speech dataset, with these prosodic features at different time scales exhibiting hierarchical dependencies. Understandably, in practical applications, more prosodic feature extraction at various time scales can be added as needed, and the network can be trained using the same method. This approach is highly scalable, simple to train, and easy to implement.

[0141] Based on this, on the one hand, the trained hierarchical prosodic prediction models at each level can be used to predict the prosodic features at each level, and on the other hand, based on the prosodic features at each level, methods such as... Figure 6 The training principle of the multi-level prosodic prediction model shown is used to train the multi-level prosodic prediction model. On the other hand, the trained multi-level prosodic prediction models can be used to predict the prosodic features of each level. Based on the prosodic features of each level, methods such as... Figure 4 The training principle of the speech synthesis model shown is used to train the speech synthesis model.

[0142] Based on the methods of the above embodiments, the target acoustic features can be acoustic features obtained by dimensionality reduction and compression of the original acoustic features extracted from the sample speech, and these acoustic features can be Mel-spectral features. Accordingly, in an example embodiment, obtaining the target acoustic features of the sample speech may include: obtaining the original acoustic features of the sample speech, and inputting the original acoustic features into a dimensionality reduction coding network to obtain the target acoustic features of the sample speech output by the dimensionality reduction coding network; wherein, the dimensionality reduction coding network is obtained by jointly training an initial coding network and an initial decoding network based on the sample acoustic features; the dimensionality reduction coding network is used to perform dimensionality reduction and compression on the original acoustic features.

[0143] For example, taking the acoustic feature as the Mel spectral feature as an example, Figure 11 An exemplary diagram illustrating the training principle of the dimensionality reduction coding network in an embodiment of the present invention is shown below. Figure 11As shown, a deep-autoencoder network structure can be used for unsupervised learning training of the model. Specifically, Mel spectral features can be extracted from the sample speech. These Mel spectral features are then input into an initial encoding network. After downsampling by the initial encoding network, the dimension of the Mel spectral features is mapped to a lower dimension, which can be denoted as representation h. Then, representation h is input into an initial decoding network. After upsampling by the initial decoding network, the Mel spectral features are reconstructed. The entire deep-autoencoder network is then trained based on a loss function, such as the mean squared error (MSE) loss function, until convergence. The converged initial encoding network can then be used as a dimensionality reduction encoding network to reduce the dimensionality of the original acoustic features, obtaining the target acoustic features. After dimensionality reduction and compression of the Mel spectral features through the dimensionality reduction encoding network, a more compact representation of the sample speech can be obtained, which can serve as the basis for prosodic feature extraction.

[0144] For example, the initial encoding network can be a network structure based on Convolutional Neural Network (CNN) and LSTM, and the initial decoding network can be a network structure based on CNN.

[0145] The speech synthesis apparatus provided by the present invention will be described below. The speech synthesis apparatus described below can be referred to in correspondence with the speech synthesis method described above.

[0146] Figure 12 An exemplary schematic diagram of the speech synthesis device provided in an embodiment of the present invention is shown, with reference to... Figure 12 As shown, the speech synthesis device 1200 may include: an acquisition module 1210 for acquiring target text; a prosody prediction module 1220 for calling a multi-level prosody prediction model to predict the prosody of the target text and obtain the prosodic features of the target text at at least two levels; wherein, different levels represent different time scales of the acoustic features of speech; the multi-level prosody prediction model is trained based on a first text sample and multi-level prosodic feature samples; and a generation module 1230 for generating speech corresponding to the target text based on the target text and the prosodic features at least two levels.

[0147] In one example embodiment, the prosody prediction module 1220 can be specifically used to: input the target text into the text encoding layer of a multi-level prosody prediction model to obtain the text features of the target text; for each of at least two levels, input the text features and the prosody features of the previous level of each level into the feature processing layer of the multi-level prosody prediction model to obtain the temporal fusion features output by the feature processing layer; the feature processing layer is used to temporally align and concatenate the text features and the prosody features of the previous level; input the temporal fusion features into the decoding layer of the multi-level prosody prediction model to obtain the prosody features corresponding to each level output by the decoding layer; wherein, the prosody features of the previous level are determined based on the prosody features of the target text at the sentence level.

[0148] In one example embodiment, the speech synthesis device 1200 may further include a first model training module, which can be used to train a multi-level prosodic prediction model. Exemplarily, the first model training module may include: a first acquisition unit, used to acquire a first text sample and acquire multi-level prosodic feature samples, the multi-level prosodic feature samples including a first prosodic feature sample at the sentence level and at least one second prosodic feature sample, wherein the second prosodic feature sample is a prosodic feature sample from a level other than the sentence level; and a training unit, used to train the initial multi-level prosodic prediction model for each level based on the first text sample and the prosodic feature sample of the level preceding each level, to obtain the multi-level prosodic prediction model; wherein the sentence level is the initial level.

[0149] In one example embodiment, the first acquisition unit may include: a first acquisition subunit, used to acquire target acoustic features of sample speech; and a second acquisition subunit, used to sequentially take each level in the sentence level and other levels as the target level, input the target acoustic features into the hierarchical prosodic prediction model corresponding to the target level, and obtain the prosodic feature samples corresponding to the target level output by the hierarchical prosodic prediction model; the hierarchical prosodic prediction model is used to downsample and quantize the target acoustic features; wherein, the downsampling scale of each target level is different.

[0150] In one example embodiment, the first acquisition subunit may be specifically used to: acquire the original acoustic features of the sample speech, and input the original acoustic features into a dimensionality reduction coding network to obtain the target acoustic features of the sample speech output by the dimensionality reduction coding network; wherein, the dimensionality reduction coding network is obtained by jointly training an initial coding network and an initial decoding network based on the sample acoustic features; the dimensionality reduction coding network is used to perform dimensionality reduction compression on the original acoustic features.

[0151] In one example embodiment, the speech synthesis device 1200 may further include a second model training module, which can be used to train a hierarchical prosody prediction model corresponding to the target level. For example, the second model training module can be specifically used for: inputting the target acoustic features into the initial hierarchical prosodic prediction model corresponding to the target level to obtain the first prosodic feature output by the initial hierarchical prosodic prediction model; obtaining the second prosodic feature, and performing temporal alignment and concatenation on the first and second prosodic features to obtain the third prosodic feature; the second prosodic feature is the prosodic feature output by the hierarchical prosodic prediction model of the level above the target level based on the target acoustic features; obtaining the second text sample, and inputting the second text sample into the text encoding network to obtain the sample text features output by the text encoding network; the text encoding network is used to extract the text features of the second text sample, and upsamples the text features of the second text sample to the same time scale as the target acoustic features; performing temporal alignment and concatenation on the third prosodic feature and the sample text features to obtain the processing result; inputting the processing result and the target acoustic features into the decoding network to obtain the output result of the decoding network; and adjusting the parameters of the initial hierarchical prosodic prediction model, the text encoding network, and the decoding network based on the output result and the loss function to obtain the hierarchical prosodic prediction model corresponding to the target level.

[0152] In one example embodiment, the generation module 1230 may include: a second acquisition unit, configured to input the target text and at least two levels of prosodic features into a speech synthesis model to obtain the target speech acoustic features output by the speech synthesis model; and a generation unit, configured to call a vocoder to generate speech corresponding to the target text from the target speech acoustic features; wherein the speech synthesis model is obtained by training an initial speech synthesis model based on a first text sample, multi-level prosodic feature samples, and the target acoustic features of the sample speech.

[0153] Figure 13 An exemplary schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown, such as... Figure 13As shown, the electronic device may include a processor 1310, a communication interface 1320, a memory 1330, and a communication bus 1340, wherein the processor 1310, the communication interface 1320, and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 may call logical instructions in the memory 1330 to execute the speech synthesis method provided in the above-described method embodiments. This method may include, for example, acquiring target text; calling a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels; generating speech corresponding to the target text based on the target text and the prosodic features at at least two levels; wherein different levels represent different time scales of the prosodic features; the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples.

[0154] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] Figure 14 A second schematic diagram of the structure of the electronic device provided in an embodiment of the present invention is shown as an example. Figure 14 As shown, the electronic device may include a processor 1410 and a speaker 1420 connected to the processor 1410.

[0156] The processor 1410 is used to acquire target text, call a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels, and generate speech corresponding to the target text based on the target text and the prosodic features at at least two levels. The different levels represent different time scales of the acoustic features of the speech. The multi-level prosodic prediction model is trained based on the first text sample and the multi-level prosodic feature samples.

[0157] Speaker 1420 is used to play the voice generated by processor 1410.

[0158] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided in the above-described method embodiments. This method may include, for example,: acquiring target text; calling a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels; generating speech corresponding to the target text based on the target text and the prosodic features at at least two levels; wherein, different levels represent different time scales of the prosodic features; the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples.

[0159] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech synthesis method provided in the above-described method embodiments. This method may include, for example,: acquiring target text; calling a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels; generating speech corresponding to the target text based on the target text and the prosodic features at at least two levels; wherein different levels represent different time scales of the prosodic features; and the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples.

[0160] For example, a computer-readable storage medium may include a non-transitory computer-readable storage medium.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the target text; A multi-level prosodic prediction model is invoked to predict the prosodic features of the target text at at least two levels; wherein, different levels represent different time scales of the prosodic features; the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples. Generate the speech corresponding to the target text based on the target text and the prosodic features of at least two levels; Among them, the different time scales in the prosodic features at least two levels include at least two of the sentence duration scale, paragraph duration scale, phrase duration scale, and syllable duration scale.

2. The speech synthesis method according to claim 1, characterized in that, The step of calling a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels includes: The target text is input into the text encoding layer of the multi-level prosody prediction model to obtain the text features of the target text; For each of the at least two levels, the text features and the prosodic features of the previous level are input into the feature processing layer of the multi-level prosodic prediction model to obtain the temporal fusion features output by the feature processing layer; the feature processing layer is used to perform temporal alignment and splicing of the text features and the prosodic features of the previous level. The temporal fusion features are input into the decoding layer of the multi-level prosodic prediction model to obtain the prosodic features corresponding to each level output by the decoding layer. The higher-level prosodic features are determined based on the prosodic features of the target text at the sentence level.

3. The speech synthesis method according to claim 1 or 2, characterized in that, The multi-level prosodic prediction model is trained based on the following steps: Obtain the first text sample and obtain the multi-level prosodic feature samples, wherein the multi-level prosodic feature samples include a first prosodic feature sample at the sentence level and at least one second prosodic feature sample; the second prosodic feature sample is a prosodic feature sample at a level other than the sentence level. Based on the first text sample and the prosodic feature sample of the previous level, the initial multi-level prosodic prediction model is trained for each level to obtain the multi-level prosodic prediction model; wherein, the sentence level is the initial level.

4. The speech synthesis method according to claim 3, characterized in that, The process of obtaining the multi-level prosodic feature samples includes: Obtain the target acoustic features of the sample speech; Each of the sentence level and other levels is sequentially taken as the target level. The target acoustic features are input into the hierarchical prosodic prediction model corresponding to the target level to obtain the prosodic feature sample corresponding to the target level output by the hierarchical prosodic prediction model. The hierarchical prosodic prediction model is used to downsample and quantize the target acoustic features. The downsampling scale is different for each of the target levels.

5. The speech synthesis method according to claim 4, characterized in that, The hierarchical prosodic prediction model corresponding to the target level is trained based on the following steps: The target acoustic features are input into the initial hierarchical prosodic prediction model corresponding to the target level to obtain the first prosodic feature output by the initial hierarchical prosodic prediction model. The second prosodic feature is obtained, and the first and second prosodic features are temporally aligned and concatenated to obtain the third prosodic feature; the second prosodic feature is the prosodic feature output by the hierarchical prosodic prediction model of the next level above the target level based on the target acoustic features. A second text sample is obtained and input into a text encoding network to obtain sample text features output by the text encoding network; the text encoding network is used to extract the text features of the second text sample and upsample the text features of the second text sample to the same time scale as the target acoustic features; The third prosodic feature and the sample text feature are time-aligned and concatenated to obtain the processing result; The processing result and the target acoustic features are input into the decoding network to obtain the output result of the decoding network; Based on the output results and loss function, the parameters of the initial hierarchical prosody prediction model, the text encoding network, and the decoding network are adjusted to obtain the hierarchical prosody prediction model corresponding to the target level.

6. The speech synthesis method according to claim 4, characterized in that, The acquisition of target acoustic features of sample speech includes: The original acoustic features of the sample speech are obtained and input into a dimensionality reduction coding network to obtain the target acoustic features of the sample speech output by the dimensionality reduction coding network; wherein, the dimensionality reduction coding network is obtained by jointly training an initial coding network and an initial decoding network based on the sample acoustic features; the dimensionality reduction coding network is used to perform dimensionality reduction and compression on the original acoustic features.

7. The speech synthesis method according to claim 1, characterized in that, The step of generating the speech corresponding to the target text based on the target text and the prosodic features of at least two levels includes: The target text and the prosodic features at least two levels are input into the speech synthesis model to obtain the target speech acoustic features output by the speech synthesis model. The vocoder is invoked to generate the speech corresponding to the target text from the acoustic features of the target speech. The speech synthesis model is obtained by training an initial speech synthesis model based on the first text sample, the multi-level prosodic feature samples, and the target acoustic features of the sample speech.

8. A speech synthesis device, characterized in that, include: The acquisition module is used to acquire the target text; The prosody prediction module is used to call a multi-level prosody prediction model to predict the prosody of the target text and obtain the prosodic features of the target text at at least two levels; wherein, different levels represent different time scales of speech acoustic features; the multi-level prosody prediction model is trained based on a first text sample and multi-level prosodic feature samples. A generation module is used to generate the speech corresponding to the target text based on the target text and the prosodic features of at least two levels; Among them, the different time scales in the prosodic features at least two levels include at least two of the sentence duration scale, paragraph duration scale, phrase duration scale, and syllable duration scale.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech synthesis method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, Includes a processor and a speaker connected to the processor; The processor is used to acquire target text, call a multi-level prosodic prediction model to predict the prosodic features of the target text at at least two levels, and generate speech corresponding to the target text based on the target text and the prosodic features at the at least two levels; wherein, different levels represent different time scales of the acoustic features of the speech; the multi-level prosodic prediction model is trained based on a first text sample and multi-level prosodic feature samples; Among them, the different time scales in the prosodic features at least two levels include at least two of the sentence duration scale, paragraph duration scale, phrase duration scale, and syllable duration scale; The speaker is used to play the voice.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 7.

Images