Speech synthesis method, related device, equipment and storage medium

Abstract

The invention discloses a speech synthesis method, a related device, equipment and a storage medium, and the method comprises the steps: predicting the pronunciation features of a to-be-synthesized character sequence based on a feature prediction model; wherein the to-be-synthesized character sequence is a text sequence or a phoneme sequence, and the pronunciation features at least comprise feature information, related to pronunciation content and pronunciation rhythm, of the to-be-synthesized character sequence; predicting the pronunciation features and sequence hidden layer features of the character sequence to be synthesized based on an acoustic model to obtain acoustic features; wherein the sequence hidden layer features are hidden layer features obtained in the process of processing the character sequence to be synthesized by the feature prediction model; and performing waveform recovery on the acoustic features based on a vocoder to obtain synthetic speech. According to the scheme, the stability and naturalness of speech synthesis can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method and related devices, equipment and storage media. Background Art

[0002] Speech synthesis technology, also known as text to speech (TTS) technology, is an important branch of the field of artificial intelligence. It involves converting text information into natural and fluent speech output, enabling computers to "speak". It can be applied to, but not limited to: assisting visually impaired people in reading, voice navigation systems, automatic customer service systems, audio book production, etc.

[0003] However, the speech synthesized by existing speech synthesis technology still has stability problems such as multiple readings, missed readings, and wrong readings, as well as naturalness problems such as unnaturalness and re-reading errors. In view of this, how to improve the stability and naturalness of speech synthesis has become an issue that needs to be solved urgently. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech synthesis method and related devices, equipment and storage media, which can improve the stability and naturalness of speech synthesis.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech synthesis method, comprising: predicting the pronunciation features of a character sequence to be synthesized based on a feature prediction model; wherein the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized; predicting the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on an acoustic model to obtain acoustic features; wherein the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model; and performing waveform recovery on the acoustic features based on a vocoder to obtain synthesized speech.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speech synthesis device, including: a pronunciation prediction module, an acoustic prediction module and a waveform recovery module, the pronunciation prediction module is used to predict the pronunciation features of the character sequence to be synthesized based on a feature prediction model; wherein the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized; the acoustic prediction module is used to predict the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain acoustic features; wherein the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model; the waveform recovery module is used to perform waveform recovery on the acoustic features based on the vocoder to obtain synthesized speech.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech synthesis method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium, which stores program instructions that can be executed by a processor, and the program instructions are used to implement the speech synthesis method of the first aspect.

[0009] The above scheme predicts the pronunciation features of the character sequence to be synthesized based on the feature prediction model, and the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, and predicts the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain the acoustic features, and the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model, and then the waveform of the acoustic features is restored based on the vocoder to obtain the synthesized speech. On the one hand, the pronunciation features predicted based on the character sequence to be synthesized at least include the feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, so Using this pronunciation feature for subsequent acoustic feature prediction and speech waveform recovery can ensure that feature information related to the pronunciation content, that is, the semantics of the character sequence to be synthesized, is referenced during the speech synthesis process, which helps to improve the stability of speech synthesis as much as possible. It can also ensure that feature information related to the pronunciation rhythm, that is, the rhythm of the character sequence to be synthesized, is referenced during the speech synthesis process, which helps to improve the naturalness of speech synthesis as much as possible. On the other hand, the acoustic model not only predicts based on pronunciation features but also further supplements with sequence hidden features. Therefore, it can further constrain the acoustic feature prediction process in combination with the sequence hidden features, which helps to avoid the predicted acoustic features from deviating from the character sequence to be synthesized as much as possible. Therefore, the stability and naturalness of speech synthesis can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the speech synthesis method of the present application; Figure 2a It is a schematic diagram of an embodiment of a feature extraction model of the present application; Figure 2b is a schematic diagram of an embodiment of a speaker enhancement model of the present application; Figure 2c It is a schematic diagram of an embodiment of a feature prediction model of the present application; Figure 2d is a schematic diagram of an embodiment of an acoustic model of the present application; Figure 2e is a schematic diagram of an embodiment of joint training of the present application; Figure 2f It is a process diagram of an embodiment of the speech synthesis method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech synthesis method of the present application. Specifically, it may include the following steps: Step S11: predicting the pronunciation features of the character sequence to be synthesized based on the feature prediction model.

[0015] In the disclosed embodiment, the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation feature at least includes feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized. It should be noted that the text sequence is a sequence composed of text characters as basic units, and the phoneme sequence is a sequence composed of phoneme characters as basic units.

[0016] In an implementation scenario, the feature prediction model may include but is not limited to network architectures such as Transformer, and may specifically adopt network structures such as but not limited to Llama. There is no limitation on the network architecture included in the feature prediction model and the network structure adopted.

[0017] In an implementation scenario, the feature prediction model can be trained based on the first pronunciation feature of the first character sequence and the first speech to which it belongs, with the first pronunciation feature as the optimization target. The first pronunciation feature of the first speech can be extracted by the feature extraction model. The feature extraction model is trained with the second speech as the model input and with the first attribute marked by the second speech as the optimization target and the second attribute marked by the second speech as the degradation target. The first attribute may include at least one of the second character sequence and the real fundamental frequency of the second speech, and the second attribute may include at least the real speaker of the second speech. In other words, the training target of the feature extraction model is to extract content-related and rhythm-related but speaker-independent pronunciation features from the input speech as much as possible, so that the first pronunciation feature extracted by the feature extraction model for the first speech can also be ensured to be as relevant as possible to the content and rhythm of the first speech but irrelevant to the speaker. Then, the feature prediction model is trained with the first pronunciation feature as the optimization target, so that the feature prediction model can be ensured to predict content-related and rhythm-related but speaker-independent pronunciation features as much as possible.

[0018] In a specific implementation scenario, please refer to Figure 2a , Figure 2a Schematic diagram of an embodiment of the feature extraction model of the present application. Figure 2a As shown, before using the feature extraction model to extract features from the first speech, the feature extraction model can be trained first. Specifically, the second speech hidden layer features of the second speech can be extracted first, and encoded based on the second speech hidden layer features to obtain the second speech coding features, and then predicted based on the second speech coding features to obtain the prediction information of the second speech about the first attribute and the prediction information about the second attribute. On this basis, the network parameters of the feature extraction model can be adjusted based on at least a first loss measured by the difference between the labeled information and the predicted information of the second speech about the first attribute, and a second loss measured by the difference between the labeled information and the predicted information of the second speech about the second attribute, and the gradient of the second loss is flipped during the parameter adjustment process. It should be noted that the feature extraction model can specifically include Figure 2a The open source models, encoders, and vector quantization (VQ) of Hubert, WavLM, AE, etc. for extracting speech hidden features are described in the above method. The network parameters of the feature extraction model are adjusted based on at least the first loss measured by the difference between the labeled information and the predicted information of the second speech about the first attribute, and the second loss measured by the difference between the labeled information and the predicted information of the second speech about the second attribute. The gradient of the second loss is reversed during the parameter adjustment process, which can force the feature extraction model to extract feature information that is irrelevant to the speaker but related to the pronunciation content and pronunciation rhythm as much as possible to obtain pronunciation features.

[0019] As a possible example, open source models such as Hubert, WavLM, and AE can be used to extract features of the second speech to obtain the second speech hidden layer features of the second speech. For ease of description, the second speech hidden layer features can be recorded as .

[0020] As a possible example, the second speech hidden layer feature can be encoded by an encoder to obtain a second speech coding feature. For the convenience of description, the second speech coding feature can be recorded as .

[0021] As a possible example, when the first attribute annotated by the second speech includes the second character sequence of the second speech, the second speech coding feature The prediction information of the second speech about the first attribute (i.e., the predicted character sequence of the second speech) can be obtained through the text enhancement model. For example, when the second character sequence is a text sequence of the second speech, the prediction information of the second speech about the first attribute can include the second speech coding feature The text sequence predicted by the text enhancement model; or, when the second character sequence is a phoneme sequence of the second voice, the prediction information of the second voice about the first attribute may include the second voice coding feature The phoneme sequence predicted by the text enhancement model. It should be noted that the text enhancement model may include but is not limited to: a convolutional layer, a long short-term memory network, etc., and the specific structure of the text enhancement model is not limited here.

[0022] As a possible example, when the first attribute annotated by the second speech includes the true fundamental frequency of the second speech, the second speech coding feature The prediction information of the second speech about the first attribute (i.e., the predicted fundamental frequency of the second speech) can be obtained through the fundamental frequency enhancement model. It should be noted that the fundamental frequency enhancement model may include but is not limited to: a convolutional layer, a long short-term memory network, etc., and the specific structure of the fundamental frequency enhancement model is not limited here.

[0023] As a possible example, the second speech coding feature The speaker enhancement model can also be used to obtain prediction information about the second attribute of the second speech (i.e., the predicted speaker of the second speech). It should be noted that the speaker enhancement model may include but is not limited to: a convolutional layer, a long short-term memory network, etc., and the specific structure of the speaker enhancement model is not limited here.

[0024] As a possible example, see Figure 2a , after obtaining the second speech coding feature Afterwards, the second speech coding feature On the other hand, vector quantization can be performed to obtain a second pronunciation feature, and decoding can be performed based on the second pronunciation feature to obtain a predicted acoustic feature of the second speech, so that a third loss can be obtained based on the second pronunciation feature and the vector quantization codebook, and a fourth loss can be obtained based on the real acoustic feature and the predicted acoustic feature of the second speech, and then the network parameters of the feature extraction model can be adjusted based on the first loss, the second loss, the third loss and the fourth loss. It should be noted that in the embodiments of the present disclosure, the "acoustic features" such as acoustic features, predicted acoustic features, and real acoustic features can specifically be any acoustic features such as MFCC and FBank, and the specific type of "acoustic features" is not limited here. For the sake of ease of description, the training loss of the feature extraction model It can be expressed as: …… (1) In the above formula (1), represents the first loss of the character sequence in the first attribute (i.e., the difference between the second character sequence and the predicted character sequence is measured by a loss function such as cross entropy), represents the first loss about the fundamental frequency in the first attribute (i.e., the difference between the true fundamental frequency and the predicted fundamental frequency is measured by a loss function such as L1), represents the second loss about the second attribute (i.e., the difference between the real speaker and the predicted speaker is measured by a loss function such as cross entropy), represents the fourth loss (e.g., the difference between the real acoustic features and the predicted acoustic features can be measured by a loss function such as L1), represents the third loss. As a possible example, It can be expressed as: …… (2) In the above formula (2), sg represents the stop-gradient operation. , is a hyperparameter, is the codebook, , where n represents the number of codebooks, d represents the dimension of a codebook, and have the same dimension. It should be noted that in the third loss When measured by the above formula (2), when adjusting the network parameters of the feature extraction model, the first term in formula (2) (i.e. ) only acts on the codebook, and the second term (i.e. ) only works on encoders. See Figure 2b , Figure 2b Schematic diagram of an embodiment of the speaker enhancement model of the present application. Figure 2b As shown, the speaker enhancement model can be composed of a gradient reversal layer and a speaker prediction network, and then when the network parameters of the feature extraction model are adjusted based on the aforementioned training loss, the feature extraction model can be forced to extract feature information that is not related to the speaker but is related to the pronunciation content and pronunciation rhythm as much as possible. In addition, in the process of adjusting parameters based on training loss, the aforementioned open source models such as Hubert, WavLM, AE, etc. for extracting hidden layer features of speech do not participate in parameter updates. After training convergence, the network parameters of the feature extraction model can be kept fixed.

[0025] In a specific implementation scenario, after the feature extraction model training converges, the feature extraction model can be used to extract features of the first speech. Specifically, the first speech latent features of the first speech can be extracted, and encoded based on the first speech latent features to obtain first speech coding features, and then vector quantization is performed based on the first speech coding features to obtain the first pronunciation features. As a possible example, please continue to refer to Figure 2a The first speech can be processed by an open source model such as Hubert, WavLM, AE, etc. in the feature extraction model for extracting speech hidden features to extract the first speech hidden features. The first hidden features can be processed by the encoder in the feature extraction model to extract the first speech coding features. The first speech coding features can be vector quantized to obtain the first pronunciation features, so that the first pronunciation features can contain feature information related to the pronunciation content and pronunciation rhythm of the first speech as much as possible without being related to the speaker of the first speech.

[0026] In a specific implementation scenario, after extracting the first pronunciation feature of the first speech, the feature prediction model can be trained based on the first character sequence of the first speech (such as the text sequence, phoneme sequence, etc. of the first speech) and the first pronunciation feature of the first speech, and the first pronunciation feature is used as the optimization target during the training process, that is, the feature prediction model is forced to predict the first pronunciation feature as accurately as possible based on the first character sequence.

[0027] As a possible example, the first character sequence can be predicted based on the feature prediction model to obtain the predicted pronunciation features of the first speech, and then the network parameters of the feature prediction model can be adjusted based on the difference between the first pronunciation features of the first speech and the predicted pronunciation features.

[0028] As another possible example, different from the previous training method, see Figure 2c , Figure 2cSchematic diagram of an embodiment of the feature prediction model of the present application. As mentioned above, the character sequence of the first speech may include a real text sequence, a real phoneme sequence, etc. of the first speech, and the real text sequence or the real phoneme sequence of the first speech may be selected as the first character sequence with a preset probability. For example, the real text sequence may be selected as the first character sequence with a probability of 50%, and the real phoneme sequence may be selected as the first character sequence with a probability of 50%. Then, the first character sequence is predicted based on the feature prediction model to obtain the predicted pronunciation features of the first character sequence, and then prediction is made based on the predicted pronunciation features to obtain a predicted character sequence aligned with the predicted pronunciation features. For example, a CTC (Connectionist Temporal Classification) module can be used to predict the predicted pronunciation features to obtain a predicted character sequence aligned with the predicted pronunciation features. Then, based on the difference between the predicted pronunciation features and the first pronunciation features of the first speech, and the difference between the annotated character sequence and the predicted character sequence of the first speech, the network parameters of the feature prediction model can be adjusted. When the first character sequence is a real text sequence, the predicted character sequence is a predicted phoneme sequence, and the annotated character sequence is a real phoneme sequence. When the first character sequence is a real phoneme sequence, the predicted character sequence is a predicted text sequence, and the annotated character sequence is a real text sequence. For ease of description, the training loss of the feature prediction model is It can be expressed as: …… (3) In the above formula (3), represents the loss value measured by the difference between the predicted pronunciation feature of the first speech and the first pronunciation feature based on a loss function such as cross entropy, Represents the loss value measured by the difference between the labeled character sequence and the predicted character sequence of the first voice based on CTC loss. The above method further adds the constraint of alignment between text and phonemes on the basis of the feature prediction model. On the one hand, it can assist the feature prediction model to better learn to align the predicted pronunciation features with the input character sequence, and ensure that the feature prediction model predicts feature information related to the pronunciation content and pronunciation rhythm but has nothing to do with the speaker. On the other hand, it can assist the feature prediction model to better learn the conversion relationship from phoneme to text and from text to phoneme, and assist the feature prediction model to better learn pronunciation. In addition, as the input of the synthesis system, the text allows the feature prediction model to better understand the semantic information conveyed in the text, making the synthesis effect more natural and the stress more reasonable.

[0029] It should be noted that the feature prediction model can be obtained through the above process steps, and then the character sequence to be synthesized can be predicted based on the feature prediction model to obtain the pronunciation features, and the pronunciation features contain feature information related to the character sequence to be synthesized and the pronunciation content and pronunciation rhythm, but have nothing to do with the speaker.

[0030] Step S12: predicting the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain acoustic features.

[0031] In the disclosed embodiment, the sequence hidden layer features are the hidden layer features obtained in the process of processing the character sequence to be synthesized by the feature prediction model. In addition, the acoustic model may include but is not limited to the network architecture of Transformer, Conformer, and convolutional neural network, and the network structure of the acoustic model is not limited here. In other words, the pronunciation features and the sequence hidden layer features of the character sequence to be synthesized can be input into the acoustic model to obtain the acoustic features predicted by the acoustic model, such as MFCC and other types, and the specific types of acoustic features are not limited here. It should be noted that the acoustic features can be trained based on the third voice and its third pronunciation features and the third character sequence after the feature extraction model training converges, and the third pronunciation features can be obtained by extracting the features of the third voice by the feature extraction model. For details, please refer to the relevant description of the aforementioned feature extraction model, which will not be repeated here. In addition, the third character sequence can be a phoneme sequence, or the third character sequence can also be a text sequence, which is not limited here.

[0032] In an implementation scenario, please refer to Figure 2d , Figure 2d Schematic diagram of an embodiment of an acoustic model of the present application. Figure 2d As shown, as a possible example, feature extraction can be performed on the third speech based on the feature extraction model to obtain a third pronunciation feature of the third speech, and the third pronunciation feature and the third character sequence of the third speech can be predicted based on the acoustic model to obtain the predicted acoustic feature of the third speech, and then the network parameters of the acoustic model can be adjusted based on the acoustic reconstruction loss measured by the difference between the real acoustic feature of the third speech and the predicted acoustic feature.

[0033] In another implementation scenario, different from the aforementioned implementation, as another possible example, the third speech may be first feature extracted based on the feature extraction model to obtain the third pronunciation feature of the third speech, and the third pronunciation feature and the third character sequence of the third speech may be predicted based on the acoustic model to obtain the predicted acoustic feature of the third speech, and then the network parameters of the acoustic model may be adjusted based on the acoustic reconstruction loss measured by the difference between the real acoustic feature and the predicted acoustic feature of the third speech, and the generative adversarial loss of the generative adversarial network, and the discriminator in the generative adversarial network is at least used to discriminate whether the predicted acoustic feature is the real acoustic feature. For ease of description, the training loss of the acoustic model It can be expressed as: …… (4) In the above formula (4), represents the acoustic reconstruction loss measured by the L1 loss function based on the difference between the true acoustic features and the predicted acoustic features of the third speech. represents the generation loss of the generator in the generative adversarial network, Represents the discriminant loss of the discriminator in the generative adversarial network. The generator is used to generate acoustic features that are difficult for the discriminator to distinguish between true and false as much as possible, while the discriminator is used to distinguish the true and false attributes of the acoustic features as accurately as possible. For details, please refer to the technical details of the generative adversarial network, which will not be repeated here. The above method, based on the acoustic reconstruction loss measured by the difference between the real acoustic features and the predicted acoustic features of the third speech, and the generative adversarial loss of the generative adversarial network, adjusts the network parameters of the acoustic model, which can be combined with the generative adversarial network to ensure the authenticity of the generated predicted acoustic features as much as possible, which helps to improve the audio quality of subsequent speech synthesis.

[0034] In another implementation scenario, after the feature prediction model and the acoustic model are trained separately, they can be jointly trained based on the fourth voice data. The fourth voice data can include a fourth voice and a fourth pronunciation feature of the fourth voice, a fourth character sequence, a labeled character sequence and a real acoustic feature. The fourth pronunciation feature can be extracted by the feature extraction model, and when the fourth character sequence is a real phoneme sequence of the fourth voice, the labeled character sequence of the fourth voice is a real text sequence of the fourth voice, and when the fourth character sequence is a real text sequence of the fourth voice, the labeled character sequence of the fourth voice is a real phoneme sequence of the fourth voice. That is to say, after the joint training is completed, the feature prediction model, the acoustic model and the following vocoder can be used to implement speech synthesis through the process steps of the embodiment of the present disclosure. Please refer to Figure 2e , Figure 2e Schematic diagram of an embodiment of joint training of the present application. Figure 2eAs shown, the fourth character sequence can be predicted based on the feature prediction model to obtain the predicted pronunciation features of the fourth character sequence, and then prediction can be made based on the predicted pronunciation features of the fourth character sequence to obtain a predicted character sequence aligned with the predicted pronunciation features of the fourth character sequence, and the predicted pronunciation features of the fourth character sequence and the sample sequence hidden features of the fourth character sequence can be predicted based on the acoustic model to obtain the predicted acoustic features of the fourth speech, and the hidden features of the sample sequence are the hidden features obtained in the process of processing the fourth character sequence with the feature prediction model, and when the fourth character sequence is the real text sequence of the fourth speech, the predicted character sequence is the predicted phoneme sequence. When the fourth character sequence is the real phoneme sequence of the fourth voice, the predicted character sequence is the predicted text sequence, and then the network parameters of the feature prediction model and the acoustic model can be adjusted based on the difference between the predicted pronunciation feature and the fourth pronunciation feature, the difference between the predicted character sequence and the annotated character sequence of the fourth voice, the difference between the predicted acoustic feature of the fourth voice and the real acoustic feature, and the generative adversarial loss of the generative adversarial network, and the discriminator in the generative adversarial network is at least used to discriminate whether the predicted acoustic feature is a real acoustic feature, which can further improve the model performance of the feature prediction model and the acoustic model, so as to help improve the audio quality of the subsequent synthesized speech. It should be noted that the detailed details of the above-mentioned joint training can be combined with the training process of the aforementioned feature prediction model and the prediction process of the aforementioned acoustic model. The main difference is that the character sequence can be directly input when the acoustic model is trained alone, while the acoustic model needs to input the hidden features obtained by the character sequence processed by the feature prediction model during the joint training process. Other details can be found in the above-mentioned related descriptions, which will not be repeated here.

[0035] Step S13: Perform waveform restoration on the acoustic features based on the vocoder to obtain synthesized speech.

[0036] Specifically, the vocoder may be a parallel vocoder, an autoregressive vocoder, etc., and the specific type of the vocoder is not limited here. Figure 2f , Figure 2f 1 is a process diagram of an embodiment of the speech synthesis method of the present application. Figure 2f As shown, the character sequence to be synthesized can be predicted by the feature prediction model to obtain the pronunciation features, the pronunciation features and the sequence hidden features of the character sequence to be synthesized can be predicted by the acoustic model to obtain the acoustic features, and finally the acoustic features can be processed by the vocoder to restore the speech waveform, i.e., the synthesized speech.

[0037] The above scheme predicts the pronunciation features of the character sequence to be synthesized based on the feature prediction model, and the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, and predicts the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain the acoustic features, and the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model, and then the waveform of the acoustic features is restored based on the vocoder to obtain the synthesized speech. On the one hand, the pronunciation features predicted based on the character sequence to be synthesized at least include the feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, so Using this pronunciation feature for subsequent acoustic feature prediction and speech waveform recovery can ensure that feature information related to the pronunciation content, that is, the semantics of the character sequence to be synthesized, is referenced during the speech synthesis process, which helps to improve the stability of speech synthesis as much as possible. It can also ensure that feature information related to the pronunciation rhythm, that is, the rhythm of the character sequence to be synthesized, is referenced during the speech synthesis process, which helps to improve the naturalness of speech synthesis as much as possible. On the other hand, the acoustic model not only predicts based on pronunciation features but also further supplements with sequence hidden features. Therefore, it can further constrain the acoustic feature prediction process in combination with the sequence hidden features, which helps to avoid the predicted acoustic features from deviating from the character sequence to be synthesized as much as possible. Therefore, the stability and naturalness of speech synthesis can be improved.

[0038] See also Figure 3 , Figure 3 It is a schematic diagram of the framework of an embodiment of the speech synthesis device of the present application. The speech synthesis device 30 includes: a pronunciation prediction module 31, an acoustic prediction module 32 and a waveform recovery module 33. The pronunciation prediction module 31 is used to predict the pronunciation features of the character sequence to be synthesized based on the feature prediction model; wherein the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized; the acoustic prediction module 32 is used to predict the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain the acoustic features; wherein the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model; the waveform recovery module 33 is used to perform waveform recovery on the acoustic features based on the vocoder to obtain the synthesized speech.

[0039] In the above scheme, the speech synthesis device 30 predicts the pronunciation features of the character sequence to be synthesized based on the feature prediction model, and the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, and the pronunciation features and the sequence hidden features of the character sequence to be synthesized are predicted based on the acoustic model to obtain acoustic features, and the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model, and then the acoustic features are restored based on the vocoder to obtain the synthesized speech. On the one hand, the pronunciation features predicted based on the character sequence to be synthesized at least include the features related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized. Information, so the subsequent acoustic feature prediction and speech waveform recovery based on this pronunciation feature can ensure that the feature information related to the pronunciation content is referenced during the speech synthesis process, that is, the semantics of the character sequence to be synthesized, which helps to improve the stability of speech synthesis as much as possible. It can also ensure that the feature information related to the pronunciation rhythm is referenced during the speech synthesis process, that is, the rhythm of the character sequence to be synthesized, which helps to improve the naturalness of speech synthesis as much as possible. On the other hand, the acoustic model not only predicts based on the pronunciation features but also further supplements the sequence hidden features. Therefore, it can further constrain the acoustic feature prediction process in combination with the sequence hidden features, which helps to avoid the predicted acoustic features from deviating from the character sequence to be synthesized as much as possible. Therefore, the stability and naturalness of speech synthesis can be improved.

[0040] In some disclosed embodiments, a feature prediction model is trained based on a first character sequence and a first pronunciation feature of a first speech to which it belongs, with the first pronunciation feature being an optimization target. The first pronunciation feature of the first speech is extracted by a feature extraction model. The feature extraction model is trained with a second speech as a model input and with a first attribute annotated by the second speech as an optimization target and a second attribute annotated by the second speech as a degradation target. The first attribute includes at least one of a second character sequence and a real fundamental frequency of the second speech, and the second attribute includes at least a real speaker of the second speech.

[0041] In some disclosed embodiments, the speech synthesis device 30 includes a first hidden layer extraction module for extracting a first speech hidden layer feature of a first speech; the speech synthesis device 30 includes a first speech encoding module for encoding based on the first speech hidden layer feature to obtain a first speech encoding feature; the speech synthesis device 30 includes a first vector quantization module for performing vector quantization based on the first speech encoding feature to obtain a first pronunciation feature.

[0042] In some disclosed embodiments, the speech synthesis device 30 includes a second hidden layer extraction module for extracting second speech hidden layer features of the second speech; the speech synthesis device 30 includes a second speech encoding module for encoding based on the second speech hidden layer features to obtain second speech encoding features; the speech synthesis device 30 includes a speech attribute prediction module for predicting based on the second speech encoding features to obtain prediction information of the second speech about the first attribute and prediction information about the second attribute; the speech synthesis device 30 includes a first model optimization module for adjusting the network parameters of the feature extraction model based at least on a first loss measured by the difference between the labeled information and the predicted information of the second speech about the first attribute, and a second loss measured by the difference between the labeled information and the predicted information of the second speech about the second attribute; wherein the gradient of the second loss is reversed during the parameter adjustment process.

[0043] In some disclosed embodiments, the speech synthesis device 30 includes a first acoustic prediction module, which is used to perform vector quantization based on the second speech coding feature to obtain a second pronunciation feature, and to perform decoding based on the second pronunciation feature to obtain a predicted acoustic feature of the second speech; the speech synthesis device 30 includes a loss function measurement module, which is used to obtain a third loss based on the second pronunciation feature and the vector quantization codebook, and to obtain a fourth loss based on the actual acoustic features and predicted acoustic features of the second speech; the first model optimization module is specifically used to adjust the network parameters of the feature extraction model based on the first loss, the second loss, the third loss and the fourth loss.

[0044] In some disclosed embodiments, the speech synthesis device 30 includes a character sequence selection module, which is used to select a real text sequence or a real phoneme sequence of the first speech as the first character sequence with a preset probability; the speech synthesis device 30 includes a first pronunciation prediction module, which is used to predict the first character sequence based on a feature prediction model to obtain a predicted pronunciation feature of the first character sequence; the speech synthesis device 30 includes a first sequence prediction module, which is used to predict based on the predicted pronunciation feature to obtain a predicted character sequence aligned with the predicted pronunciation feature; the speech synthesis device 30 includes a second model optimization module, which is used to adjust the network parameters of the feature prediction model based on the difference between the predicted pronunciation feature and the first pronunciation feature of the first speech, and the difference between the marked character sequence and the predicted character sequence of the first speech; wherein, when the first character sequence is a real text sequence, the predicted character sequence is a predicted phoneme sequence, and the marked character sequence is a real phoneme sequence, and when the first character sequence is a real phoneme sequence, the predicted character sequence is a predicted text sequence, and the marked character sequence is a real text sequence.

[0045] In some disclosed embodiments, the acoustic model is trained after the feature extraction model training converges, and the speech synthesis device 30 includes a pronunciation feature extraction module, which is used to extract features of the third speech based on the feature extraction model to obtain a third pronunciation feature of the third speech; the speech synthesis device 30 includes a second acoustic prediction module, which is used to predict the third pronunciation feature and the third character sequence of the third speech based on the acoustic model to obtain the predicted acoustic feature of the third speech; wherein the third character sequence is a real text sequence or a real phoneme sequence of the third speech; the speech synthesis device 30 includes a third model optimization module, which is used to adjust the network parameters of the acoustic model based on the acoustic reconstruction loss measured by the difference between the real acoustic features of the third speech and the predicted acoustic features, and the generative adversarial loss of the generative adversarial network; wherein the discriminator in the generative adversarial network is at least used to discriminate whether the predicted acoustic features are real acoustic features.

[0046] In some disclosed embodiments, the feature prediction model and the acoustic model are trained separately and then jointly trained based on fourth speech data. The fourth speech data includes a fourth speech and a fourth pronunciation feature of the fourth speech, a fourth character sequence, a labeled character sequence and real acoustic features. The fourth pronunciation feature is extracted by a feature extraction model, and when the fourth character sequence is a real phoneme sequence of the fourth speech, the labeled character sequence of the fourth speech is a real text sequence of the fourth speech, and when the fourth character sequence is a real text sequence of the fourth speech, the labeled character sequence of the fourth speech is a real phoneme sequence of the fourth speech.

[0047] In some disclosed embodiments, the speech synthesis device 30 includes a second pronunciation prediction module, which is used to predict the fourth character sequence based on the feature prediction model to obtain the predicted pronunciation features of the fourth character sequence; the speech synthesis device 30 includes a second sequence prediction module, which is used to predict based on the predicted pronunciation features of the fourth character sequence to obtain a predicted character sequence aligned with the predicted pronunciation features of the fourth character sequence; the speech synthesis device 30 includes a third acoustic prediction module, which is used to predict the predicted pronunciation features of the fourth character sequence and the sample sequence hidden features of the fourth character sequence based on the acoustic model to obtain the predicted acoustic features of the fourth speech; wherein the sample sequence hidden features are the process of the feature prediction model processing the fourth character sequence The hidden features obtained in the fourth speech are obtained by using the predicted character sequence as the predicted phoneme sequence when the fourth character sequence is the real text sequence of the fourth speech, and the predicted character sequence is the predicted text sequence when the fourth character sequence is the real phoneme sequence of the fourth speech; the speech synthesis device 30 includes a fourth model optimization module, which is used to adjust the network parameters of the feature prediction model and the acoustic model based on the difference between the predicted pronunciation feature and the fourth pronunciation feature, the difference between the predicted character sequence and the marked character sequence of the fourth speech, the difference between the predicted acoustic feature of the fourth speech and the real acoustic feature, and the generative adversarial loss of the generative adversarial network; wherein the discriminator in the generative adversarial network is at least used to discriminate whether the predicted acoustic feature is a real acoustic feature.

[0048] See also Figure 4 , Figure 4 : is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned speech synthesis method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to an office notebook, a smart phone, a tablet computer, a learning machine, a server, etc., and the specific type of the electronic device 40 is not limited here.

[0049] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned speech synthesis method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0050] In the above scheme, the electronic device 40 predicts the pronunciation features of the character sequence to be synthesized based on the feature prediction model, and the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, and the pronunciation features and the sequence hidden features of the character sequence to be synthesized are predicted based on the acoustic model to obtain acoustic features, and the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model, and then the waveform of the acoustic features is restored based on the vocoder to obtain synthesized speech. On the one hand, the pronunciation features predicted based on the character sequence to be synthesized at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized. Therefore, the pronunciation feature is used for subsequent acoustic feature prediction and speech waveform recovery, which can ensure that the feature information related to the pronunciation content is referenced during the speech synthesis process, that is, the semantics of the character sequence to be synthesized, which helps to improve the stability of speech synthesis as much as possible. It can also ensure that the feature information related to the pronunciation rhythm is referenced during the speech synthesis process, that is, the rhythm of the character sequence to be synthesized, which helps to improve the naturalness of speech synthesis as much as possible. On the other hand, the acoustic model not only predicts based on the pronunciation feature but also further supplements the sequence hidden layer feature. Therefore, it can further constrain the acoustic feature prediction process in combination with the sequence hidden layer feature, which helps to avoid the predicted acoustic feature from deviating from the character sequence to be synthesized as much as possible. Therefore, the stability and naturalness of speech synthesis can be improved.

[0051] See also Figure 5 , Figure 51 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned speech synthesis method embodiments.

[0052] In the above scheme, the computer-readable storage medium 50 predicts the pronunciation features of the character sequence to be synthesized based on the feature prediction model, and the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized, and the pronunciation features and the sequence hidden features of the character sequence to be synthesized are predicted based on the acoustic model to obtain acoustic features, and the sequence hidden features are the hidden features obtained in the process of processing the character sequence to be synthesized by the feature prediction model, and then the acoustic features are restored based on the vocoder to obtain the synthesized speech. On the one hand, the pronunciation features predicted based on the character sequence to be synthesized at least include the feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized. Feature information, so the subsequent acoustic feature prediction and speech waveform recovery based on this pronunciation feature can ensure that the feature information related to the pronunciation content is referenced during the speech synthesis process, that is, the semantics of the character sequence to be synthesized, which helps to improve the stability of speech synthesis as much as possible. It can also ensure that the feature information related to the pronunciation rhythm is referenced during the speech synthesis process, that is, the rhythm of the character sequence to be synthesized, which helps to improve the naturalness of speech synthesis as much as possible. On the other hand, the acoustic model not only predicts based on the pronunciation features but also further supplements the sequence hidden features. Therefore, it can further constrain the acoustic feature prediction process in combination with the sequence hidden features, which helps to avoid the predicted acoustic features from deviating from the character sequence to be synthesized as much as possible. Therefore, the stability and naturalness of speech synthesis can be improved.

[0053] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0054] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0055] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0056] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0057] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0058] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0059] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech synthesis method, characterized in that: include: Predicting pronunciation features of a character sequence to be synthesized based on a feature prediction model; wherein the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized; Predicting the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain acoustic features; wherein the sequence hidden features are the hidden features obtained in the process of the feature prediction model processing the character sequence to be synthesized; The waveform of the acoustic feature is restored based on the vocoder to obtain synthesized speech.

2. The method according to claim 1, characterized in that The feature prediction model is trained based on the first pronunciation feature of the first character sequence and the first speech to which it belongs, with the first pronunciation feature as the optimization target. The first pronunciation feature of the first speech is extracted by a feature extraction model. The feature extraction model is trained with the second speech as the model input and with the first attribute marked by the second speech as the optimization target and the second attribute marked by the second speech as the degradation target. The first attribute includes at least one of the second character sequence and the real fundamental frequency of the second speech, and the second attribute includes at least the real speaker of the second speech.

3. The method according to claim 2, characterized in that The step of extracting the first pronunciation feature comprises: Extracting first speech hidden layer features of the first speech; Encoding based on the first speech hidden layer feature to obtain a first speech coding feature; Vector quantization is performed based on the first speech coding feature to obtain the first pronunciation feature.

4. The method according to claim 2 or 3, characterized in that: The training steps of the feature extraction model include: Extracting second speech hidden layer features of the second speech; Encoding based on the second speech hidden layer feature to obtain a second speech coding feature; Predicting based on the second speech coding feature to obtain prediction information of the second speech with respect to the first attribute and prediction information with respect to the second attribute; Adjust the network parameters of the feature extraction model based at least on a first loss measured by the difference between the labeling information and the prediction information of the second speech regarding the first attribute, and a second loss measured by the difference between the labeling information and the prediction information of the second speech regarding the second attribute; wherein the gradient of the second loss is flipped during the parameter adjustment process.

5. The method according to claim 4, characterized in that Before adjusting the network parameters of the feature extraction model based at least on the first loss measured by the difference between the labeling information and the prediction information of the second speech with respect to the first attribute, and the second loss measured by the difference between the labeling information and the prediction information of the second speech with respect to the second attribute, the method further includes: Performing vector quantization based on the second speech coding feature to obtain a second pronunciation feature, and performing decoding based on the second pronunciation feature to obtain a predicted acoustic feature of the second speech; Based on the second pronunciation feature and the vector quantized codebook, a third loss is obtained, and based on the real acoustic feature and the predicted acoustic feature of the second speech, a fourth loss is obtained; The adjusting of the network parameters of the feature extraction model based on at least a first loss measured by a difference between the labeling information and the prediction information of the second speech with respect to the first attribute, and a second loss measured by a difference between the labeling information and the prediction information of the second speech with respect to the second attribute, comprises: Based on the first loss, the second loss, the third loss and the fourth loss, a network parameter of the feature extraction model is adjusted.

6. The method according to claim 2, characterized in that The training steps of the feature prediction model include: Selecting a real text sequence or a real phoneme sequence of the first speech as the first character sequence with a preset probability; Predicting the first character sequence based on the feature prediction model to obtain predicted pronunciation features of the first character sequence; Perform prediction based on the predicted pronunciation feature to obtain a predicted character sequence aligned with the predicted pronunciation feature; Based on the difference between the predicted pronunciation feature of the first speech and the first pronunciation feature, and the difference between the marked character sequence of the first speech and the predicted character sequence, the network parameters of the feature prediction model are adjusted; wherein, when the first character sequence is the real text sequence, the predicted character sequence is a predicted phoneme sequence, and the marked character sequence is the real phoneme sequence, and when the first character sequence is the real phoneme sequence, the predicted character sequence is a predicted text sequence, and the marked character sequence is the real text sequence.

7. The method according to claim 2, characterized in that: The acoustic model is trained after the feature extraction model training converges, and the training steps of the acoustic model include: Performing feature extraction on the third speech based on the feature extraction model to obtain a third pronunciation feature of the third speech; Predicting the third pronunciation feature and the third character sequence of the third speech based on the acoustic model to obtain the predicted acoustic feature of the third speech; wherein the third character sequence is a real text sequence or a real phoneme sequence of the third speech; Based on the acoustic reconstruction loss measured by the difference between the real acoustic features and the predicted acoustic features of the third speech, and the generative adversarial loss of the generative adversarial network, the network parameters of the acoustic model are adjusted; wherein the discriminator in the generative adversarial network is at least used to determine whether the predicted acoustic features are the real acoustic features.

8. The method according to claim 2, characterized in that: The feature prediction model and the acoustic model are trained separately and then jointly trained based on fourth speech data. The fourth speech data includes a fourth speech and a fourth pronunciation feature, a fourth character sequence, a marked character sequence and a real acoustic feature of the fourth speech. The fourth pronunciation feature is extracted by the feature extraction model, and when the fourth character sequence is a real phoneme sequence of the fourth speech, the marked character sequence of the fourth speech is a real text sequence of the fourth speech, and when the fourth character sequence is a real text sequence of the fourth speech, the marked character sequence of the fourth speech is a real phoneme sequence of the fourth speech.

9. The method according to claim 8, characterized in that The steps of the joint training include: Predicting the fourth character sequence based on the feature prediction model to obtain predicted pronunciation features of the fourth character sequence; Predicting based on the predicted pronunciation features of the fourth character sequence to obtain a predicted character sequence aligned with the predicted pronunciation features of the fourth character sequence, and predicting the predicted pronunciation features of the fourth character sequence and the sample sequence hidden features of the fourth character sequence based on the acoustic model to obtain predicted acoustic features of the fourth speech; wherein the sample sequence hidden features are hidden features obtained in the process of processing the fourth character sequence by the feature prediction model, and when the fourth character sequence is a real text sequence of the fourth speech, the predicted character sequence is a predicted phoneme sequence, and when the fourth character sequence is a real phoneme sequence of the fourth speech, the predicted character sequence is a predicted text sequence; Based on the difference between the predicted pronunciation feature and the fourth pronunciation feature, the difference between the predicted character sequence and the annotated character sequence of the fourth voice, the difference between the predicted acoustic feature of the fourth voice and the real acoustic feature, and the generative adversarial loss of the generative adversarial network, the network parameters of the feature prediction model and the acoustic model are adjusted; wherein the discriminator in the generative adversarial network is at least used to discriminate whether the predicted acoustic feature is the real acoustic feature.

10. A speech synthesis device, characterized in that: include: A pronunciation prediction module, used to predict the pronunciation features of the character sequence to be synthesized based on the feature prediction model; wherein the character sequence to be synthesized is a text sequence or a phoneme sequence, and the pronunciation features at least include feature information related to the pronunciation content and pronunciation rhythm of the character sequence to be synthesized; An acoustic prediction module, used for predicting the pronunciation features and the sequence hidden features of the character sequence to be synthesized based on the acoustic model to obtain acoustic features; wherein the sequence hidden features are the hidden features obtained in the process of the feature prediction model processing the character sequence to be synthesized; The waveform recovery module is used to perform waveform recovery on the acoustic features based on a vocoder to obtain synthesized speech.

11. An electronic device, characterized in that: The invention at least comprises a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech synthesis method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech synthesis method according to any one of claims 1 to 9.

Images