Speech synthesis method, system, electronic device and storage medium

Abstract

The present invention provides a speech synthesis method, system, electronic device, and storage medium, wherein the method comprises: inputting a target text to be synthesized into a text processing module in a speech synthesis model to obtain speech features output by the text processing module; inputting emotional information in the target text into an emotion encoder in the speech synthesis model to obtain emotion features output by the emotion encoder; inputting the speech features and emotion features into an acoustic model in the speech synthesis model to obtain acoustic features output by a combination of multiple expert modules in the acoustic model; and inputting the acoustic features into a vocoder in the speech synthesis model to obtain synthesized speech corresponding to the target text output by the vocoder. The present invention achieves emotionally rich and highly anthropomorphic speech synthesis, improving the emotional effect and speech synthesis quality in super-anthropomorphic, multi-emotional speech synthesis.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a speech synthesis method, system, electronic device and storage medium. Background Art

[0002] Speech not only conveys information but also carries a rich tapestry of emotions, such as joy, sadness, and anger. These emotions are crucial for interpersonal communication. Therefore, developing a speech synthesis method that can accurately synthesize speech with high naturalness and emotional authenticity is a critical research topic.

[0003] Currently, mainstream speech synthesis methods typically use a pre-trained emotion encoder to extract emotional features to directly control the emotion of the synthesized speech. This results in unnatural connections between different emotions, limits the improvement of the emotional effect in the synthesized speech, and results in poor speech synthesis quality. Summary of the Invention

[0004] The present invention provides a speech synthesis method, system, electronic device and storage medium to solve the defect in the prior art that the connection between different emotions is unnatural, resulting in poor emotional effect in the synthesized speech, and to improve the emotional effect in the synthesized speech, thereby improving the quality of speech synthesis.

[0005] The present invention provides a speech synthesis method, comprising:

[0006] Inputting the target text to be synthesized into the text processing module in the speech synthesis model to obtain the speech features output by the text processing module;

[0007] Inputting the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder;

[0008] Inputting the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain acoustic features output by a combination of multiple expert modules in the acoustic model;

[0009] Inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder;

[0010] The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0011] According to a speech synthesis method provided by the present invention, the speech synthesis model is trained based on the following steps:

[0012] Regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech;

[0013] Annotating the sample speech and the sample text to obtain sample annotation information;

[0014] The initialized language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

[0015] According to a speech synthesis method provided by the present invention, iteratively training an initialized language model based on the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model includes:

[0016] Inputting the sample text into the initial text processing module in the initialization language model to obtain the sample speech features output by the initial text processing module;

[0017] Inputting the regularized speech into the initial emotion encoder in the initialized language model to obtain the sample emotion features output by the initial emotion encoder;

[0018] Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initialized language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules;

[0019] Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information;

[0020] The initialized language model is iteratively trained according to the target loss value to obtain the speech synthesis model.

[0021] According to a speech synthesis method provided by the present invention, determining a target loss value based on the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information includes:

[0022] Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information;

[0023] Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information;

[0024] Determining a third loss value based on the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information;

[0025] determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label;

[0026] The target loss value is determined by combining the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0027] According to a speech synthesis method provided by the present invention, regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech includes:

[0028] Regularizing the emotional speech segment in the sample speech according to a preset regularization processing method to obtain the regularized speech;

[0029] The preset regularization processing method includes energy balancing processing.

[0030] According to a speech synthesis method provided by the present invention, the tagging of the sample speech and the sample text to obtain sample tagging information includes:

[0031] Performing language feature tagging on the sample text to obtain a language feature label;

[0032] Performing sentiment category labeling on the sample text to obtain a sentiment label;

[0033] Performing acoustic feature labeling on the sample speech to obtain an acoustic feature label;

[0034] Performing pronunciation duration labeling on the sample speech to obtain a pronunciation duration label;

[0035] Determining the sample labeling information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label;

[0036] The language feature annotation process includes at least one of pinyin annotation, prosody annotation and non-language information annotation.

[0037] According to a speech synthesis method provided by the present invention, inputting a target text to be synthesized into a text processing module in a speech synthesis model to obtain speech features output by the text processing module includes:

[0038] converting the target text into a phoneme sequence;

[0039] Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder;

[0040] Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module;

[0041] The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

[0042] The present invention also provides a speech synthesis system, comprising:

[0043] A first processing unit is configured to input a target text to be synthesized into a text processing module in a speech synthesis model, and obtain speech features output by the text processing module;

[0044] A second processing unit is configured to input the emotional information in the target text into an emotional encoder in the speech synthesis model to obtain emotional features output by the emotional encoder;

[0045] a third processing unit, configured to input the speech feature and the emotion feature into an acoustic model in the speech synthesis model, and obtain acoustic features output by a combination of multiple expert modules in the acoustic model;

[0046] a speech synthesis unit, configured to input the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder;

[0047] The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0048] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described speech synthesis methods when executing the program.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned speech synthesis methods when executed by a processor.

[0050] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0051] The speech synthesis method, system, electronic device and storage medium provided by the present invention extract fine-grained speech features through a text processing module and extract fine-grained emotional features through an emotional encoder to avoid information loss, thereby improving the naturalness and accuracy of emotional expression, and utilizing a hybrid expert layer deployed in an acoustic model to dynamically select the most appropriate expert module according to the emotional features to process speech features and emotional features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, making the emotional conversion in the generated acoustic features more natural, and utilizing a vocoder to convert the acoustic features into synthesized speech, thereby improving the naturalness and accuracy of emotional expression, solving the problem of unnatural connection between different emotions, realizing emotional and highly anthropomorphic speech synthesis, and improving the emotional effect and speech synthesis quality in super-anthropomorphic multi-emotion speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is one of the flow charts of the speech synthesis method provided by the present invention.

[0054] Figure 2 This is one of the structural diagrams of the speech synthesis model provided by the present invention.

[0055] Figure 3 This is the second structural diagram of the speech synthesis model provided by the present invention.

[0056] Figure 4 It is a flowchart of the speech synthesis model training steps provided by the present invention.

[0057] Figure 5 It is a schematic diagram of the distribution of text annotation information provided by the present invention.

[0058] Figure 6 It is a structural diagram of a speech synthesis system provided by the present invention.

[0059] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0061] Speech synthesis, also known as text-to-speech (TTS), aims to convert input text into fluent and natural-sounding speech output. It is a key technology for intelligent human-computer voice interaction. In the field of artificial intelligence, speech synthesis technology has been developing for several years. From early rule-based methods to modern statistical and deep learning-based models, technological advances have significantly improved the naturalness and comprehensibility of synthesized speech. However, while existing TTS systems can generate fluent and clear speech output, they still have limitations in expressing rich emotions. Speech is not just a conveyor of information; it also contains a rich range of emotions, such as joy, sadness, and anger, and the expression of these emotions is crucial for interpersonal communication. Therefore, it is particularly important to develop a highly human-like, multi-emotional speech synthesis technology that can accurately capture and reproduce these emotions.

[0062] The goal of hyper-humanized, multi-emotional speech synthesis technology is to create a voice that automatically adjusts its emotional tone, including pitch, rhythm, and intensity, based on emotional needs, resulting in highly natural and emotionally authentic speech. This technology not only enhances user experience in areas such as smart assistants, virtual assistants, audiobooks, game character dubbing, and automated customer service, but also plays a vital role in emotional communication and social interaction.

[0063] With the development of deep learning technology, particularly the emergence of recurrent neural networks (RNNs), convolutional neural networks (CNNs), and transformer models, the performance of text-to-speech (TTS) systems has been greatly improved. These models are capable of learning complex acoustic and linguistic features, providing a technical foundation for highly human-like, multi-emotional speech synthesis. Furthermore, with the advancement of computing power and the application of big data, it is possible to train more complex and accurate models, creating new opportunities for the development of highly human-like, multi-emotional speech synthesis technology.

[0064] However, the development of hyper-anthropomorphic, multi-emotional speech synthesis technology faces numerous challenges. First, the expression of emotion is highly subjective and complex, with significant variations across text and context. Second, accurately and effectively mapping emotions to speech parameters is a technical challenge. Furthermore, balancing the naturalness of speech and the authenticity of emotion, as well as achieving customization and optimization for different application scenarios, remain key challenges in the development process.

[0065] To address this, mainstream speech synthesis methods typically use a pre-trained emotion encoder to extract emotional features to directly control the emotion of the synthesized speech, without further consideration of the coherence between different emotions. Furthermore, these pre-determined features are fundamentally limited. The latent vectors extracted by the pre-trained emotion encoder lose some information, resulting in unnatural coherence between different emotions, limiting the emotional impact of the synthesized speech and resulting in poor speech synthesis quality.

[0066] In response to the above-mentioned defects, this embodiment provides a speech synthesis method, which improves the connection effect between different emotions of the model by using each expert module in the mixed expert layer to model each emotion separately, and performs speech synthesis by combining multiple expert modules in the mixed expert layer. It can effectively improve the emotional effect in super-anthropomorphic multi-emotion speech synthesis, realize emotional and highly anthropomorphic speech synthesis, thereby improving the quality of speech synthesis.

[0067] It should be noted that the executor of the method can be a speech synthesis system; the speech synthesis system can be a mobile phone, tablet computer, laptop computer, PDA, vehicle-mounted electronic equipment, wearable device, super mobile personal computer, server, network attached storage, personal computer, etc., and this application does not make specific limitations.

[0068] Figure 1 This is one of the flow charts of the speech synthesis method provided by the present invention; Figure 1 As shown, the speech synthesis method includes step 110, step 120, step 130 and step 140.

[0069] Step 110: input the target text to be synthesized into a text processing module in the speech synthesis model to obtain speech features output by the text processing module;

[0070] Step 120: input the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder;

[0071] Step 130: input the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain acoustic features output by a combination of multiple expert modules in the acoustic model;

[0072] Step 140: input the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder;

[0073] The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0074] It is understandable that the related technology directly controls the emotion of the synthesized speech by using a pre-trained emotion encoder to extract emotion features, without additional consideration of the connectivity between different emotions. Moreover, this predetermined feature is fundamentally limited. The hidden vector extracted by the pre-trained emotion encoder loses some information, resulting in unnatural connectivity between different emotions, limiting the improvement of the emotional effect in the synthesized speech, and poor speech synthesis quality.

[0075] To address this problem, an embodiment of the present application establishes a speech synthesis model including a text processing module, an emotion encoder, an acoustic model based on a hybrid expert model, and a vocoder, so as to extract fine-grained speech features through the text processing module and extract fine-grained emotion features through the emotion encoder to avoid information loss, so as to improve the naturalness and accuracy of emotion expression, and utilize the hybrid expert layer deployed in the acoustic model to dynamically select the most suitable expert module to process speech features and emotion features according to the emotion features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, so that the emotion conversion in the generated acoustic features is more natural, and utilize the vocoder to convert the acoustic features into synthesized speech, which not only improves the naturalness and accuracy of emotion expression, but also solves the problem of unnatural connection between different emotions, effectively improves the emotional effect in super-anthropomorphic multi-emotion speech synthesis, and realizes emotional and highly anthropomorphic speech synthesis, thereby improving the quality of speech synthesis.

[0076] Optionally, before executing step 110, a speech synthesis model may be first obtained through training. Specifically, an initialization language model may be constructed first, and the initialization language model here may include an initial text processing module, an initial emotion encoder, an initial acoustic model constructed based on a hybrid expert model, and an initial vocoder. The initial text processing module may be a model prepared for implicit feature extraction of text after parameter initialization, or a pre-trained model with the function of extracting implicit features of text; the initial emotion encoder may be a model prepared for emotion feature extraction after parameter initialization, or a pre-trained model with the function of extracting emotion features; the initial acoustic model may be a hybrid expert model prepared for multi-emotion feature combination output after parameter initialization, or a pre-trained hybrid expert model with the function of outputting multi-emotion feature combination; the initial vocoder may be a model prepared for speech conversion after parameter initialization, or a pre-trained model with the function of speech conversion, and this embodiment does not make any specific restrictions on this.

[0077] In addition, you can also collect sample speech and sample text corresponding to the sample speech. The sample text here refers to the content description text corresponding to the sample speech. The sample speech can be obtained by collecting and recording high-quality spoken language data in the form of conversations, which contains rich emotional information.

[0078] Subsequently, the sample speech and the sample text corresponding to the sample speech can be used to iteratively optimize the parameters of the initialized language model, thereby obtaining a speech synthesis model that can perform emotional and highly anthropomorphic speech synthesis.

[0079] The iterative parameter optimization here can be achieved by optimizing the loss function corresponding to the output layer of the initialized language model, or by jointly optimizing the loss functions corresponding to each module in the initialized language model. This embodiment does not specifically limit this.

[0080] Figure 2 This is one of the structural diagrams of the speech synthesis model provided by the present invention; Figure 3 This is the second structural diagram of the speech synthesis model provided by the present invention.

[0081] like Figure 2 As shown, in practical applications, the target text to be synthesized can be processed by the text processing module in the speech synthesis model to perform fine-grained intermediate implicit feature extraction to obtain corresponding speech features.

[0082] The text processing here can be achieved by feature encoding the target text, or by first feature encoding the target text and then performing duration prediction and duration adjustment based on the encoding features. This embodiment does not specifically limit this.

[0083] The speech features herein include various speech synthesis attribute information, including but not limited to pronunciation duration, prosodic features, pronunciation intensity, etc., which are not specifically limited in this embodiment.

[0084] In addition, the corresponding emotional information can be obtained from the target text, and the emotional information in the target text can be feature encoded through the emotional encoder in the speech synthesis model to extract fine-grained emotional features from the emotional information.

[0085] Among them, speech features and emotion features are both frame-level features.

[0086] It should be noted that the emotion feature extraction step in step 120 and the speech feature extraction step in step 120 may be executed in parallel or sequentially, and this embodiment does not specifically limit the execution order of the two.

[0087] like Figure 3 As shown, the acoustic model in the speech synthesis model includes multiple operation layers and mixed expert layers, wherein the mixed expert layer can be a layer set below the first operation layer.

[0088] After acquiring emotional features through the emotion encoder and speech features through the text processing module, the speech and emotion features are fed into the acoustic model of the speech synthesis model. The first operation layer (also referred to as operation layer 1) in the acoustic model then uses the speech and emotion features as input information to the hybrid expert layer. The hybrid expert layer then uses a gating network to determine the probability of using each expert module. The probability is then used to determine the combination of each expert module's input and output. The outputs of the multiple expert modules are weighted and added together as the output of the hybrid expert layer. The output of the hybrid expert layer is then used as the input to the next operation layer of the hybrid expert layer. After sequential processing by the next operation layer and subsequent operation layers in the acoustic model, the final acoustic features are formed. The acoustic features herein may be mel-spectrograms, etc., which are not specifically limited in this embodiment.

[0089] Here, by taking advantage of the fact that each expert model can accurately model different emotional data distributions and patterns to dynamically extract acoustic features, we can more finely control the acoustic features of the synthesized speech, maintain emotional consistency, avoid unnatural emotional conversions, and significantly improve the naturalness, adaptability, computational efficiency, and synthesis quality of the emotional expression of the synthesized speech. This improves the emotional effect, anthropomorphism, and generalization ability of the model when synthesizing emotional colloquial texts, enabling the model to perform highly anthropomorphic and emotional speech synthesis in a variety of emotional texts, thereby better adapting to the complexity of the task.

[0090] After obtaining the acoustic features through the acoustic model, the vocoder can be used to synthesize the acoustic features into the corresponding speech waveform, thereby obtaining a synthesized speech whose content is consistent with that of the target text and whose emotional connection is natural.

[0091] The method provided in this embodiment extracts fine-grained speech features through a text processing module and extracts fine-grained emotional features through an emotional encoder to avoid information loss, thereby improving the naturalness and accuracy of emotional expression, and utilizing a hybrid expert layer deployed in an acoustic model to dynamically select the most appropriate expert module according to the emotional features to process speech features and emotional features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, making the emotional conversion in the generated acoustic features more natural, and utilizing a vocoder to convert the acoustic features into synthesized speech, thereby improving the naturalness and accuracy of emotional expression, solving the problem of unnatural connection between different emotions, realizing emotional and highly anthropomorphic speech synthesis, and improving the emotional effect and speech synthesis quality in super-anthropomorphic multi-emotional speech synthesis.

[0092] Figure 4 This is a flow chart of the speech synthesis model training steps provided by the present invention. Figure 4 As shown, the training steps of the speech synthesis model specifically include step 410, step 420, and step 430.

[0093] Step 410: performing regularization processing on the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech;

[0094] Step 420: annotate the sample speech and the sample text to obtain sample annotation information;

[0095] Step 430 : Iteratively train the initialized language model based on the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

[0096] Alternatively, because audio data for different emotional sounds may vary in sound quality and volume, the transitions between emotions may not be natural. For example, when transitioning from neutral to happy, the overall mood may suddenly become high, reducing the naturalness of the synthesized speech. Therefore, it is necessary to perform the same standard normalization processing on the different emotional speech segments in the sample speech to reduce the differences in sound quality and volume between the different emotional speech segments and obtain normalized speech.

[0097] The regularization processing here includes but is not limited to energy equalization processing, adding smooth transition signals between different emotional speech segments, spectrum shaping processing, etc., which are not specifically limited in this embodiment.

[0098] For example, in some embodiments, regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech includes:

[0099] Regularizing the emotional speech segment in the sample speech according to a preset regularization processing method to obtain the regularized speech;

[0100] The preset regularization processing method includes energy balancing processing.

[0101] Optionally, during the regularization process, the energy information of the emotional speech segments in the sample speech can be obtained, and the energy information of the emotional speech segments in the sample speech can be regularized according to the energy balancing method to reduce the energy difference under different emotional states, thereby reducing the differences in listening experience, volume, etc. of different emotional speech segments, and obtaining regularized speech with more uniform energy distribution and more natural connection between emotions, thereby improving the listening experience, naturalness and coherence of the synthesized speech.

[0102] In addition, the sample speech and the sample text can also be annotated to obtain sample annotation information containing rich annotation information, which can then assist in the training of the speech synthesis model to effectively improve the accuracy and efficiency of the language model training.

[0103] Exemplarily, the marking process in step 420 specifically includes:

[0104] Performing language feature tagging on the sample text to obtain a language feature label;

[0105] Performing sentiment category labeling on the sample text to obtain a sentiment label;

[0106] Performing acoustic feature labeling on the sample speech to obtain an acoustic feature label;

[0107] Performing pronunciation duration labeling on the sample speech to obtain a pronunciation duration label;

[0108] Determining the sample labeling information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label;

[0109] The language feature annotation process includes at least one of pinyin annotation, prosody annotation and non-language information annotation.

[0110] Optionally, when annotating the sample text, sentiment and speech feature annotations may be performed, thereby obtaining text annotation information including sentiment tags and language feature tags. The speech feature annotations include at least one of pinyin annotation, prosody annotation, and non-verbal information annotation. Non-verbal information annotation refers to the annotation of information other than language information, such as laughter, crying, and other information.

[0111] Figure 5 is a schematic diagram of the text annotation information distribution provided by the present invention; for example, Figure 5 As shown, for sample text containing both content description and emotional information, when annotating at the pinyin level, the content description can be annotated with the corresponding pinyin form, and non-verbal information such as emotional information can be annotated with the corresponding string form, such as using "XS" to mark laughter and "KS" to mark crying. When annotating at the prosody level, different marking forms can be used for different forms of prosody. For example, "L0" can be used to mark prosodic characters, which are single characters in Chinese, such as "you, me, him,..."; "L1" can be used to mark prosodic words, which are groups of several syllables closely connected in spoken language. These are usually minimal phrases consisting of a single word or a content word plus an affix, such as "Sleep, sleep, dead skin, delicious words,..."; and "L2" can be used to mark secondary prosodic phrases, which have a noticeable pause, but the pause is less than the first preset duration (e.g., 90ms), and the pitch does not dip or dips slightly. The final rhyme cannot be the end of a sentence. Using "L3" prosodic phrases, which have long pauses exceeding the first preset duration (e.g., 90ms), or fricative stops exceeding the second preset duration (e.g., 120ms), with a relatively complete pitch dip, the final rhyme can be used as the sentence end. Using "L4" intonation phrases, which are used to mark the end of a sentence. When tagging at the emotion level, different tagging forms can be used for clauses with different emotions.

[0112] When labeling the sample speech, acoustic feature extraction, such as mel-spectrogram extraction, may be performed on the sample speech to obtain an acoustic feature label. The sample speech may also be labeled with the pronunciation duration of the phoneme to obtain a pronunciation duration label.

[0113] After obtaining language feature labels, emotion labels, acoustic feature labels and pronunciation duration labels through annotation processing, the language feature labels, emotion labels, acoustic feature labels and pronunciation duration labels can be integrated to obtain complete sample annotation information for assisting speech synthesis model training.

[0114] Here, by performing multi-dimensional annotation processing on the sample speech and sample text, rich and complete sample annotation information can be obtained, which can assist in the training of the speech synthesis model and effectively improve the accuracy and efficiency of the language model training.

[0115] After obtaining the regularized speech and sample annotation information, a training dataset can be constructed by combining the sample text, regularized speech, and sample annotation information to iteratively optimize the parameters of the initialized language model based on the training dataset, thereby obtaining a speech synthesis model that can perform emotional and highly anthropomorphic speech synthesis.

[0116] The method provided in this embodiment reduces the differences in listening experience, volume, etc. of different emotional speech segments through regularization processing, and obtains rich and complete sample annotation information through multi-dimensional annotation processing. It then combines sample text, regularized speech and sample annotation information to construct a training data set, and iteratively optimizes the initialized language model, ultimately achieving a speech synthesis model that can generate emotional and highly anthropomorphic speech, significantly improving the naturalness, coherence and training efficiency of speech synthesis.

[0117] In some embodiments, step 430 specifically includes:

[0118] Inputting the sample text into the initial text processing module in the initialization language model to obtain the sample speech features output by the initial text processing module;

[0119] Inputting the regularized speech into the initial emotion encoder in the initialized language model to obtain the sample emotion features output by the initial emotion encoder;

[0120] Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initialized language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules;

[0121] Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information;

[0122] The initialized language model is iteratively trained according to the target loss value to obtain the speech synthesis model.

[0123] Optionally, during the training of the speech synthesis model, the sample text may be first converted into a sample phoneme sequence, and the sample phoneme sequence may be input into the initial text encoder in the initial text processing module, so that the initial text encoder encodes the sample phoneme sequence to obtain hidden features, i.e., sample language features. After obtaining the sample language features output by the initial text encoder, the sample language features may be input into the initial duration prediction module in the initial text processing module, so that the initial duration prediction module uses the sample language features to predict the pronunciation duration of each phoneme, and obtain corresponding sample pronunciation duration information. After obtaining the sample pronunciation duration information output by the initial duration prediction module, the sample language features and the sample pronunciation duration information may be jointly input into the initial duration adjustment module in the initial text processing module, so that the initial duration adjustment module uses the sample pronunciation duration information to adjust the sample language features, and obtain corresponding sample speech features.

[0124] In addition, the regularized speech can be input into the initial emotion encoder to perform feature encoding on the emotion information in the regularized speech, so as to extract fine-grained sample emotion features from the emotion information.

[0125] After obtaining the sample emotion features through the initial emotion encoder and the sample speech features through the initial text processing module, the sample speech features and the sample emotion features can be sent to the initial acoustic model, and the sample speech features and the sample emotion features are input as input information to the mixed expert layer by the first operation layer in the initial acoustic model. The initial mixed expert layer determines the use probability of each initial expert module through the gated network, and determines the combination of the input and output of each expert module by using the probability, obtains the routing decision value of the corresponding input of each initial expert module and the sample acoustic sub-feature of the corresponding output, and performs weighted addition and combination of the outputs of multiple initial expert modules as the output of the initial mixed expert layer, and uses the output of the initial mixed expert layer as the input of the next operation layer of the initial mixed expert layer, so as to form the final sample acoustic features after sequential operation processing by the next operation layer and the operation layer after the next operation layer in the initial acoustic model. The sample acoustic features here can be mel-spectrograms, etc., which are not specifically limited in this embodiment.

[0126] After obtaining the sample speech features, sample emotion features, routing decision values, sample acoustic sub-features, and sample acoustic features, the sample speech features, sample emotion features, routing decision values, sample acoustic sub-features, sample acoustic features, and sample annotation information can be combined to perform a loss function calculation to obtain a multi-dimensional loss value, and the target loss value is determined by combining the multi-dimensional loss value to iteratively train the initialized language model based on the target loss value to obtain a speech synthesis model that is emotional and highly anthropomorphic.

[0127] The multi-dimensional loss value here includes the loss values ​​corresponding to some initial modules or the loss values ​​corresponding to all initial modules in the initialized language model, which is not specifically limited in this embodiment.

[0128] Exemplarily, the step of determining the target loss value specifically includes:

[0129] Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information;

[0130] Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information;

[0131] Determining a third loss value based on the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information;

[0132] determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label;

[0133] The target loss value is determined by combining the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0134] Optionally, the sample acoustic features output by the initial acoustic model may be compared with the acoustic feature labels in the sample annotation information to determine the acoustic feature loss value corresponding to the initial acoustic model as the first loss value.

[0135] In addition, the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information can be compared to determine the loss value corresponding to the initial duration prediction model as the second loss value.

[0136] In addition, the language features corresponding to the sample speech features output by the initial text encoder can be compared with the language feature labels in the sample annotation information, and the emotion categories corresponding to the sample emotion features output by the initial emotion encoder can be compared with the emotion labels in the sample annotation information to determine the loss value corresponding to the encoder as the third loss value.

[0137] In addition, the correctness of the routing decision can be evaluated based on the routing decision values ​​corresponding to the inputs of each initial expert module in the initial acoustic model and the corresponding output sample acoustic sub-features and acoustic feature labels to determine the loss value corresponding to the initial acoustic model, thereby obtaining the fourth loss value.

[0138] Subsequently, the first loss value, the second loss value, the third loss value and the fourth loss value are fused to obtain the target loss value. The fusion here can be direct addition or weighted addition, etc., which is not specifically limited in this embodiment.

[0139] For example, the target loss value may be calculated as:

[0140] ;

[0141] in, is the first loss value; is the second loss value; is the third loss value; is the fourth loss value.

[0142] The method provided in this embodiment performs iterative model training by combining the loss values ​​corresponding to multi-dimensional modules to quickly obtain an emotional and highly anthropomorphic speech synthesis model, thereby achieving multiple improvements in the accuracy, naturalness, and efficiency of speech synthesis.

[0143] In some embodiments, step 110 specifically includes:

[0144] converting the target text into a phoneme sequence;

[0145] Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder;

[0146] Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module;

[0147] The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

[0148] like Figure 3 As shown in the figure, the text processing module includes a text encoder, a duration prediction module and a duration adjustment module (LR for short). The text encoder is used to encode the phoneme sequence corresponding to the text to obtain hidden features, that is, language features; the duration prediction module is used to predict the utterance duration; and the duration adjustment module is used to adjust the utterance duration.

[0149] Optionally, when extracting speech features from text, the target text may be first converted into a phoneme sequence, which is then input into a text encoder in the text processing module to encode the phoneme sequence to obtain latent features, i.e., language features.

[0150] After obtaining the language features output by the text encoder, the language features can be input into the duration prediction module in the text processing module, so that the duration prediction module can use the language features to predict the pronunciation duration of each phoneme and obtain the corresponding pronunciation duration information.

[0151] After obtaining the pronunciation duration information output by the duration prediction module, the language features and the pronunciation duration information can be jointly input into the duration adjustment module in the text processing module, so that the duration adjustment module applies the pronunciation duration information to adjust the language features to obtain corresponding speech features, so that the pronunciation duration information of the final speech features matches the pronunciation duration information predicted by the duration prediction module.

[0152] The method provided in this embodiment achieves precise control of the pronunciation duration and emotion of the target text by integrating a text encoder, a duration prediction and an adjustment module, thereby generating natural and fluent diverse emotional speech.

[0153] The speech synthesis system provided by the present invention is described below. The speech synthesis system described below and the speech synthesis method described above can be referenced to each other.

[0154] Figure 6 Schematic diagram of the structure of a speech synthesis system provided by the present invention; Figure 6 As shown, the system includes:

[0155] The first processing unit 610 is used to input the target text to be synthesized into the text processing module in the speech synthesis model, and obtain the speech features output by the text processing module;

[0156] The second processing unit 620 is used to input the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder;

[0157] The third processing unit 630 is used to input the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model;

[0158] The speech synthesis unit 640 is configured to input the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder;

[0159] The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0160] The system provided in this embodiment extracts fine-grained speech features through a text processing module and extracts fine-grained emotional features through an emotional encoder to avoid information loss, thereby improving the naturalness and accuracy of emotional expression. The system also utilizes a hybrid expert layer deployed in the acoustic model to dynamically select the most appropriate expert module to process speech features and emotional features based on the emotional features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, making the emotional conversion in the generated acoustic features more natural. The system also utilizes a vocoder to convert the acoustic features into synthesized speech, thereby improving the naturalness and accuracy of emotional expression, solving the problem of unnatural connection between different emotions, realizing emotional and highly anthropomorphic speech synthesis, and improving the emotional effect and speech synthesis quality in super-anthropomorphic multi-emotional speech synthesis.

[0161] In some embodiments, the speech synthesis system further includes a model training unit, specifically configured to:

[0162] Regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech;

[0163] Annotating the sample speech and the sample text to obtain sample annotation information;

[0164] The initialized language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

[0165] In some embodiments, the model training unit is further configured to:

[0166] Inputting the sample text into the initial text processing module in the initialization language model to obtain the sample speech features output by the initial text processing module;

[0167] Inputting the regularized speech into the initial emotion encoder in the initialized language model to obtain the sample emotion features output by the initial emotion encoder;

[0168] Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initialized language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules;

[0169] Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information;

[0170] The initialized language model is iteratively trained according to the target loss value to obtain the speech synthesis model.

[0171] In some embodiments, the model training unit is further configured to:

[0172] Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information;

[0173] Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information;

[0174] Determining a third loss value based on the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information;

[0175] determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label;

[0176] The target loss value is determined by combining the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0177] In some embodiments, the model training unit is further configured to:

[0178] Regularizing the emotional speech segment in the sample speech according to a preset regularization processing method to obtain the regularized speech;

[0179] The preset regularization processing method includes energy balancing processing.

[0180] In some embodiments, the model training unit is further configured to:

[0181] Performing language feature tagging on the sample text to obtain a language feature label;

[0182] Performing sentiment category labeling on the sample text to obtain a sentiment label;

[0183] Performing acoustic feature labeling on the sample speech to obtain an acoustic feature label;

[0184] Performing pronunciation duration labeling on the sample speech to obtain a pronunciation duration label;

[0185] Determining the sample labeling information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label;

[0186] The language feature annotation process includes at least one of pinyin annotation, prosody annotation and non-language information annotation.

[0187] In some embodiments, the first processing unit is specifically configured to:

[0188] converting the target text into a phoneme sequence;

[0189] Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder;

[0190] Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module;

[0191] The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

[0192] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0193] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communications bus 740. The processor 710 may call logic instructions in the memory 730 to execute a speech synthesis method, which includes: inputting a target text to be synthesized into a text processing module in a speech synthesis model to obtain speech features output by the text processing module; inputting emotional information in the target text into an emotion encoder in the speech synthesis model to obtain emotion features output by the emotion encoder; inputting the speech features and the emotion features into an acoustic model in the speech synthesis model to obtain acoustic features output by a combination of multiple expert modules in the acoustic model; and inputting the acoustic features into a vocoder in the speech synthesis model to obtain synthesized speech corresponding to the target text output by the vocoder. The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0194] Furthermore, the logic instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0195] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above methods, which includes: inputting the target text to be synthesized into the text processing module in the speech synthesis model to obtain the speech features output by the text processing module; inputting the emotional information in the target text into the emotion encoder in the speech synthesis model to obtain the emotional features output by the emotion encoder; inputting the speech features and the emotional features into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; inputting the acoustic features into the vocoder in the speech synthesis model to obtain the synthesized speech corresponding to the target text output by the vocoder; wherein, the speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0196] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech synthesis method provided by the above-mentioned methods, the method comprising: inputting the target text to be synthesized into the text processing module in the speech synthesis model to obtain the speech features output by the text processing module; inputting the emotional information in the target text into the emotion encoder in the speech synthesis model to obtain the emotional features output by the emotion encoder; inputting the speech features and the emotional features into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; inputting the acoustic features into the vocoder in the speech synthesis model to obtain the synthesized speech corresponding to the target text output by the vocoder; wherein the speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0197] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0198] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Inputting the target text to be synthesized into the text processing module in the speech synthesis model to obtain the speech features output by the text processing module; Inputting the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; Inputting the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain acoustic features output by a combination of multiple expert modules in the acoustic model; Inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech; The step of inputting the target text to be synthesized into a text processing module in the speech synthesis model and obtaining speech features output by the text processing module includes: converting the target text into a phoneme sequence; Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder; Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module; The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

2. The speech synthesis method according to claim 1, wherein: The speech synthesis model is trained based on the following steps: Regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech; Annotating the sample speech and the sample text to obtain sample annotation information; The initialized language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

3. The speech synthesis method according to claim 2, wherein: The iterative training of the initialized language model according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model includes: Inputting the sample text into the initial text processing module in the initialization language model to obtain the sample speech features output by the initial text processing module; Inputting the regularized speech into the initial emotion encoder in the initialized language model to obtain the sample emotion features output by the initial emotion encoder; Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initialized language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules; Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information; The initialized language model is iteratively trained according to the target loss value to obtain the speech synthesis model.

4. The speech synthesis method according to claim 3, wherein: The determining of a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information includes: Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information; Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information; Determining a third loss value based on the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information; determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label; The target loss value is determined by combining the first loss value, the second loss value, the third loss value, and the fourth loss value.

5. The speech synthesis method according to claim 2, wherein: The step of regularizing the emotional speech segment in the sample speech to obtain a regularized speech corresponding to the sample speech includes: Regularizing the emotional speech segment in the sample speech according to a preset regularization processing method to obtain the regularized speech; The preset regularization processing method includes energy balancing processing.

6. The speech synthesis method according to claim 2, wherein: The tagging of the sample speech and the sample text to obtain sample tag information includes: Performing language feature tagging on the sample text to obtain a language feature label; Performing sentiment category labeling on the sample text to obtain a sentiment label; Performing acoustic feature labeling on the sample speech to obtain an acoustic feature label; Performing pronunciation duration labeling on the sample speech to obtain a pronunciation duration label; Determining the sample labeling information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label; The language feature annotation process includes at least one of pinyin annotation, prosody annotation and non-language information annotation.

7. A speech synthesis system, characterized in that: include: A first processing unit is configured to input a target text to be synthesized into a text processing module in a speech synthesis model, and obtain speech features output by the text processing module; A second processing unit is configured to input the emotional information in the target text into an emotional encoder in the speech synthesis model to obtain emotional features output by the emotional encoder; a third processing unit, configured to input the speech feature and the emotion feature into an acoustic model in the speech synthesis model, and obtain acoustic features output by a combination of multiple expert modules in the acoustic model; a speech synthesis unit, configured to input the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech; The first processing unit is specifically configured to: converting the target text into a phoneme sequence; Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder; Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module; The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech synthesis method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 6 is implemented.

Images