Speech synthesis method and system, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting a to-be-synthesized target text into a text processing module in a speech synthesis model, and obtaining speech features outputted by the text processing module; inputting emotional information in the target text into an emotional encoder in the speech synthesis model to obtain emotional features output by the emotional encoder; inputting the speech features and the emotional features into an acoustic model in the speech synthesis model to obtain acoustic features combined and output by a plurality of expert modules in the acoustic model; and inputting the acoustic features into a vocoder in the speech synthesis model to obtain synthesized speech corresponding to the target text output by the vocoder. According to the invention, speech synthesis rich in emotions and high in personification degree is realized, and the emotional effect and speech synthesis quality in hypersimulation multi-emotion speech synthesis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a speech synthesis method, system, electronic device and storage medium. Background Art

[0002] Speech is not only the transmission of information, but also contains rich emotional colors, such as joy, sadness, anger, etc. These emotions are crucial for interpersonal interaction. Therefore, developing a speech synthesis method that can accurately synthesize speech with high naturalness and emotional authenticity is an important topic that needs to be studied urgently.

[0003] At present, mainstream speech synthesis methods usually use a pre-trained emotion encoder to extract emotional features to directly control the emotion of the synthesized speech, which results in unnatural connections between different emotions, limits the improvement of emotional effects in synthesized speech, and results in poor speech synthesis quality. Summary of the invention

[0004] The present invention provides a speech synthesis method, system, electronic device and storage medium, which are used to solve the defect in the prior art that the connection between different emotions is unnatural, resulting in poor emotional effect in the synthesized speech, and to improve the emotional effect in the synthesized speech, so as to improve the speech synthesis quality.

[0005] The present invention provides a speech synthesis method, comprising: Inputting the target text to be synthesized into a text processing module in the speech synthesis model to obtain speech features output by the text processing module; Inputting the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; Inputting the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; Inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0006] According to a speech synthesis method provided by the present invention, the speech synthesis model is trained based on the following steps: Regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech; Annotating the sample speech and the sample text to obtain sample annotation information; The initialization language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

[0007] According to a speech synthesis method provided by the present invention, the initialization language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model, including: Inputting the sample text into an initial text processing module in the initialization language model to obtain sample speech features output by the initial text processing module; Inputting the regularized speech into the initial emotion encoder in the initialization language model to obtain the sample emotion features output by the initial emotion encoder; Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initial language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules; Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information; According to the target loss value, the initialized language model is iteratively trained to obtain the speech synthesis model.

[0008] According to a speech synthesis method provided by the present invention, determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information includes: Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information; Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information; Determine a third loss value according to the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information; Determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label; The target loss value is determined by combining the first loss value, the second loss value, the third loss value and the fourth loss value.

[0009] According to a speech synthesis method provided by the present invention, the emotional speech segment in the sample speech is regularized to obtain a regularized speech corresponding to the sample speech, including: According to a preset regularization processing method, regularization processing is performed on the emotional speech segment in the sample speech to obtain the regularized speech; Among them, the preset regularization processing method includes energy balancing processing.

[0010] According to a speech synthesis method provided by the present invention, the step of annotating the sample speech and the sample text to obtain sample annotation information includes: Performing language feature annotation processing on the sample text to obtain a language feature label; Performing sentiment category labeling processing on the sample text to obtain a sentiment label; Performing acoustic feature labeling processing on the sample speech to obtain an acoustic feature label; Performing pronunciation duration labeling processing on the sample speech to obtain a pronunciation duration label; Determining the sample annotation information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label; The language feature annotation processing includes at least one of pinyin annotation processing, rhythm annotation processing and non-language information annotation processing.

[0011] According to a speech synthesis method provided by the present invention, the step of inputting a target text to be synthesized into a text processing module in a speech synthesis model to obtain speech features output by the text processing module includes: Converting the target text into a phoneme sequence; Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder; Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module; The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

[0012] The present invention also provides a speech synthesis system, comprising: A first processing unit, configured to input a target text to be synthesized into a text processing module in a speech synthesis model, and obtain speech features output by the text processing module; A second processing unit is used to input the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; A third processing unit, configured to input the speech feature and the emotion feature into the acoustic model in the speech synthesis model, and obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; A speech synthesis unit, used for inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned speech synthesis methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0016] The speech synthesis method, system, electronic device and storage medium provided by the present invention extract fine-grained speech features through a text processing module and extract fine-grained emotional features through an emotional encoder to avoid information loss, so as to improve the naturalness and accuracy of emotional expression, and utilize a hybrid expert layer deployed in an acoustic model to dynamically select the most suitable expert module according to the emotional features to process speech features and emotional features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, so that the emotional conversion in the generated acoustic features is more natural, and utilize a vocoder to convert the acoustic features into synthesized speech, thereby improving the naturalness and accuracy of emotional expression, solving the problem of unnatural connection between different emotions, realizing emotional and highly anthropomorphic speech synthesis, and improving the emotional effect and speech synthesis quality in super-anthropomorphic multi-emotion speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1This is one of the flow charts of the speech synthesis method provided by the present invention.

[0019] Figure 2 It is one of the structural schematic diagrams of the speech synthesis model provided by the present invention.

[0020] Figure 3 This is the second structural schematic diagram of the speech synthesis model provided by the present invention.

[0021] Figure 4 It is a flowchart of the speech synthesis model training steps provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the distribution of text annotation information provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of a speech synthesis system provided by the present invention.

[0024] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Speech synthesis, also known as text-to-speech (TTS), aims to convert input text into fluent and natural output speech. It is a key technology for realizing intelligent human-computer voice interaction. In the field of artificial intelligence, speech synthesis technology has been developed for several years. From early rule-based methods to modern statistical and deep learning-based models, technological advances have significantly improved the naturalness and comprehensibility of synthesized speech. However, although existing TTS systems can generate fluent and clear speech output, they are still limited in expressing rich emotions. Speech is not only the transmission of information, but also contains rich emotional colors, such as joy, sadness, anger, etc. The expression of these emotions is crucial for interpersonal communication. Therefore, it is particularly important to develop a super-anthropomorphic multi-emotional speech synthesis technology that can accurately capture and reproduce these emotions.

[0027] The goal of super-human multi-emotion speech synthesis technology is to create a voice that can automatically adjust the emotional color of the voice according to emotional needs, including pitch, rhythm, intensity, etc., so as to generate a voice with high naturalness and emotional authenticity. This technology can not only improve the user experience in fields such as smart assistants, virtual assistants, audiobooks, game character dubbing and automatic customer service, but also play an important role in emotional communication and social interaction.

[0028] With the development of deep learning technology, especially the emergence of recurrent neural network (RNN), convolutional neural network (CNN) and transformer models, the performance of TTS systems has been greatly improved. These models can learn complex acoustic and language features, providing a technical foundation for super-human multi-emotional speech synthesis. In addition, with the improvement of computing power and the application of big data, it is possible to train more complex and accurate models, which brings new opportunities for the development of super-human multi-emotional speech synthesis technology.

[0029] However, the development of hyper-anthropomorphic multi-emotional speech synthesis technology faces many challenges. First, the expression of emotion is highly subjective and complex, and the expression of emotion varies greatly in different texts and contexts. Secondly, how to accurately and effectively map emotions to speech parameters is a technical challenge. In addition, how to balance the naturalness of speech and the authenticity of emotions, and how to achieve customization and optimization in different application scenarios are also issues that need to be solved during the research and development process.

[0030] In this regard, mainstream speech synthesis methods usually use a pre-trained emotion encoder to extract emotional features to directly control the emotion of the synthesized speech, without additional consideration of the connection between different emotions. Moreover, this pre-determined feature is fundamentally limited. The hidden vector extracted by the pre-trained emotion encoder loses some information, resulting in unnatural connection between different emotions, limiting the improvement of the emotional effect in the synthesized speech and poor speech synthesis quality.

[0031] In response to the above-mentioned defects, this embodiment provides a speech synthesis method, which improves the connection effect between different emotions of the model by using each expert module in the mixed expert layer to model each emotion respectively, and performs speech synthesis by combining multiple expert modules in the mixed expert layer. It can effectively improve the emotional effect in super-anthropomorphic multi-emotion speech synthesis, and realize emotional and highly anthropomorphic speech synthesis, thereby improving the quality of speech synthesis.

[0032] It should be noted that the executor of the method may be a speech synthesis system; the speech synthesis system may be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer, a server, a network attached storage, a personal computer, etc., and this application does not make any specific limitation.

[0033] Figure 1 It is one of the flowcharts of the speech synthesis method provided by the present invention; Figure 1 As shown, the speech synthesis method includes step 110, step 120, step 130 and step 140.

[0034] Step 110, inputting the target text to be synthesized into a text processing module in the speech synthesis model to obtain speech features output by the text processing module; Step 120, inputting the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; Step 130, inputting the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; Step 140, inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0035] It is understandable that the related art directly controls the emotion of synthesized speech by using a pre-trained emotion encoder to extract emotion features, without additional consideration of the connectivity between different emotions, and this predetermined feature is fundamentally limited. The latent vector extracted by the pre-trained emotion encoder loses some information, resulting in unnatural connectivity between different emotions, limiting the improvement of the emotional effect in synthesized speech, and resulting in poor speech synthesis quality.

[0036] To address this problem, an embodiment of the present application establishes a speech synthesis model that includes a text processing module, an emotion encoder, an acoustic model based on a hybrid expert model, and a vocoder, so as to extract fine-grained speech features through the text processing module and extract fine-grained emotion features through the emotion encoder to avoid information loss, so as to improve the naturalness and accuracy of emotion expression, and utilize the hybrid expert layer deployed in the acoustic model to dynamically select the most suitable expert module to process speech features and emotion features according to the emotion features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, so that the emotion conversion in the generated acoustic features is more natural, and utilize the vocoder to convert the acoustic features into synthesized speech, which not only improves the naturalness and accuracy of emotion expression, but also solves the problem of unnatural connection between different emotions, effectively improves the emotional effect in super-anthropomorphic multi-emotion speech synthesis, and realizes emotional and highly anthropomorphic speech synthesis, thereby improving the quality of speech synthesis.

[0037] Optionally, before executing step 110, a speech synthesis model may be first obtained through training. Specifically, an initialization language model may be first constructed, and the initialization language model here may include an initial text processing module, an initial emotion encoder, an initial acoustic model constructed based on a hybrid expert model, and an initial vocoder. The initial text processing module may be a model prepared for implicit feature extraction of text after parameter initialization, or a pre-trained model with the function of extracting implicit features of text; the initial emotion encoder may be a model prepared for emotional feature extraction after parameter initialization, or a pre-trained model with the function of extracting emotional features; the initial acoustic model may be a hybrid expert model prepared for multi-emotional feature combination output after parameter initialization, or a pre-trained hybrid expert model with the function of outputting multi-emotional feature combination; the initial vocoder may be a model prepared for speech conversion after parameter initialization, or a pre-trained model with the function of speech conversion, and this embodiment does not make specific limitations on this.

[0038] In addition, sample speech and sample text corresponding to the sample speech can also be collected. The sample text here is the content description text corresponding to the sample speech. The sample speech can be obtained by collecting and recording high-quality spoken voice library data in the form of dialogue, which contains rich emotional information.

[0039] Subsequently, the sample speech and the sample text corresponding to the sample speech can be used to iteratively optimize the parameters of the initialized language model, thereby obtaining a speech synthesis model that can synthesize speech that is rich in emotion and highly anthropomorphic.

[0040] The iterative optimization of parameters here can be achieved by optimizing the loss function corresponding to the output layer of the initialized language model, or by jointly optimizing the loss functions corresponding to each module in the initialized language model. This embodiment does not specifically limit this.

[0041] Figure 2 It is one of the structural schematic diagrams of the speech synthesis model provided by the present invention; Figure 3 This is the second structural schematic diagram of the speech synthesis model provided by the present invention.

[0042] like Figure 2 As shown, in practical applications, the text processing module in the speech synthesis model can be used to process the target text to be synthesized, so as to extract fine-grained intermediate implicit features and obtain corresponding speech features.

[0043] The text processing here may be achieved by encoding the target text, or by first encoding the target text and then predicting and adjusting the duration based on the encoding features, which is not specifically limited in this embodiment.

[0044] The speech features here include various speech synthesis attribute information, including but not limited to pronunciation duration, prosodic features, pronunciation intensity, etc., which are not specifically limited in this embodiment.

[0045] In addition, the corresponding emotional information can be obtained from the target text, and the emotional information in the target text can be feature encoded through the emotional encoder in the speech synthesis model to extract fine-grained emotional features from the emotional information.

[0046] Among them, speech features and emotion features are both frame-level features.

[0047] It should be noted that the emotion feature extraction step of step 120 and the speech feature extraction step of step 120 may be executed in parallel or sequentially, and this embodiment does not specifically limit the execution order of the two.

[0048] like Figure 3 As shown, the acoustic model in the speech synthesis model includes multiple operation layers and mixed expert layers, wherein the mixed expert layer can be a layer set below the first operation layer.

[0049] After acquiring the emotional features through the emotional encoder and the speech features through the text processing module, the speech features and the emotional features can be sent to the acoustic model in the speech synthesis model, so that the first operation layer (also called operation layer 1) in the acoustic model inputs the speech features and the emotional features as input information to the mixed expert layer, and the mixed expert layer determines the use probability of each expert module through the gated network, and determines the combination of the input and output of each expert module by using the probability, and performs weighted addition combination of the outputs of multiple expert modules as the output of the mixed expert layer, and uses the output of the mixed expert layer as the input of the next operation layer of the mixed expert layer, so that the next operation layer in the acoustic model and the operation layer after the next operation layer are sequentially operated and processed to form the final acoustic features. The acoustic features here can be mel-spectrograms, etc., which are not specifically limited in this embodiment.

[0050] Here, by taking advantage of the fact that each expert model can accurately model different emotional data distributions and patterns to dynamically extract acoustic features, it is possible to more finely control the acoustic features of the synthesized speech, maintain the consistency of emotions, avoid unnatural phenomena of emotional conversion, and significantly improve the naturalness, adaptability, computational efficiency and synthesis quality of emotional expression of the synthesized speech. This improves the emotional effect, personification ability and generalization ability of the model when synthesizing emotional spoken texts, so that the model can perform highly personified and emotional speech synthesis in various emotional texts, thereby better adapting to the complexity of the task.

[0051] After the acoustic features are acquired through the acoustic model, the vocoder can be used to synthesize the acoustic features into the corresponding speech waveform, thereby obtaining a synthesized speech whose content is consistent with that of the target text and whose emotions are naturally connected.

[0052] The method provided in this embodiment extracts fine-grained speech features through a text processing module and extracts fine-grained emotional features through an emotional encoder to avoid information loss, so as to improve the naturalness and accuracy of emotional expression, and utilizes the hybrid expert layer deployed in the acoustic model to dynamically select the most suitable expert module to process speech features and emotional features according to the emotional features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, so that the emotional conversion in the generated acoustic features is more natural, and utilizes a vocoder to convert the acoustic features into synthesized speech, thereby improving the naturalness and accuracy of emotional expression, solving the problem of unnatural connection between different emotions, realizing emotional and highly anthropomorphic speech synthesis, and improving the emotional effect and speech synthesis quality in super-anthropomorphic multi-emotion speech synthesis.

[0053] Figure 4 Schematic diagram of the flow of speech synthesis model training steps provided by the present invention. Figure 4As shown, the training steps of the speech synthesis model specifically include step 410, step 420, and step 430.

[0054] Step 410, regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech; Step 420, annotating the sample speech and the sample text to obtain sample annotation information; Step 430, iteratively training the initialized language model according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

[0055] Optionally, due to the differences in the sense of hearing and volume of the audio of different emotional data, the connection between emotions may not be natural enough, for example, when neutral and happy are connected, the overall emotion may suddenly become high, thereby reducing the naturalness of the synthesized speech. Therefore, it is necessary to perform regularization processing of different emotional speech segments in the sample speech with the same standard to reduce the differences in the sense of hearing and volume of different emotional speech segments and obtain regularized speech.

[0056] The regularization processing here includes but is not limited to energy equalization processing, adding smooth transition signals between different emotional speech segments, spectrum shaping processing, etc., which are not specifically limited in this embodiment.

[0057] Exemplarily, in some embodiments, the regularizing the emotional speech segment in the sample speech to obtain the regularized speech corresponding to the sample speech includes: According to a preset regularization processing method, regularization processing is performed on the emotional speech segment in the sample speech to obtain the regularized speech; Wherein, the preset regularization processing method includes energy balancing processing.

[0058] Optionally, during the regularization process, energy information of emotional speech segments in the sample speech can be obtained, and the energy information of the emotional speech segments in the sample speech can be regularized in accordance with energy equalization processing to reduce the energy difference under different emotional states, thereby reducing the differences in listening experience, volume, etc. of different emotional speech segments, and obtaining regularized speech with more uniform energy distribution and more natural connection between emotions, thereby improving the listening experience, naturalness and coherence of the synthesized speech.

[0059] In addition, the sample speech and the sample text may be annotated to obtain sample annotation information containing rich annotation information, thereby assisting in the training of the speech synthesis model to effectively improve the accuracy and efficiency of the speech model training.

[0060] Exemplarily, the marking process in step 420 specifically includes: Performing language feature annotation processing on the sample text to obtain a language feature label; Performing sentiment category labeling processing on the sample text to obtain a sentiment label; Performing acoustic feature labeling processing on the sample speech to obtain an acoustic feature label; Performing pronunciation duration labeling processing on the sample speech to obtain a pronunciation duration label; Determining the sample annotation information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label; The language feature annotation processing includes at least one of pinyin annotation processing, rhythm annotation processing and non-language information annotation processing.

[0061] Optionally, when the sample text is annotated, the sample text may be annotated with emotions and speech features, thereby obtaining text annotation information including emotion tags and language feature tags; wherein the speech feature annotation includes at least one of pinyin annotation, rhythm annotation and non-language information annotation. wherein the non-language information annotation refers to the annotation of information other than language information, such as the annotation of information such as laughter and crying.

[0062] Figure 5 is a schematic diagram of the distribution of text annotation information provided by the present invention; illustratively, Figure 5 As shown, for sample texts containing content description information and emotional information, when annotating at the pinyin level, the content description information can be annotated as the corresponding pinyin form, and the non-linguistic information such as emotional information can be annotated as the corresponding string form, such as using "XS" to mark laughter and "KS" to mark crying. When annotating at the rhythm level, different marking forms can be used for different forms of rhythm, such as using "L0" to mark rhythmic characters, that is, single characters in Chinese, such as "you, me, him,..."; using "L1" to mark rhythmic words, that is, a combination of several syllables closely connected in spoken language. Usually, it is a minimum phrase consisting of a word or a content word + affix. For example, "sleep, sleep, dead skin, delicious words,..."; using "L2" to mark secondary rhythmic phrases, that is, there is an obvious pause, but the pause is less than the first preset duration (such as 90ms), and the pitch does not decline or slightly declines, and the end of the rhyme cannot be the end of the sentence. Using "L3" first-level prosodic phrases, that is, there is a long pause, the pause exceeds the first preset duration (such as 90ms), or the fricative stop exceeds the second preset duration (such as 120ms), and the pitch is completely down-tilted, the rhyme end can be the end of the sentence. Using "L4" to mark the intonation phrase, that is, the word used to mark the end of the sentence. When marking at the emotion level, different marking forms can be used for clauses with different emotions.

[0063] When the sample speech is annotated, acoustic features may be extracted from the sample speech, such as mel-spectrogram extraction, to obtain acoustic feature labels. The sample speech may also be annotated with the pronunciation duration of the phonemes to obtain pronunciation duration labels.

[0064] After obtaining language feature labels, emotion labels, acoustic feature labels and pronunciation duration labels through annotation processing, the language feature labels, emotion labels, acoustic feature labels and pronunciation duration labels can be integrated to obtain complete sample annotation information for assisting the training of the speech synthesis model.

[0065] Here, by performing multi-dimensional annotation processing on sample speech and sample text, rich and complete sample annotation information can be obtained, which can be used to assist in the training of the speech synthesis model and effectively improve the accuracy and efficiency of the language model training.

[0066] After obtaining the regularized speech and sample annotation information, a training data set can be constructed by combining the sample text, regularized speech, and sample annotation information to iteratively optimize the parameters of the initialized language model based on the training data set, thereby obtaining a speech synthesis model that can perform emotional and highly anthropomorphic speech synthesis.

[0067] The method provided in this embodiment reduces the differences in listening experience, volume, etc. of different emotional speech segments through regularization processing, and obtains rich and complete sample annotation information through multi-dimensional annotation processing, and then constructs a training data set by combining sample text, regularized speech and sample annotation information, and iteratively optimizes the initialized language model, and finally realizes a speech synthesis model that can generate emotional and highly anthropomorphic speech, which significantly improves the naturalness, coherence and training efficiency of speech synthesis.

[0068] In some embodiments, step 430 specifically includes: Inputting the sample text into an initial text processing module in the initialization language model to obtain sample speech features output by the initial text processing module; Inputting the regularized speech into the initial emotion encoder in the initialization language model to obtain the sample emotion features output by the initial emotion encoder; Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initial language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules; Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information; According to the target loss value, the initialized language model is iteratively trained to obtain the speech synthesis model.

[0069] Optionally, during the training of the speech synthesis model, the sample text may be first converted into a sample phoneme sequence, and the sample phoneme sequence may be input into the initial text encoder in the initial text processing module, so that the sample phoneme sequence is encoded by the initial text encoder to obtain hidden features, that is, sample language features. After obtaining the sample language features output by the initial text encoder, the sample language features may be input into the initial duration prediction module in the initial text processing module, so that the initial duration prediction module uses the sample language features to predict the utterance duration of each phoneme, and obtain corresponding sample pronunciation duration information. After obtaining the sample pronunciation duration information output by the initial duration prediction module, the sample language features and the sample pronunciation duration information may be jointly input into the initial duration adjustment module in the initial text processing module, so that the initial duration adjustment module uses the sample pronunciation duration information to adjust the sample language features, and obtain corresponding sample speech features.

[0070] In addition, the regular speech can be input into the initial emotion encoder to perform feature encoding on the emotion information in the regular speech, so as to extract fine-grained sample emotion features from the emotion information.

[0071] After obtaining the sample emotion feature through the initial emotion encoder and the sample speech feature through the initial text processing module, the sample speech feature and the sample emotion feature can be sent to the initial acoustic model, and the sample speech feature and the sample emotion feature are input as input information to the mixed expert layer by the first operation layer in the initial acoustic model. The initial mixed expert layer determines the use probability of each initial expert module through the gated network, and determines the combination of the input and output of each expert module by using the probability, and obtains the routing decision value of the corresponding input and the sample acoustic sub-feature of the corresponding output of each initial expert module, and the outputs of multiple initial expert modules are weighted and added together as the output of the initial mixed expert layer, and the output of the initial mixed expert layer is used as the input of the next operation layer of the initial mixed expert layer, so as to form the final sample acoustic feature after sequential operation processing by the next operation layer and the operation layer after the next operation layer in the initial acoustic model. The sample acoustic feature here can be a mel-spectrogram, etc., which is not specifically limited in this embodiment.

[0072] After obtaining the sample speech features, sample emotion features, routing decision values, sample acoustic sub-features, and sample acoustic features, the sample speech features, sample emotion features, routing decision values, sample acoustic sub-features, sample acoustic features, and sample annotation information may be combined to perform a loss function calculation to obtain a multi-dimensional loss value, and the target loss value is determined by combining the multi-dimensional loss value, so as to iteratively train the initialized language model according to the target loss value to obtain a speech synthesis model for speech synthesis that is rich in emotion and has a high degree of anthropomorphism.

[0073] The multi-dimensional loss value here includes the loss values ​​corresponding to some initial modules or the loss values ​​corresponding to all initial modules in the initialized language model, which is not specifically limited in this embodiment.

[0074] Exemplarily, the step of determining the target loss value specifically includes: Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information; Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information; Determine a third loss value according to the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information; Determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label; The target loss value is determined by combining the first loss value, the second loss value, the third loss value and the fourth loss value.

[0075] Optionally, the sample acoustic features output by the initial acoustic model may be compared with the acoustic feature labels in the sample annotation information to determine the acoustic feature loss value corresponding to the initial acoustic model as the first loss value.

[0076] In addition, the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information may be compared to determine the loss value corresponding to the initial duration prediction model as the second loss value.

[0077] In addition, the language features corresponding to the sample speech features output by the initial text encoder can be compared with the language feature labels in the sample annotation information, and the emotion categories corresponding to the sample emotion features output by the initial emotion encoder can be compared with the emotion labels in the sample annotation information to determine the loss value corresponding to the encoder as the third loss value.

[0078] In addition, the correctness of the routing decision can be evaluated based on the routing decision values ​​corresponding to the inputs of each initial expert module in the initial acoustic model and the corresponding output sample acoustic sub-features and acoustic feature labels to determine the loss value corresponding to the initial acoustic model, thereby obtaining the fourth loss value.

[0079] Subsequently, the first loss value, the second loss value, the third loss value and the fourth loss value are fused to obtain a target loss value. The fusion here may be direct addition or weighted addition, etc., which is not specifically limited in this embodiment.

[0080] For example, the target loss value may be calculated as: ; in, is the first loss value; is the second loss value; is the third loss value; is the fourth loss value.

[0081] The method provided in this embodiment performs iterative model training by combining the loss values ​​corresponding to the multi-dimensional modules, so as to quickly obtain an emotional and highly anthropomorphic speech synthesis model, thereby achieving multiple improvements in the accuracy, naturalness and efficiency of speech synthesis.

[0082] In some embodiments, step 110 specifically includes: Converting the target text into a phoneme sequence; Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder; Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module; The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

[0083] like Figure 3 As shown, the text processing module includes a text encoder, a duration prediction module and a duration adjustment module (LR for short). The text encoder is used to encode the phoneme sequence corresponding to the text to obtain hidden features, that is, language features; the duration prediction module is used to predict the utterance duration; and the duration adjustment module is used to adjust the utterance duration.

[0084] Optionally, when performing speech feature extraction of text, the target text may be first converted into a phoneme sequence, and the phoneme sequence is input into a text encoder in the text processing module, so that the text encoder encodes the phoneme sequence to obtain hidden features, that is, language features.

[0085] After obtaining the language features output by the text encoder, the language features can be input into the duration prediction module in the text processing module, so that the duration prediction module can use the language features to predict the pronunciation duration of each phoneme to obtain corresponding pronunciation duration information.

[0086] After obtaining the pronunciation duration information output by the duration prediction module, the language features and the pronunciation duration information can be jointly input into the duration adjustment module in the text processing module, so that the duration adjustment module can use the pronunciation duration information to adjust the language features to obtain corresponding speech features, so that the pronunciation duration information of the final speech features matches the pronunciation duration information predicted by the duration prediction module.

[0087] The method provided in this embodiment achieves precise control of the pronunciation duration and emotion of the target text by integrating a text encoder, a duration prediction and an adjustment module, thereby generating natural and fluent diverse emotional speech.

[0088] The speech synthesis system provided by the present invention is described below. The speech synthesis system described below and the speech synthesis method described above can be referred to each other.

[0089] Figure 6 is a structural schematic diagram of a speech synthesis system provided by the present invention; Figure 6 As shown, the system includes: The first processing unit 610 is used to input the target text to be synthesized into the text processing module in the speech synthesis model to obtain the speech features output by the text processing module; The second processing unit 620 is used to input the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; The third processing unit 630 is used to input the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; The speech synthesis unit 640 is used to input the acoustic features into the vocoder in the speech synthesis model to obtain the synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0090] The system provided in this embodiment extracts fine-grained speech features through a text processing module and extracts fine-grained emotional features through an emotional encoder to avoid information loss, so as to improve the naturalness and accuracy of emotional expression, and utilizes the hybrid expert layer deployed in the acoustic model to dynamically select the most suitable expert module to process speech features and emotional features according to the emotional features, so as to smoothly transition the connection between different emotions through a dynamic selection mechanism, so that the emotional conversion in the generated acoustic features is more natural, and utilizes a vocoder to convert the acoustic features into synthesized speech, thereby improving the naturalness and accuracy of emotional expression, solving the problem of unnatural connection between different emotions, realizing emotional and highly anthropomorphic speech synthesis, and improving the emotional effect and speech synthesis quality in super-anthropomorphic multi-emotion speech synthesis.

[0091] In some embodiments, the speech synthesis system further includes a model training unit, specifically configured to: Regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech; Annotating the sample speech and the sample text to obtain sample annotation information; The initialization language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

[0092] In some embodiments, the model training unit is further used to: Inputting the sample text into an initial text processing module in the initialization language model to obtain sample speech features output by the initial text processing module; Inputting the regularized speech into the initial emotion encoder in the initialization language model to obtain the sample emotion features output by the initial emotion encoder; Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initial language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules; Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information; According to the target loss value, the initialized language model is iteratively trained to obtain the speech synthesis model.

[0093] In some embodiments, the model training unit is further configured to: Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information; Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information; Determine a third loss value according to the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information; Determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label; The target loss value is determined by combining the first loss value, the second loss value, the third loss value and the fourth loss value.

[0094] In some embodiments, the model training unit is further configured to: According to a preset regularization processing method, regularization processing is performed on the emotional speech segment in the sample speech to obtain the regularized speech; Wherein, the preset regularization processing method includes energy balancing processing.

[0095] In some embodiments, the model training unit is further configured to: Performing language feature annotation processing on the sample text to obtain a language feature label; Performing sentiment category labeling processing on the sample text to obtain a sentiment label; Performing acoustic feature labeling processing on the sample speech to obtain an acoustic feature label; Performing pronunciation duration labeling processing on the sample speech to obtain a pronunciation duration label; Determining the sample annotation information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label; The language feature annotation processing includes at least one of pinyin annotation processing, rhythm annotation processing and non-language information annotation processing.

[0096] In some embodiments, the first processing unit is specifically configured to: Converting the target text into a phoneme sequence; Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder; Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module; The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

[0097] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0098] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the speech synthesis method, which includes: inputting the target text to be synthesized into the text processing module in the speech synthesis model to obtain the speech features output by the text processing module; inputting the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; inputting the speech features and the emotional features into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; inputting the acoustic features into the vocoder in the speech synthesis model to obtain the synthesized speech corresponding to the target text output by the vocoder; wherein the speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0099] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above-mentioned methods, which includes: inputting the target text to be synthesized into a text processing module in a speech synthesis model to obtain the speech features output by the text processing module; inputting the emotional information in the target text into an emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; inputting the speech features and the emotional features into an acoustic model in the speech synthesis model to obtain the acoustic features output by a combination of multiple expert modules in the acoustic model; inputting the acoustic features into a vocoder in the speech synthesis model to obtain the synthesized speech corresponding to the target text output by the vocoder; wherein the speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0101] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech synthesis method provided by the above-mentioned methods, the method comprising: inputting the target text to be synthesized into a text processing module in a speech synthesis model to obtain speech features output by the text processing module; inputting the emotional information in the target text into an emotional encoder in the speech synthesis model to obtain emotional features output by the emotional encoder; inputting the speech features and the emotional features into an acoustic model in the speech synthesis model to obtain acoustic features output by a combination of multiple expert modules in the acoustic model; inputting the acoustic features into a vocoder in the speech synthesis model to obtain synthesized speech corresponding to the target text output by the vocoder; wherein the speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Inputting the target text to be synthesized into a text processing module in the speech synthesis model to obtain speech features output by the text processing module; Inputting the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; Inputting the speech feature and the emotion feature into the acoustic model in the speech synthesis model to obtain the acoustic features output by the combination of multiple expert modules in the acoustic model; Inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

2. The speech synthesis method according to claim 1, characterized in that: The speech synthesis model is trained based on the following steps: Regularizing the emotional speech segment in the sample speech to obtain regularized speech corresponding to the sample speech; Annotating the sample speech and the sample text to obtain sample annotation information; The initialization language model is iteratively trained according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model.

3. The speech synthesis method according to claim 2, characterized in that: The iterative training of the initialized language model according to the sample text, the regularized speech, and the sample annotation information to obtain the speech synthesis model includes: Inputting the sample text into an initial text processing module in the initialization language model to obtain sample speech features output by the initial text processing module; Inputting the regularized speech into the initial emotion encoder in the initialization language model to obtain the sample emotion features output by the initial emotion encoder; Inputting the sample speech features and the sample emotion features into the initial acoustic model in the initial language model, obtaining the routing decision values ​​corresponding to the inputs and the sample acoustic sub-features corresponding to the outputs of the initial expert modules in the initial acoustic model, and the sample acoustic features output by the combination of the multiple initial expert modules; Determining a target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information; According to the target loss value, the initialized language model is iteratively trained to obtain the speech synthesis model.

4. The speech synthesis method according to claim 3, characterized in that: The determining of the target loss value according to the sample speech feature, the sample emotion feature, the routing decision value, the sample acoustic sub-feature, the sample acoustic feature, and the sample annotation information includes: Determining a first loss value according to the acoustic feature of the sample and the acoustic feature label in the sample annotation information; Determining a second loss value according to the pronunciation duration information corresponding to the sample speech feature and the pronunciation duration label in the sample annotation information; Determine a third loss value according to the language feature corresponding to the sample speech feature and the language feature label in the sample annotation information, and the emotion category corresponding to the sample emotion feature and the emotion label in the sample annotation information; Determining a fourth loss value according to the routing decision value, the sample acoustic sub-feature, and the acoustic feature label; The target loss value is determined by combining the first loss value, the second loss value, the third loss value and the fourth loss value.

5. The speech synthesis method according to claim 2, characterized in that: The step of regularizing the emotional speech segment in the sample speech to obtain a regularized speech corresponding to the sample speech includes: According to a preset regularization processing method, regularization processing is performed on the emotional speech segment in the sample speech to obtain the regularized speech; Wherein, the preset regularization processing method includes energy balancing processing.

6. The speech synthesis method according to claim 2, characterized in that: The labeling process of the sample speech and the sample text to obtain sample labeling information includes: Performing language feature annotation processing on the sample text to obtain a language feature label; Performing sentiment category labeling processing on the sample text to obtain a sentiment label; Performing acoustic feature labeling processing on the sample speech to obtain an acoustic feature label; Performing pronunciation duration labeling processing on the sample speech to obtain a pronunciation duration label; Determining the sample annotation information according to the language feature label, the emotion label, the acoustic feature label, and the pronunciation duration label; The language feature annotation processing includes at least one of pinyin annotation processing, rhythm annotation processing and non-language information annotation processing.

7. The speech synthesis method according to any one of claims 1 to 6, characterized in that: The step of inputting the target text to be synthesized into a text processing module in the speech synthesis model to obtain speech features output by the text processing module includes: Converting the target text into a phoneme sequence; Inputting the phoneme sequence into a text encoder in the text processing module to obtain language features output by the text encoder; Inputting the language features into a duration prediction module in the text processing module to obtain pronunciation duration information output by the duration prediction module; The language feature and the pronunciation duration information are input into a duration adjustment module in the text processing module to obtain the speech feature output by the duration adjustment module.

8. A speech synthesis system, characterized in that: include: A first processing unit, configured to input a target text to be synthesized into a text processing module in a speech synthesis model, and obtain speech features output by the text processing module; A second processing unit is used to input the emotional information in the target text into the emotional encoder in the speech synthesis model to obtain the emotional features output by the emotional encoder; A third processing unit, used for inputting the speech feature and the emotion feature into the acoustic model in the speech synthesis model, and obtaining the acoustic features output by the combination of multiple expert modules in the acoustic model; A speech synthesis unit, used for inputting the acoustic features into a vocoder in the speech synthesis model to obtain a synthesized speech corresponding to the target text output by the vocoder; The speech synthesis model is trained based on sample speech and sample text corresponding to the sample speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images