Speech synthesis method, apparatus, and electronic device

Abstract

A speech synthesis method, comprising: acquiring a target text to be synthesized into a speech and a target emotion category (S101); on the basis of a source speaker category and the target emotion category, processing the target text into source speech data having a source speaker timbre and the target emotion category (S102); determining from the source speech data phoneme feature data corresponding to the target text (S103); and, on the basis of the source speech data and a target speaker category, processing the phoneme feature data into a speech signal having a target speaker timbre and the target emotion category (S104). The method allows synthesized speeches to keep target emotion categories and target speaker timbres.

Description

Speech synthesis method, device and electronic equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application number 202410160305.2, filed on February 4, 2024, entitled “Speech Synthesis Method, Device and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a speech synthesis method, device, electronic device, and computer-readable storage medium. Background Art

[0004] With the continuous development of artificial intelligence (AI), speech synthesis technology has made significant progress and is now being applied in many everyday scenarios, such as smart speakers, audiobooks, and virtual live broadcasts. In human-computer interaction scenarios, speech synthesis technology enables machines to "speak" and interact with users. The quality of the synthesized speech directly affects the user experience and immersion in human-computer interaction.

[0005] In related technologies, in order to generate speech with target timbre and target emotion, the timbre information and emotion information of a source speech data with target emotion and source timbre are usually decoupled, and speech synthesis is performed in combination with the target timbre.

[0006] However, when decoupling timbre information and emotional information, incomplete decoupling often leads to the introduction of the source timbre into the timbre corresponding to the final generated speech, or the change of timbre affects the expression of emotion, resulting in more serious timbre leakage, emotional transfer failure, and emotional flatness in the final generated speech. Summary of the Invention

[0007] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a speech synthesis method, which includes: obtaining a target text to be synthesized into speech and a target emotion category; processing the target text into source speech data having the timbre of the source speaker and the target emotion category according to the source speaker category and the target emotion category, wherein the source speaker category has at least audio data under the target emotion category; determining phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; processing the phoneme feature data into a speech signal having the timbre of the target speaker and the target emotion category according to the source speech data and the target speaker category.

[0008] According to one aspect of the present disclosure, an embodiment of the present disclosure provides a speech synthesis device, which includes: an acquisition unit for acquiring a target text to be synthesized into speech and a target emotion category; a first processing unit for processing the target text into source speech data having a source speaker timbre and a target emotion category based on a source speaker category and a target emotion category, wherein the source speaker category has at least audio data under the target emotion category; a determination unit for determining phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; and a second processing unit for processing the phoneme feature data into a speech signal having a target speaker timbre and a target emotion category based on the source speech data and the target speaker category.

[0009] According to one aspect of the present disclosure, the present disclosure also provides an electronic device, including: a processor; and a memory for storing a data processing program. After the electronic device is powered on and the program is run by the processor, the speech synthesis method as described above is executed.

[0010] According to one aspect of the present disclosure, an embodiment of the present disclosure further provides a computer-readable storage medium storing a data processing program, which is executed by a processor to perform the speech synthesis method as described above.

[0011] Compared with the related art, the present disclosure has the following advantages:

[0012] The speech synthesis method provided by the present disclosure first obtains a target text to be synthesized into speech and a target emotion category; secondly, based on the source speaker category and the target emotion category, processes the target text into frame-level source speech data having the timbre of the source speaker and the target emotion category. Since the source speaker category has at least audio data under the target emotion category, in this way, real and natural source speech data of the target text that presents the target emotion category through the timbre of the source speaker can be obtained; then, the phoneme feature data corresponding to the target text is determined from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; finally, based on the source speech data and the target speaker category, the phoneme feature data is processed into a speech signal having the timbre of the target speaker and the target emotion category. Since the source speech data is real and natural data with the target emotion category, the target speaker category can reflect the timbre of the target speaker, and the phoneme feature data can reflect the text content of the target text. In this way, by referring to the performance of the source speaker timbre in the source speech data when presenting the target text of the target emotion category, the performance of the target speaker timbre when presenting the target text of the target emotion category can be obtained. Moreover, since the source speech data is relatively real and natural data, the obtained speech signal can retain the target emotion category in a real and natural way while retaining the timbre of the target speaker.

[0013] As can be seen, the speech synthesis method provided by the present disclosure, when it is desired to generate a speech signal with the timbre of a target speaker and a target emotional category, first generates source speech data with a source speaker category having audio data in various emotional categories, and then generates a relatively realistic and natural speech signal with the source speaker's timbre presenting the target emotional category when the target text is presented. The speech synthesis method provided by the present disclosure can avoid serious timbre leakage, emotional transfer failure, and emotional flatness in the synthesized speech, so that the synthesized speech retains the target emotional category and timbre of the target speaker to the greatest extent possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of the structure of one of the current cross-speaker emotion transfer models;

[0015] FIG2 is a flow chart of a speech synthesis method provided by an embodiment of the present disclosure;

[0016] FIG3 is a flowchart of extracting phoneme feature data through a model in one of the speech synthesis methods provided by an embodiment of the present disclosure;

[0017] FIG4 is a schematic diagram of determining second prosody information by a prosody predictor in one of the speech synthesis methods provided by an embodiment of the present disclosure;

[0018] FIG5 is a schematic diagram of a speech synthesis system used in one of the speech synthesis methods provided in an embodiment of the present disclosure;

[0019] FIG6 is a schematic diagram of a data flow direction in one of the speech synthesis systems provided in an embodiment of the present disclosure;

[0020] FIG7 is a structural block diagram of an example of a speech synthesis device provided in an embodiment of the present disclosure;

[0021] FIG8 is a structural block diagram of an example of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following description sets forth many specific details to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.

[0023] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present disclosure are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] Before describing the embodiments of the present disclosure in detail, the related art will be further described first.

[0025] In recent years, with the development of technologies like deep learning, speech synthesis has garnered widespread attention. Speech synthesis, which uses mechanical and electronic methods to generate artificial speech, can convert any text message into standard, fluent speech in real time. In human-computer interaction scenarios, users often desire more realistic machine-synthesized speech to enhance their experience.

[0026] Among them, the cross-speaker emotion transfer task refers to making a speaker with only neutral emotions output "inflected" speech, and having the ability to synthesize speech with various emotions. At this stage, cross-speaker emotion transfer models are generally built based on speech synthesis models. In terms of data, it is necessary to prepare in advance speech data with emotion and source timbre, as well as target timbre data, and corresponding text annotations. In the inference stage, text annotation helps the model better understand and generate speech with rich emotion and expressiveness. By decoupling the emotion and timbre in the speech data with emotion and source timbre, the decoupled emotion and target timbre data are combined to obtain the final speech.

[0027] Figure 1 shows the current structure of the cross-speaker emotion transfer model. The cross-speaker emotion transfer model takes three inputs: a phoneme sequence, a speaker ID, and the source speech. The timbre of the speaker ID is the target timbre of the generated speech, and the emotion of the source speech is the target emotion of the generated speech. Specifically, the emotion encoder extracts an emotion vector from the source speech and decouples the source timbre and emotion through domain adversarial training. The goal is to enable the emotion encoder to extract an emotion vector that is independent of speaker information and contains only prosodic information. The phoneme sequence is embedded in the text embedding layer to obtain a phoneme vector, which is then passed through the encoder. The speaker ID is then passed through the timbre embedding layer to obtain a timbre vector. The emotion vector, phoneme vector, and timbre vector are combined through the prosody prediction module, duration prediction module, and decoder to generate acoustic features. These acoustic features in the frequency domain are then converted into sound signals in the time domain to produce speech.

[0028] However, the above solution has at least the following two problems:

[0029] First, the above solution requires a large amount of annotated data. Currently, cross-speaker emotion transfer models are generally built based on speech synthesis models. Specifically, a single model aims to accomplish two tasks: speech synthesis and emotion transfer. The goal of the speech synthesis task is to complete the conversion from text to audio modality, while the emotion transfer task is to enable a speaker with only neutral emotions to produce cadenced speech, capable of synthesizing speech across seven categories of emotions. Due to the complexity of the task, speech synthesis typically requires a large amount of structured annotated data, including speech data (including various emotions), corresponding text, and speaker information. Finer-grained annotation includes prosody information and alignment information. Text can be divided into phonemes, and alignment information refers to the temporal correspondence between phonemes and audio. The entire annotation process relies on extensive manual annotation, which is labor-intensive and time-consuming, with limited accuracy. This results in high overall costs and limited data availability.

[0030] Second, there's a discrepancy between the training and inference processes. The goal of model training is to enable speakers with only neutral emotions to synthesize speech with various emotional characteristics. During training, the task is speech reconstruction. This means the source speech input to the emotion encoder and the resulting speech are identical. In this case, the emotion vector and timbre vector originate from the same speaker. However, due to limited data size, even with the use of methods like domain adversarial training, the decoupling of the emotion and timbre vectors is incomplete. During model inference, the source speech input to the emotion encoder is emotionally charged speech from a non-target speaker to be transferred, while the timbre vector contains information about the target speaker. In this case, the emotion vector and timbre vector originate from different speakers. In other words, during training, the emotion vector and timbre vector originate from the same speaker, while during inference, the emotion vector originates from the source speaker and the timbre vector originates from the target speaker. This leads to a discrepancy between the training and inference processes. During inference, the model encounters combinations of "emotion vector, timbre vector" that it has never encountered during training. This "hard switching" often produces poor results. The generated speech will have serious timbre leakage, and may also fail to transfer emotions and be emotionally flat.

[0031] Based on the above reasons, in order to enable the synthesized speech to retain the target emotion category and the target speaker's timbre to the greatest extent, the first embodiment of the present disclosure provides a speech synthesis method, which is applied to an electronic device, which can be a desktop computer, a laptop computer, a mobile phone, a tablet computer, a server, a terminal device, etc., or other electronic devices capable of speech synthesis, which is not specifically limited in the embodiment of the present disclosure.

[0032] The speech synthesis method provided by the embodiment of the present disclosure is described below with reference to FIG. 2 to FIG. 4 .

[0033] As shown in FIG2 , the speech synthesis method provided by the present disclosure includes the following steps S101 to S104 .

[0034] Step S101: Obtain target text to be synthesized into speech and target emotion category.

[0035] In the embodiment of the present disclosure, the target text can be a sentence or a paragraph, and the content of the target text can be Chinese, English or any other language. The target emotion category is the emotion category required by the synthesized speech, such as happy, angry, fear, etc.

[0036] In the present disclosure, emotion categories can be divided into seven categories according to Paul Ekman's basic emotion theory, namely: anger, surprise, disgust, happiness, fear, sadness, and neutral. Neutral means no emotion at all. Emotion categories can also be divided into four categories: joy, anger, sorrow, and happiness. Emotion categories can also be weighted according to other standards, which are not specifically limited in the present disclosure.

[0037] The target emotion intensity is the emotion intensity corresponding to the speech to be generated from the target text. The emotion intensity is the strength of each emotion category. Different emotion intensities reflect different degrees of the corresponding emotion category. For example, the emotion category "anger" can correspond to emotion intensities such as "very angry" and "slightly angry." The emotion categories and emotion intensities are specifically categorized based on actual circumstances in specific applications and are not specifically limited in this disclosure.

[0038] It should be noted that the emotion category corresponding to each word (or each phoneme) in a text will usually remain consistent. That is to say, for the target text in the present disclosure, the emotion category corresponding to each word (or each phoneme) is the same. For example, if the target text is "The weather is really good today" and the target emotion category is happy, then the emotion categories of the six words "today", "day", "weather", "air", "real", and "good" are all happy.

[0039] Step S102: According to the source speaker category and the target emotion category, the target text is processed into source speech data having the source speaker's timbre and the target emotion category.

[0040] The source speaker category currently has at least audio data under the target emotion category.

[0041] In a specific embodiment, a source speaker category may have at least one audio data item for each emotion category. That is, the source speaker category may correspond to audio data for anger, surprise, disgust, happiness, fear, sadness, and neutrality. This allows for the acquisition of realistic and natural speech using the source speaker's timbre when expressing any emotion category. Furthermore, the textual content corresponding to the audio data for each emotion category may differ.

[0042] It should be noted that the audio data of the source speaker category in each emotion category can be greater than or equal to the preset time length. The more audio data the source speaker category has in each emotion category, the more stable the performance of the source speaker category in each emotion category.

[0043] Alternatively, the source speaker category may correspond only to at least one audio data item of the target emotion category. This allows for authentic and natural speech when the source speaker's timbre expresses the target emotion category. The audio data of the source speaker category in the target emotion category may be greater than or equal to a preset duration. The more audio data a source speaker category has in the target emotion category, the more stable its performance in the target emotion category.

[0044] In the present disclosure, there can be one or more source speaker categories. When there are multiple source speaker categories, one source speaker category can be selected from the multiple categories to process the target text into source speech data that does not have the selected source speaker timbre and target emotion category through the selected source speaker category and target emotion category.

[0045] As shown in Table 1, it is a schematic table of an example of audio data of each emotion category corresponding to the source speaker category in the speech synthesis method provided by the embodiment of the present disclosure.

[0046] Table 1.

[0047] In this way, based on the source speaker category and the required target emotion category corresponding to the speech data of each emotion category, the target text can be processed into real and natural source speech data with the source speaker's timbre and the target emotion category.

[0048] It should be noted that the source speech data obtained in step S102 can be an acoustic spectrum or a speech signal in the time domain. The acoustic spectrum refers to a Mel-spectrogram, which can reflect the acoustic characteristics of the speech signal to be generated. A speech signal refers to the physical sound waves produced by the human voice. It is a one-dimensional time series data that represents the amplitude of the sound over time. The speech signal contains rich information, such as the speaker's timbre, intonation, prosody, and the content of the speech. The acoustic spectrum and speech signal can be converted into each other.

[0049] As you can understand, the Mel spectrum (Mel spectrogram) shows the distribution of speech signals at different frequencies. The Mel scale is based on the perceptual judgment of pitch by listeners who are equidistant from each other. Since the human ear is more sensitive to the distinction between low-frequency signals and less sensitive to the distinction between high-frequency signals, the human ear is more likely to distinguish the frequencies in the low-frequency band from the frequencies in the high-frequency band that are equally spaced on the normal frequency. Therefore, the Mel scale was proposed so that the frequencies in the low-frequency band and the frequencies in the high-frequency band that are equally spaced on the new scale appear the same to the human ear. Therefore, the Mel spectrogram is a spectral image based on the auditory characteristics of the human ear.

[0050] In the present disclosure, the source speech data obtained can be frame-level data, that is, in the present disclosure, the features of each frame can be obtained frame by frame. The features of each frame include both the features corresponding to the timbre of the source speaker and the features corresponding to the target emotion category. In this way, the source speech data at the frame level can be obtained based on the features of each frame.

[0051] The following introduces the concepts related to audio frames.

[0052] The sampling rate refers to the number of samples extracted from a continuous signal and composed into a discrete signal per second, and its unit is Hertz (Hz); in order to make the transition between frames smooth and maintain its continuity, the audio framing generally adopts the overlapping segmentation method to ensure that two adjacent frames overlap each other, and the number of sampling points collected between the starting positions of two adjacent frames is called frame shift; the number of frames per second is sampling rate / frame shift. In the embodiment of the present disclosure, the frame shift and sampling rate can be set (for example, the sampling rate can be set to 22050Hz and the frame shift to 256 sampling points), and the characteristics of each frame are obtained according to the set frame shift and sampling rate, and then the source speech data is generated. Step S103: Determine the phoneme feature data corresponding to the target text from the source speech data.

[0053] Among them, the phoneme feature data is mainly used to represent the text content of the target text, and can also include prosodic information. That is, the phoneme feature data reflects the text content of the target text and can implicitly include the prosodic information corresponding to the source speaker's timbre when expressing the target text under the target emotion category.

[0054] In specific implementation, information irrelevant to the target text can be eliminated from the source speech data, such as information corresponding to the source speaker's timbre. In this way, the influence of information irrelevant to the target text in the source speech data is removed, and only information related to the target text is retained, thereby obtaining the phoneme feature data corresponding to the target text.

[0055] It should be noted that the phoneme feature data obtained in step S103 may be data having the same frame number as the source speech data obtained in step S102, so that the final generated speech signal can be consistent with the frame number of the source speech data.

[0056] Through step S103, information irrelevant to the target text is removed from the generated real and natural source speech data, and phoneme feature data related to the target text and with the same frame number as the source speech data is obtained. This provides a basis for synthesizing a real and natural speech signal based on the real and natural source speech data and the target speaker category.

[0057] Step S104: According to the source speech data and the target speaker category, the phoneme feature data is processed into a frame-level speech signal having the timbre of the target speaker and the target emotion category.

[0058] In the disclosed embodiments, the target speaker category can be a speaker category that does not have audio data for each emotion category, or a speaker category that only has audio data with a neutral emotion. Alternatively, the target speaker category can be a speaker category that does not have audio data for the target emotion category. Thus, given a target text and a target emotion category, directly combining the target speaker category, target text, and target emotion category to generate a speech signal will result in a stilted, unnatural, and unrealistic speech signal due to the lack of reference data.

[0059] Step S104 can essentially be processing the phoneme feature data into a speech signal having the target speaker's timbre and the target emotion category, using the emotion features corresponding to the source speech data and the timbre features corresponding to the target speaker category. Since the source speech data is authentic and natural data with the target emotion category, the target speaker category can reflect the target speaker's timbre, and the phoneme feature data can reflect the text content of the target text. Thus, by combining the source speech data, the target speaker category, and the phoneme feature data, a speech signal of the target text with the target emotion category, represented by the target speaker's timbre, can be obtained.

[0060] The following is a specific example to introduce the speech synthesis method provided by the present disclosure:

[0061] Assuming that it is desired to generate a text of the happy category "Today is such a happy day" presented by Xiao Ming's voice, and when Xiao Ming's voice does not have the corresponding audio of various emotions or does not have the audio of the happy category, the source voice data of the text "Today is such a happy day" presented by Xiao Hong's voice can be generated through Xiao Hong, who currently has audio of various emotion categories (or currently has the happy category). Afterwards, the phoneme feature data related to the text content of the target text is extracted from the source voice data, and the voice signal of the text "Today is such a happy day" presented by Xiao Ming's voice is generated by combining the emotion when Xiao Hong's voice presents the text of the happy category "Today is such a happy day", Xiao Ming's voice and the phoneme feature data.

[0062] The speech synthesis method provided by the present disclosure first obtains a target text to be synthesized into speech and a target emotion category; secondly, based on the source speaker category and the target emotion category, processes the target text into frame-level source speech data having the timbre of the source speaker and the target emotion category. Since the source speaker category has at least audio data under the target emotion category, in this way, real and natural source speech data of the target text that presents the target emotion category through the timbre of the source speaker can be obtained; then, the phoneme feature data corresponding to the target text is determined from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; finally, based on the source speech data and the target speaker category, the phoneme feature data is processed into a speech signal having the timbre of the target speaker and the target emotion category. Since the source speech data is real and natural data with the target emotion category, the target speaker category can reflect the timbre of the target speaker, and the phoneme feature data can reflect the text content of the target text. In this way, by referring to the performance of the source speaker timbre in the source speech data when presenting the target text of the target emotion category, the performance of the target speaker timbre when presenting the target text of the target emotion category can be obtained. Moreover, since the source speech data is relatively real and natural data, the obtained speech signal can retain the target emotion category in a real and natural way while retaining the timbre of the target speaker.

[0063] As can be seen, the speech synthesis method provided by the present disclosure, when it is desired to generate a speech signal with the timbre of a target speaker and a target emotional category, first generates source speech data with a source speaker category having audio data in various emotional categories, and then generates a relatively realistic and natural speech signal with the source speaker's timbre presenting the target emotional category when the target text is presented. The speech synthesis method provided by the present disclosure can avoid serious timbre leakage, emotional transfer failure, and emotional flatness in the synthesized speech, so that the synthesized speech retains the target emotional category and timbre of the target speaker to the greatest extent possible.

[0064] In an optional implementation, step S103 may be implemented by the following steps:

[0065] When the source speech data is an acoustic spectrum, converting the acoustic spectrum into a speech signal in the time domain;

[0066] The information irrelevant to the target text in the speech signal in the time domain is removed to obtain the phoneme feature data corresponding to the target text.

[0067] In specific implementation, the source speech data can be an acoustic spectrum or a speech signal in the time domain. When the source speech data is an acoustic spectrum, feature conversion can be performed first to convert the acoustic spectrum into a speech signal in the time domain, and then the phoneme feature data can be determined from it. When the source speech data is a speech signal in the time domain, there is no need to perform feature conversion and the phoneme feature data can be determined directly from it.

[0068] In an embodiment of the present disclosure, phoneme feature data can be extracted from source speech data using a pre-trained model. Specifically, when the source speech data is an acoustic spectrum, the source speech data can be input into a pre-trained feature conversion model so that the feature conversion model can output phoneme feature data. The feature conversion model can be used to directly extract text-related phoneme feature data from the source speech data in the form of an acoustic spectrum. When the source speech data is a speech signal in the time domain, the source speech data can be input into a pre-trained speech recognition model so that the speech recognition model can output phoneme feature data. The speech recognition model can be used to extract text-related phoneme feature data from the source speech data in the form of a speech signal.

[0069] As shown in Figure 3, it is a flowchart of extracting phoneme feature data through a model in the speech synthesis method provided by an embodiment of the present disclosure. After obtaining the source speech data, first, it is determined whether the source speech data is an acoustic spectrum. If the source speech data is a speech signal in the time domain, the source speech data is input into the speech recognition model to output the phoneme feature data through the speech recognition model; if the source speech data is not a speech signal in the time domain, the source speech data is input into the feature conversion model to output the phoneme feature data based on the feature conversion model.

[0070] The task of a speech recognition model is to input a variety of speech sounds and output the corresponding text. These various speech sounds refer to speech sounds from different speakers (timbre information) and in different acoustic environments (noisy or indoor). The features obtained by the speech recognition model are only related to the text content and are unrelated to other information (such as timbre). The task of a feature conversion model is to input an acoustic spectrum, convert it into speech, and output the corresponding text. The features obtained by the feature conversion model are also only related to the text content and are unrelated to other information (such as timbre).

[0071] In a specific embodiment, when the source speech data is an acoustic spectrum, the present disclosure can also use the HIFI-GAN model (a generative adversarial network model for efficient and high-fidelity speech synthesis) to convert the acoustic spectrum into a speech signal in the time domain. The input of HIFI-GAN is a Mel spectrum, which is upsampled through multiple convolutional layers until the output is a time domain waveform.

[0072] Through this technical means, irrelevant information to the target text can be removed from the source speech data by a pre-trained model, and phoneme feature data related to the target text can be effectively obtained.

[0073] In an optional implementation manner, step S104 can be implemented through the following steps:

[0074] Extract the first prosody information corresponding to the source speech data, where the first prosody information is used to characterize the target emotion category;

[0075] According to the first prosody information and the target speaker category, determine the second prosody information when the target speaker's timbre presents each phoneme for the target emotion category;

[0076] Fuse the second prosody information and the phoneme feature data to generate a speech signal with the target speaker's timbre and the target emotion category.

[0077] It can be understood that human speech usually has certain prosody information. The prosody information represents the cadence in the speech details. The prosody information can be used to characterize the emotion category. Specifically, the prosody information can reflect the emotion characteristics through each phoneme, or through each frame. That is, the prosody information can show the emotion characteristics through the pitch and energy of each phoneme, or through the pitch and energy of each frame.

[0078] The following first introduces phonemes:

[0079] A phoneme is the most basic pronunciation unit in human language. It is a concept in phonetics and corresponds to the smallest speech unit that can distinguish meanings in a language.

[0080] In practical applications, the target text can be split into a phoneme sequence. A phoneme is the smallest speech unit divided according to the natural attributes of speech. It is divided according to the pronunciation actions in a syllable, and one pronunciation action constitutes one phoneme. For example, the Chinese syllable "啊(a1)" has only one phoneme, "代(dai)" has two phonemes d and ai4, etc. "a1" represents the first tone of "a", and "ai4" represents the fourth tone of "ai". Therefore, each text can be split into a corresponding phoneme sequence.

[0081] Exemplarily, if the target text is "欢迎来到这里", the corresponding phonemes are: h, uan1, y, ing2, l, ai2, d, ao4, zh, e4, l, i3. After that, the phonemes can be converted into a corresponding phoneme sequence according to the serial numbers of each phoneme.

[0082] The first prosodic information in this step is the prosodic information corresponding to when the target text of the target emotional category is presented through the timbre of the source speaker. Based on the first prosodic information and the target speaker category, the second prosodic information can be obtained when the target speaker timbre performs speech presentation for each phoneme of the target emotional category. The second prosodic information is the prosodic information corresponding to when the target text of the target emotional category is presented through the timbre of the target speaker.

[0083] In a specific implementation, the source speech data and the target speaker category can be input into a pre-trained prosody predictor. The prosody predictor can extract first prosody information from the source speech data, and combine the first prosody information with the target speaker category to obtain second prosody information of the target speaker category when the target text of the target emotion category is presented in speech.

[0084] It should be noted that both the first prosody information and the second prosody information may be information at the phoneme level, wherein each phoneme may correspond to its own prosody in the first prosody information and the second prosody information, respectively. The first prosody information and the second prosody information may also be information at the frame level, wherein each frame may correspond to its own prosody in the first prosody information and the second prosody information, respectively.

[0085] In a specific implementation, the second prosody information at the phoneme level can be predicted from the first prosody information at the phoneme level, and the second prosody information at the frame level can be predicted from the first prosody information at the frame level.

[0086] As shown in FIG4 , it is a schematic diagram of determining the second prosody information through a prosody predictor in the speech synthesis method provided by an embodiment of the present disclosure, wherein the source speech data and the target speaker category are inputs of the prosody predictor. In this way, the prosody predictor can output the second prosody information based on the first prosody information corresponding to the source speech data and the target speaker category.

[0087] It should be noted that when the prosody information is inferred through the prosody predictor, the source speech data input into the prosody predictor is an acoustic spectrum. When the source speech data is a speech signal in the time domain, the source speech data in the form of a speech signal can be first converted into an acoustic spectrum, and then the converted acoustic spectrum is input into the prosody predictor to obtain the second prosody information.

[0088] Afterwards, the second prosody information and the phoneme feature data are fused to obtain a speech signal with the timbre of the target speaker and the target emotion category. Since the second prosody information is the information when the target speaker's timbre is used to present the target text of the target emotion category, and the phoneme feature data is used to represent the text content of the target text, the combination of the two can obtain a speech signal with the timbre of the target speaker and the target emotion category. The obtained speech signal has prosody information and is more real and natural in auditory performance.

[0089] This technology explicitly normalizes the prosodic information of the target speaker's timbre when presenting the target text of the target emotional category using the prosodic information in the source speech data. This ensures that the generated speech signal corresponds to the source speech data not only in terms of textual content but also in terms of prosodic information. Because the source speech data is relatively realistic and natural, and the prosodic information reflects the target emotional category through the pitch and energy of each phoneme, the generated speech signal not only appears realistic and natural, but also accurately reflects the target emotional category.

[0090] In an optional implementation, step S102 may be implemented by the following steps:

[0091] The source speaker category, target emotion category and target text are input into a pre-trained emotion speech synthesis model so that the emotion speech synthesis model outputs source speech data with the source speaker's timbre and the target emotion category, wherein the emotion speech synthesis model is trained at least based on the source speaker category.

[0092] Specifically, the training data of the pre-trained emotional speech synthesis model includes at least the source speaker category and the target emotion category. The training process of the emotional speech synthesis model is as follows:

[0093] When the training data only includes a source speaker category and a target emotion category, the source speaker category, target emotion category, and sample text are first fed into an untrained emotional speech synthesis model. The source speaker category may currently only have audio data with neutral emotion. Secondly, the untrained emotional speech synthesis model performs duration prediction, specifically predicting the number of frames corresponding to each phoneme when the source speaker category presents sample text with the target emotion category based on the input data. Next, the untrained emotional speech synthesis model performs prosody prediction, specifically predicting the prosodic information corresponding to each phoneme when the source speaker category presents sample text with the target emotion category based on the input data. Finally, the untrained emotional speech synthesis model generates speech data corresponding to the sample text of the target emotion category presented by the source speaker category, frame by frame, combining the predicted number of frames corresponding to each phoneme and the prosodic information. The untrained emotional speech synthesis model then adjusts parameters based on the generated speech data to ensure that the output speech data represents the target emotion category in a realistic and natural manner, enabling the source speaker category with only neutral emotion to synthesize speech data of the target emotion category.

[0094] When the training data includes a source speaker category and various emotion categories, the source speaker category, emotion categories, and sample text are first input into an untrained emotional speech synthesis model. The untrained emotional speech synthesis model then performs duration prediction, specifically predicting the number of frames corresponding to each phoneme when the source speaker category presents sample text of each emotion category based on the input data. The untrained emotional speech synthesis model then performs prosody prediction, specifically predicting the prosodic information corresponding to each phoneme when the source speaker category presents sample text of each emotion category based on the input data. Finally, the untrained emotional speech synthesis model generates speech data corresponding to the sample text of each emotion category presented by the source speaker category, frame by frame, based on the predicted number of frames corresponding to each phoneme and the prosodic information. The untrained emotional speech synthesis model then adjusts parameters based on the generated speech data to ensure that the output speech data is realistic and natural across all emotion categories, enabling a source speaker category with only neutral emotion to synthesize speech data of all emotion categories.

[0095] When the training data includes multiple speaker categories and various emotion categories, the multiple speaker categories include the source speaker category but do not include the target speaker category. First, the multiple speaker categories, various emotion categories and sample texts are input into the untrained emotion speech synthesis model data; secondly, the untrained emotion speech synthesis model can perform duration prediction, specifically by predicting the number of frames corresponding to each phoneme when the multiple speaker categories present the sample texts of each emotion category based on the above input data; then, the untrained emotion speech synthesis model can perform rhythm prediction, specifically by predicting the rhythm information corresponding to each phoneme when the multiple speaker categories present the sample texts of each emotion category based on the above input data; finally, the speech data corresponding to the sample texts of each emotion category presented by multiple speaker categories are generated frame by frame based on the predicted number of frames corresponding to each phoneme and the rhythm information corresponding to each phoneme. Afterwards, the parameters of the untrained emotional speech synthesis model are adjusted based on the generated speech data so that the output speech data is realistic and natural in the performance of various emotional categories, so that multiple speaker categories with only neutral emotions have the ability to synthesize speech data of various emotional categories.

[0096] The above is the pre-trained emotional speech synthesis model.

[0097] In this way, when it is expected to obtain a speech signal of a target text presenting a target emotion category through the timbre of a target speaker, at least the source speech speaker category, the target emotion category and the target text can be input into a pre-trained emotion speech synthesis model. Since the emotion speech synthesis model generates real and natural speech data for each combination of (source speaker category, each emotion category) during the training phase, the emotion speech synthesis model can also generate source speech data with the timbre of the source speaker, the target emotion category and the text content of the target text during the inference phase.

[0098] Through this technical means, the timbre vectors of the emotional speech synthesis model in the training stage and the inference stage come from the same speaker, and the combinations of (timbre vectors, emotion vectors) encountered by the emotional speech synthesis model in the inference stage have been trained in the training stage. In this way, the emotional speech synthesis model can output real and natural source speech data in the inference stage.

[0099] In an optional implementation, step S102 may be implemented by following steps S201 to S203:

[0100] Step S201: Determine emotional feature data according to the target emotional category.

[0101] In a specific implementation, the target emotion category may be embedded to obtain an emotion embedding vector, and the emotion embedding vector is determined as the emotion feature data in step S201.

[0102] In another specific implementation, multiple emotion intensities may be preset for each emotion category. In the present disclosure, the speech signal to be generated may have a target emotion intensity in addition to the target emotion category. In this case, step S201 may also be implemented by the following steps:

[0103] Get the target emotion intensity;

[0104] Determining the starting data and the ending data of the emotional intensity interval to which the target emotional intensity belongs from a plurality of emotional intensities preset for the target emotional category;

[0105] Based on the initial data and the final data, the emotional feature data corresponding to the target text is generated.

[0106] Among the multiple emotion intensities preset in the target emotion category, the starting data and the ending data of the emotion intensity interval to which the target emotion intensity belongs are determined.

[0107] In the present disclosure, an emotion category can be preset with multiple emotion intensities, and the preset multiple emotion intensities corresponding to an emotion category are sorted in order from weak to strong or from strong to weak, so as to obtain at least one emotion intensity interval, that is, an emotion category can be divided into at least one emotion intensity interval. In the embodiment of the present disclosure, the minimum value of the emotion intensity of each emotion category can be set to 0 and the maximum value to 1. Therefore, each emotion category can be divided into an emotion intensity interval [0, 1], or each emotion category can be divided into three emotion intensity intervals [0, 0.33], (0.33, 0.67], (0.67, 1], where when the emotion intensity is 0, it represents the absence of such emotion, that is, the emotion category corresponding to the emotion intensity of 0 under each emotion category is neutral. The two endpoints of each emotion intensity interval are the starting data and the ending data, the starting data includes the emotion category and the starting emotion intensity, and the ending data includes the emotion category and the ending emotion intensity.

[0108] For example, the starting data corresponding to the angry emotion intensity interval [0, 0.33] is: emotion category - neutral, starting emotion intensity - 0, and the corresponding end data is: emotion category - angry, ending emotion intensity - 0.33; the starting data corresponding to another angry emotion intensity interval (0.33, 0.67] is: emotion category - angry, starting emotion intensity - 0.33, and the corresponding end data is: emotion category - angry, ending emotion intensity - 0.67.

[0109] The following is an introduction to the emotional categories divided into different emotional intensity intervals according to the emotional intensity in combination with Table 2:

[0110] Table 2.

[0111] In Table 2, each emotion category is divided into three emotion intensity intervals: anger includes weak anger [0, 0.33], medium anger (0.33, 0.67] and strong anger (0.67, 1]; surprise includes weak surprise [0, 0.33], medium surprise (0.33, 0.67] and strong surprise (0.67, 1]; disgust includes weak disgust [0, 0.33], medium disgust (0.33, 0.67) and strong disgust (0.67, 1]; happiness includes weak happiness [0, 0.3 3], happiness (0.33, 0.67] and happiness (0.67, 1]), fear includes fear (weak [0, 0.33], fear (0.33, 0.67] and fear (0.67, 1]), sadness includes sadness (weak [0, 0.33], sadness (0.33, 0.67] and sadness (0.67, 1]. It should be noted that in the subsequent steps of the embodiment of the present disclosure, the division of the emotion intensity intervals shown in Table 2 is used as an example, but it is not used to limit the present disclosure.

[0112] In this way, after obtaining the target emotion category and target emotion intensity of the target text, the emotion intensity interval to which the target text belongs can be determined, and then the starting data and the ending data of the emotion intensity interval can be determined.

[0113] It is understandable that the starting data and ending data of the emotional intensity interval reflect the minimum emotional intensity and maximum emotional intensity of the emotional intensity interval. The emotional feature data of the target text obtained based on the starting data and ending data can reflect the proportion of the emotional intensity of the target text in the emotional intensity interval to which it belongs, that is, the emotional feature data reflects the emotional strength of the target text in the emotional interval divided under a certain emotional category. The emotional feature data is data that integrates emotional categories and emotional strengths. It is understandable that the more emotional intensities preset in an emotional category, the more corresponding emotional intervals. Accordingly, the more accurate the emotional strength reflected by the emotional feature data generated by the target text, the closer it is to the emotional strength represented by the target emotional intensity to be generated.

[0114] Specifically, the above step of "generating emotional feature data corresponding to the target text based on the starting data and the ending data" can be achieved by the following steps:

[0115] Normalize the target emotion intensity according to the emotion intensity range to obtain the normalized target emotion intensity;

[0116] The normalized target emotion intensity is determined as the first weight, and the difference between the value 1 and the first weight is determined as the second weight;

[0117] Embedding the starting data to obtain first emotion feature data corresponding to the starting data, and embedding the ending data to obtain second emotion feature data corresponding to the ending data;

[0118] The first emotion feature data and the second emotion feature data are weightedly summed according to the second weight and the first weight respectively to obtain the emotion feature data corresponding to the target emotion category.

[0119] Normalization is an important preprocessing step in the fields of data processing and machine learning. It refers to the process of transforming the original data so that the processed data has a uniform scale, range or distribution characteristics.

[0120] In this disclosure, normalizing the target emotion intensity according to the emotion intensity interval means calculating the percentage of the target emotion intensity within the corresponding emotion intensity interval to obtain the normalized target emotion intensity. In this way, within the emotion intensity intervals finely divided for a particular emotion category, the percentage of the target emotion intensity within the corresponding emotion intensity interval is obtained.

[0121] Assume that the target emotion category corresponding to the target text "The weather is so nice today" is happy, the target emotion intensity is t_1, the emotion intensity interval is divided into the emotion intensity intervals in Table 2, the emotion intensity interval is determined according to the value of w, and the normalized target emotion intensity is t_2.

[0122] As shown in Table 3, it is an example table of emotion intensity provided by an embodiment of the present disclosure, in which the emotion intensity in each emotion intensity interval in Table 2 is normalized.

[0123] Table 3.

[0124] Afterwards, the first weight and the second weight are determined according to the normalized target emotion intensity, as shown in Table 4, which is an example table of the first weight and the second weight obtained according to the normalized emotion intensity in Table 3 provided in an embodiment of the present disclosure.

[0125] Table 4.

[0126] Afterwards, the beginning data of the emotion intensity interval determined according to the target emotion intensity and the target emotion category are embedded to obtain emotion feature data corresponding to the beginning data, and the ending data are embedded to obtain emotion feature data corresponding to the ending data.

[0127] Specifically, the emotion category in the starting data can be embedded to obtain the starting emotion category embedding vector, the starting emotion intensity in the starting data can be embedded to obtain the starting emotion intensity embedding vector, and the starting emotion category embedding vector and the starting emotion intensity embedding vector can be integrated to obtain the first emotion feature data corresponding to the starting data; correspondingly, the emotion category in the terminal data can be embedded to obtain the terminal emotion category embedding vector, the terminal emotion intensity in the terminal data can be embedded to obtain the terminal emotion intensity embedding vector, and the terminal emotion category embedding vector and the terminal emotion intensity embedding vector can be integrated to obtain the second emotion feature data corresponding to the terminal data.

[0128] After obtaining the first emotional feature data and the second emotional feature data, the first weight can be used as the weight of the second emotional feature data, and the second weight can be used as the weight of the first emotional feature data. The first emotional feature data and the second emotional feature data are weighted and summed to obtain the emotional feature data corresponding to the target emotional category.

[0129] Step S202: Determine a source speaker embedding vector according to the source speaker category.

[0130] Step S203: Based on the emotion feature data and the source speaker embedding vector, the target text is processed into source speech data having the source speaker's timbre and the target emotion category.

[0131] Specifically, in step S202 , the source speaker category may be embedded to obtain a source speaker embedding vector, and the source speaker embedding vector is used to represent the timbre of the source speaker.

[0132] In step S203, the target text can be processed into source speech data having the source speaker's timbre and the target emotion category based on the emotion feature data representing the target emotion category and the source speaker embedding vector representing the source speaker's timbre. It should be noted that when the emotion feature data represents both the target emotion category and the target emotion intensity, the target text can be processed into source speech data having the source speaker's timbre, the target emotion category, and the target emotion intensity based on the emotion feature data.

[0133] In an optional implementation, the source voice data is frame-level data, and step S203 can be implemented by the following steps:

[0134] Determine, based on the source speaker embedding vector, each phoneme feature vector having the source speaker's timbre corresponding to each phoneme in the phoneme sequence corresponding to the target text;

[0135] Prosody prediction is performed on each phoneme according to the emotional feature data to obtain the prosody information of the phoneme level corresponding to each phoneme;

[0136] According to each phoneme feature vector and each prosody information at the phoneme level, source speech data at the frame level having the timbre of the source speaker and the target emotion category is determined.

[0137] The above-mentioned phoneme feature vector is used to reflect the timbre of the source speaker through each phoneme, that is, the phoneme feature vector represents both the text content of the target text and the timbre of the source speaker. It can be understood that the phoneme feature vector is data at the phoneme level.

[0138] The prosodic information at the phoneme level can reflect the target emotional category through the energy and fundamental frequency of each phoneme. It should be noted that when performing prosodic prediction, the emotional feature data and the source speaker category can be combined to obtain the prosodic information corresponding to each phoneme, that is, the prosodic information is the prosodic information corresponding to each phoneme when the target text of the target emotional category is presented through the timbre of the source speaker.

[0139] In specific implementation, the prosody information corresponding to each phoneme can be predicted by the pre-trained prosody predictor mentioned above. Specifically, the emotional feature data and the source speaker category can be input into the pre-trained prosody predictor, so that the pre-trained prosody predictor outputs the prosody information corresponding to each phoneme.

[0140] It can be understood that the phoneme level is for each phoneme in the target text, that is, the phoneme feature vector is for generating each phoneme feature vector, and the prosody information is for generating each prosody information for each phoneme.

[0141] In this way, based on each phoneme feature vector at the phoneme level and each prosodic component at the phoneme level, the frame-level source speech data having the source speaker's embedded timbre and the target emotion category can be determined.

[0142] Specifically, the step of "determining frame-level source speech data having the source speaker's timbre and the target emotion category based on each phoneme feature vector and phoneme-level prosodic information" can be implemented by the following steps:

[0143] Predict the number of frames for each phoneme to obtain the number of speech frames for each phoneme;

[0144] Each phoneme feature vector is copied according to the corresponding number of speech frames to obtain each frame feature vector at the frame level;

[0145] Copy each prosodic information at the phoneme level according to the corresponding number of speech frames to obtain each prosodic information at the frame level;

[0146] According to the feature vectors of each frame and the prosody information at the frame level, the source speech data at the frame level having the timbre of the source speaker and the target emotion category is determined.

[0147] In the embodiments of the present disclosure, the number of speech frames occupied by each phoneme can be predicted. According to the predicted number of speech frames, each phoneme feature vector and each prosodic information are respectively replicated to obtain each frame feature vector at the frame level and each prosodic information at the frame level. Specifically, a pre-trained duration predictor can be used to predict the number of frames for each phoneme, so as to obtain the number of speech frames corresponding to each phoneme, and the number of speech frames corresponding to each phoneme is the number of frames occupied by each phoneme in the generated source speech data.

[0148] Among them, each frame feature vector at the frame level represents the text content of the target text corresponding to each frame and the timbre of the source speaker. Each prosodic information at the frame level can reflect the emotional characteristics through the energy and fundamental frequency of each frame. It should be noted that each prosodic information at the frame level can be understood as the first prosodic information extracted from the source speech data mentioned above.

[0149] In the specific implementation manner, for each phoneme, its corresponding phoneme-level prosodic information and phoneme feature vector can be replicated according to the corresponding number of speech frames to obtain the frame feature vector corresponding to the phoneme and the prosodic information at the frame level. In this way, each frame feature vector corresponding to each phoneme and each prosodic information at the frame level can be obtained.

[0150] As shown in Table 5, it is an example table of each frame feature vector and each prosodic information at the frame level in the speech synthesis method provided by the embodiments of the present disclosure.

[0151] Table 5.

[0152] In Table 5, the phonemes corresponding to the target text "hao a" are: h, ao2, a1. The phoneme feature vectors corresponding to each phoneme are (c1, c2, c3), and the phoneme-level prosodic information corresponding to each phoneme is (d, e, f). The predicted number of speech frames occupied by each phoneme is 2 frames, 3 frames, and 2 frames respectively. The prosodic information at the frame level and the frame feature vector can be obtained according to the number of speech frames occupied by each phoneme. Among them, the prosodic information at the frame level corresponding to the phoneme h is (d, d), the prosodic information at the frame level corresponding to the phoneme ao1 is (e, e, e), and the prosodic information at the frame level corresponding to the phoneme a1 is (f, f); the frame feature vector corresponding to the phoneme h is (c1, c1), the frame feature vector corresponding to the phoneme ao1 is (c2, c2, c2), and the frame feature vector corresponding to the phoneme a1 is (c3, c3).

[0153] It can be understood that the feature vector of each frame is a vector that integrates the text content of the target text and the timbre of the source speaker. The prosodic information at the frame level represents the prosodic information of each frame. That is to say, the prosodic information at the frame level reflects the target emotion category through each frame. In this way, the source speech data at the frame level with the timbre of the source speaker and the target emotion category can be determined through the feature vectors of each frame and the prosodic information at the frame level.

[0154] In practical applications, prosodic information typically includes fundamental frequency and energy. Fundamental frequency represents the pitch of a sound, while energy represents its intensity. Within a speech, the pitch of a voice is determined by the fundamental frequency, which is the frequency of the fundamental pitch, produced by the periodic vibration of the vocal cords. Generally, when the emotion category is happiness, the speech signal has a larger amplitude and a correspondingly larger fundamental frequency value; when the emotion category is sadness, the speech signal has a smaller amplitude and a correspondingly smaller fundamental frequency value.

[0155] In the case where the prosody information includes fundamental frequency and energy, the step of "performing prosody prediction for each phoneme according to the emotional feature data to obtain prosody information at the phoneme level corresponding to each phoneme" can be implemented by the following steps:

[0156] According to the emotional feature data, the fundamental frequency and energy of each phoneme are predicted respectively to obtain the fundamental frequency value and energy value corresponding to each phoneme;

[0157] Embed the fundamental frequency value and energy value respectively to obtain the fundamental frequency embedding vector and energy embedding vector;

[0158] The fundamental frequency embedding vector and energy embedding vector belonging to the same phoneme are integrated to obtain the prosodic information at the phoneme level corresponding to each phoneme. In a specific embodiment, the fundamental frequency and energy of each phoneme in the target text can be predicted separately according to the emotional feature data to obtain the fundamental frequency value and energy value of each phoneme, and the fundamental frequency value and energy value are embedded separately and mapped into the fundamental frequency embedding vector and the energy embedding vector. Then, for each phoneme, the corresponding fundamental frequency embedding vector and energy embedding vector can be integrated to obtain the prosodic information at the phoneme level corresponding to the phoneme. It should be noted that the fundamental frequency embedding vector and the energy embedding vector are both phoneme-level vectors, and the dimensions of the fundamental frequency embedding vector and the energy embedding vector are the same.

[0159] Through this technical means, the phoneme-level prosodic information corresponding to each phoneme is obtained by predicting the fundamental frequency and energy of each phoneme, making the prediction of the phoneme-level prosodic information corresponding to each phoneme more efficient and accurate. The obtained phoneme-level prosodic information accurately reflects the pitch and intensity of each phoneme.

[0160] In an optional embodiment, the above step of "determining, based on the source speaker embedding vector, each phoneme feature vector having the source speaker's timbre corresponding to each phoneme in the phoneme sequence corresponding to the target text" can also be implemented by the following steps:

[0161] Embed the target text to obtain the text embedding vector corresponding to each phoneme;

[0162] The text embedding vectors corresponding to the phonemes are encoded according to the source speaker embedding vector to obtain the phoneme feature vectors corresponding to the phonemes with the timbre of the source speaker.

[0163] When the speech synthesis method provided by the present disclosure performs data processing, the input is textual data of the target text. The textual data can be first embedded to obtain a text embedding vector corresponding to each phoneme, and then the text embedding vector corresponding to each phoneme is encoded using the source speaker embedding vector to obtain a phoneme feature vector corresponding to each phoneme with the timbre of the source speaker.

[0164] In specific applications, the speech signal generated for the target text may need to adjust the frame number and / or rhythm for certain words. Therefore, in an optional implementation, the speech synthesis method provided in the embodiment of the present disclosure may further include the following steps:

[0165] Adjustment of the number of frames: obtaining the adjusted number of frames for the first phoneme input of the target text; using the adjusted number of frames as the number of speech frames of the first phoneme, returning to execute the step of copying each phoneme feature vector according to the corresponding number of speech frames to obtain each frame feature vector at the frame level, until the source speech data at the frame level is obtained.

[0166] Adjustment of the rhythm may specifically include adjustment of the fundamental frequency value and the energy value: obtaining the adjusted fundamental frequency value and energy value of the second phoneme input for the target text; using the adjusted fundamental frequency value and energy value as the fundamental frequency value and energy value of the second phoneme, returning to execute the steps of embedding the fundamental frequency value and energy value respectively to obtain the fundamental frequency embedding vector and the energy embedding vector, until the source speech data at the frame level is obtained.

[0167] It should be noted that the first phoneme is the phoneme for which the number of speech frames needs to be adjusted, and the second phoneme is the phoneme for which the prosody information needs to be adjusted. In addition, the first phoneme and the second phoneme can be the same phoneme or different phonemes, and this disclosure does not limit this.

[0168] In a specific implementation, the number of speech frames of the first phoneme and / or the prosodic information of the second phoneme can be adjusted through user operations. Therefore, the above step of "obtaining the adjusted number of frames input for the first phoneme in the target text" can be in response to the frame number adjustment operation, and the adjusted number of frames input for the first phoneme in the target text can be obtained. The above step of "obtaining the adjusted fundamental frequency value and energy value input for the second phoneme in the target text" can be in response to the prosodic adjustment operation, and the adjusted fundamental frequency value and energy value input for the second phoneme in the target text can be obtained.

[0169] Specifically, for the adjustment of the frame number, there is at least one first phoneme in the target text that needs to adjust the frame number. The user can input the adjusted frame number for the first phoneme, and then return to execute the step of "copying each phoneme feature vector according to the corresponding speech frame number to obtain each frame feature vector at the frame level", and copy the corresponding phoneme feature vector and phoneme-level prosody information of the first phoneme that needs to adjust the frame number according to the adjusted frame number, and replace the frame feature vector and frame-level prosody information corresponding to the copied first phoneme with the position of the first phoneme in the frame-level prosody information and frame feature vector obtained before the adjustment operation, so as to obtain the adjusted frame-level prosody information and frame feature vector, and based on this, obtain the frame-level source speech data after the adjusted frame number.

[0170] Specifically, for prosody adjustment, there is at least one second phoneme in the target text that requires prosody adjustment. The user can input adjusted prosody information for this second phoneme, which can specifically be a fundamental frequency value and an energy value. The adjusted fundamental frequency value and energy value are then used as the fundamental frequency value and energy value of the second phoneme, and the process returns to the step of "embedding the fundamental frequency value and energy value to obtain a fundamental frequency embedding vector and an energy embedding vector" to obtain the phoneme-level adjusted prosody information corresponding to the second phoneme. The adjusted prosody information is then copied to the predicted number of speech frames for the second phoneme that requires prosody adjustment, obtaining the frame-level adjusted prosody information corresponding to the second phoneme. The information is then replaced at the position of the second phoneme in the frame-level prosody information to obtain the adjusted frame-level prosody information. Based on the adjusted frame-level prosody information and the frame feature vectors, the frame-level source speech data with adjusted prosody is obtained.

[0171] In addition, when the emotion feature also includes emotion intensity data, the present disclosure may also make adjustments based on the emotion intensity.

[0172] Specifically, the following steps can be taken to adjust the intensity of emotions:

[0173] In response to the emotion intensity adjustment operation, obtaining an adjusted emotion intensity of a target word inputted into the target text; the target word includes at least one phoneme;

[0174] Determining adjusted emotional feature data corresponding to each target phoneme in the target word according to the adjusted emotional intensity;

[0175] The adjusted emotional feature data corresponding to the target phoneme is used as the emotional feature data of the target phoneme, and the process returns to step S201 until the source speech data at the frame level is obtained. It should be noted that when the source speech data generated before adjusting at least one of the number of frames, prosody information, and emotional intensity is a speech signal in the time domain, the source speech data can be played to the user for the user to audition, and the user adjusts at least one of the number of frames, prosody information, and emotional intensity based on the audition result. When the source speech data generated before adjusting at least one of the number of frames, prosody information, and emotional intensity is an acoustic spectrum, the source speech data in the form of an acoustic spectrum can be first converted into a speech signal in the time domain, and then the speech signal in the time domain can be played to the user for the user to audition, and the user adjusts at least one of the number of frames, prosody information, and emotional intensity based on the audition result.

[0176] By adjusting at least one of the number of speech frames, prosody information, and emotional intensity, the generated source speech data can be made more realistic and natural in terms of emotion, intonation, and dwell time of each word, closer to real-person speech.

[0177] The following is an introduction to the speech synthesis system used in the speech synthesis method provided in the embodiment of the present disclosure: As shown in Figure 5, the speech synthesis system 10 may include two subsystems, one is the emotional speech synthesis subsystem 11, and the other is the speech conversion subsystem 12, wherein the emotional speech synthesis subsystem 11 can be used to synthesize source speech data, and the speech conversion subsystem 12 can be used to identify information related to the target text in the source speech data, and finally successfully convert the source speaker's timbre in the source speech data into the target speaker's timbre, and make the emotional features in the final generated speech signal the emotional features when the target speaker's timbre presents the target emotional category.

[0178] Figure 6 is a schematic diagram of the data flow in the speech synthesis system provided by an embodiment of the present disclosure. First, the source speaker category 20, target text 21, and target emotion category 22 are input into the affective speech synthesis subsystem 11, causing the affective speech synthesis subsystem 11 to output source speech data 23. Second, the source speech data 23 and target speaker category 24 are input into the speech conversion subsystem 12, causing the speech conversion subsystem 12 to output a speech signal 25.

[0179] Corresponding to the speech synthesis method provided in the first embodiment of the present disclosure, the second embodiment of the present disclosure further provides a speech synthesis device. As shown in Figure 7, the speech synthesis device 700 includes: an acquisition unit 701, which is used to acquire the target text and the target emotion category to be synthesized into speech; a first processing unit 702, which is used to process the target text into source speech data with the timbre of the source speaker and the target emotion category according to the source speaker category and the target emotion category, wherein the source speaker category has at least audio data under the target emotion category; a determination unit 703, which is used to determine the phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; and a second processing unit 704, which is used to process the phoneme feature data into a speech signal with the timbre of the target speaker and the target emotion category according to the source speech data and the target speaker category.

[0180] By using the various units in the aforementioned speech synthesis device, when it is desired to generate a speech signal with the timbre of a target speaker and a target emotional category, a source speaker category having audio data in each emotional category is first used to generate relatively realistic and natural source speech data when the source speaker's timbre presents a target text of the target emotional category. A relatively realistic and natural speech signal is then obtained by combining the source speech data and the target speaker category. The speech synthesis method provided in this application can avoid serious timbre leakage, emotional transfer failure, and emotional flatness in the synthesized speech, thereby ensuring that the synthesized speech retains the target emotional category and timbre of the target speaker to the greatest extent possible.

[0181] Optionally, determination unit 703 is specifically configured to: if the source speech data is an acoustic spectrum, convert the acoustic spectrum into a time-domain speech signal; and remove information from the time-domain speech signal that is irrelevant to the target text to obtain phoneme feature data corresponding to the target text. The technical means employed by the determination unit can effectively remove information irrelevant to the target text from the source speech data, thereby obtaining phoneme feature data relevant to the target text.

[0182] Optionally, the second processing unit 704 is specifically configured to: extract first prosodic information corresponding to the source speech data, the first prosodic information being used to characterize the target emotion category; determine second prosodic information when the target speaker's timbre is used to present each phoneme of the target emotion category based on the first prosodic information and the target speaker category; and fuse the second prosodic information with the phoneme feature data to generate a speech signal having the target speaker's timbre and the target emotion category. Through a technical approach demonstrated by the second processing unit, the prosodic information of the target speaker's timbre when presenting the target text of the target emotion category is explicitly standardized using the prosodic information in the source speech data. Thus, the generated speech signal corresponds to the source speech data not only in terms of textual content but also in terms of prosodic information. Because the source speech data is relatively realistic and natural, and the prosodic information reflects the target emotion category through the pitch and energy of each phoneme, the generated speech signal not only produces realistic and natural speech but also accurately reflects the target emotion category.

[0183] Optionally, the first processing unit 702 is specifically configured to input the source speaker category, target emotion category, and target text into a pre-trained emotional speech synthesis model, so that the emotional speech synthesis model outputs source speech data having the source speaker's timbre and the target emotion category, wherein the emotional speech synthesis model is trained based at least on the source speaker category. Through the technical means demonstrated by the first processing unit, the timbre vectors of the emotional speech synthesis model in the training phase and the inference phase come from the same speaker, and the combinations of (timbre vectors, emotion vectors) encountered by the emotional speech synthesis model in the inference phase have been trained in the training phase. In this way, the emotional speech synthesis model can output realistic and natural source speech data in the inference phase.

[0184] Optionally, the first processing unit 702 is specifically used to: determine emotional feature data based on the target emotional category; determine the source speaker embedding vector based on the source speaker category; and process the target text into source speech data with the source speaker's timbre and the target emotional category based on the emotional feature data and the source speaker embedding vector.

[0185] Optionally, the source speech data is frame-level data, and the first processing unit 702 is specifically used to: determine, based on the source speaker embedding vector, each phoneme feature vector with the source speaker's timbre corresponding to each phoneme in the phoneme sequence corresponding to the target text; perform prosody prediction on each phoneme based on the emotional feature data to obtain each phoneme-level prosody information corresponding to each phoneme; and determine, based on each phoneme feature vector and each phoneme-level prosody information, the frame-level source speech data with the source speaker's timbre and the target emotional category.

[0186] Optionally, the first processing unit 702 is specifically used to: predict the number of frames for each phoneme to obtain the number of speech frames for each phoneme; copy the feature vectors of each phoneme according to the corresponding number of speech frames to obtain frame-level feature vectors; copy the prosodic information at the phoneme level according to the corresponding number of speech frames to obtain frame-level prosodic information; and determine the frame-level source speech data having the timbre of the source speaker and the target emotion category based on the feature vectors of each frame and the prosodic information at the frame level.

[0187] Optionally, the first processing unit 702 is specifically configured to: perform fundamental frequency prediction and energy prediction on each phoneme based on the emotional feature data to obtain a fundamental frequency value and energy value corresponding to each phoneme; embed the fundamental frequency value and energy value to obtain a fundamental frequency embedding vector and an energy embedding vector; and integrate the fundamental frequency embedding vector and energy embedding vector belonging to the same phoneme to obtain phoneme-level prosodic information corresponding to each phoneme. By using the above-mentioned technical means, the phoneme-level prosodic information corresponding to each phoneme is obtained by performing fundamental frequency prediction and energy prediction on each phoneme, making the prediction of the phoneme-level prosodic information corresponding to each phoneme more efficient and accurate. The obtained phoneme-level prosodic information accurately reflects the pitch and intensity of each phoneme.

[0188] Optionally, the speech synthesis device 700 also includes an adjustment unit, which is used to: obtain the adjusted frame number for the first phoneme input in the target text; use the adjusted frame number as the speech frame number of the first phoneme, and return to the step of copying each phoneme feature vector according to the corresponding speech frame number to obtain each frame feature vector at the frame level, until the source speech data at the frame level is obtained.

[0189] Optionally, the adjustment unit is specifically used to: obtain the adjusted fundamental frequency value and energy value for the second phoneme input in the target text; use the adjusted fundamental frequency value and energy value as the fundamental frequency value and energy value of the second phoneme, and return to the step of embedding the fundamental frequency value and energy value respectively to obtain the fundamental frequency embedding vector and the energy embedding vector, until the source speech data at the frame level is obtained.

[0190] In the above embodiment, by adjusting at least one of the number of speech frames, prosody information and emotion intensity, the emotion, intonation and dwell time of each word corresponding to the generated source speech data are made more real and natural, and closer to real human speech.

[0191] Corresponding to the speech synthesis method provided in the first embodiment of the present disclosure, the third embodiment of the present disclosure further provides an electronic device for speech synthesis. As shown in Figure 8, electronic device 800 includes: a processor 801; and a memory 802 for storing a program for the speech synthesis method. After the device is powered on and the speech synthesis method program is executed by the processor, the following steps are performed:

[0192] Obtain target text to be synthesized into speech and a target emotion category; process the target text into source speech data having the timbre of the source speaker and the target emotion category according to the source speaker category and the target emotion category, wherein the source speaker category has at least audio data under the target emotion category; determine phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; and process the phoneme feature data into a speech signal having the timbre of the target speaker and the target emotion category according to the source speech data and the target speaker category.

[0193] The speech synthesis method provided in this application, when it is desired to generate a speech signal with the timbre of a target speaker and a target emotional category, first generates source speech data with a source speaker category having audio data in various emotional categories, and then generates a relatively realistic and natural speech signal with the source speaker's timbre reflecting the target emotional category when the target text is presented. The speech synthesis method provided in this application can avoid serious timbre leakage, emotional transfer failure, and emotional flatness in the synthesized speech, ensuring that the synthesized speech retains the target emotional category and the timbre of the target speaker to the greatest extent possible.

[0194] Optionally, determining the phoneme feature data corresponding to the target text from the source speech data includes: if the source speech data is an acoustic spectrum, converting the acoustic spectrum into a time-domain speech signal; and removing information from the time-domain speech signal that is irrelevant to the target text to obtain the phoneme feature data corresponding to the target text. This technical approach can effectively remove information irrelevant to the target text from the source speech data and obtain phoneme feature data relevant to the target text.

[0195] Optionally, based on the source speech data and the target speaker category, the phoneme feature data is processed into a speech signal having the timbre of the target speaker and the target emotion category, including: extracting first prosody information corresponding to the source speech data, the first prosody information being used to characterize the target emotion category; based on the first prosody information and the target speaker category, determining second prosody information when the target speaker timbre is used to present each phoneme of the target emotion category; and fusing the second prosody information with the phoneme feature data to generate a speech signal having the timbre of the target speaker and the target emotion category.

[0196] This technical approach explicitly normalizes the prosodic information of the target speaker's timbre when presenting the target text of the target emotional category using the prosodic information in the source speech data. This ensures that the generated speech signal corresponds to the source speech data not only in terms of textual content but also in terms of prosodic information. Because the source speech data is relatively realistic and natural, and the prosodic information reflects the target emotional category through the pitch and energy of each phoneme, the generated speech signal not only appears realistic and natural, but also accurately reflects the target emotional category.

[0197] Optionally, based on the source speaker category and the target emotion category, the target text is processed into source speech data having the source speaker's timbre and the target emotion category, including: inputting the source speaker category, the target emotion category, and the target text into a pre-trained emotion speech synthesis model, so that the emotion speech synthesis model outputs source speech data having the source speaker's timbre and the target emotion category, wherein the emotion speech synthesis model is trained based at least on the source speaker category. Through the technical means demonstrated above, the timbre vectors of the emotion speech synthesis model in the training phase and the inference phase come from the same speaker, and the combination of (timbre vector, emotion vector) encountered by the emotion speech synthesis model in the inference phase has been trained in the training phase, so that the emotion speech synthesis model can output real and natural source speech data in the inference phase.

[0198] Optionally, based on the source speaker category and the target emotion category, the target text is processed into source speech data having the source speaker timbre and the target emotion category, including: determining emotion feature data based on the target emotion category; determining a source speaker embedding vector based on the source speaker category; and processing the target text into source speech data having the source speaker timbre and the target emotion category based on the emotion feature data and the source speaker embedding vector.

[0199] Optionally, the source speech data is frame-level data, and the target text is processed into source speech data with the source speaker's timbre and target emotion category based on the emotional feature data and the source speaker embedding vector, including: determining each phoneme feature vector with the source speaker's timbre corresponding to each phoneme in the phoneme sequence corresponding to the target text based on the source speaker embedding vector; performing prosody prediction on each phoneme based on the emotional feature data to obtain each prosody information at the phoneme level corresponding to each phoneme; and determining the frame-level source speech data with the source speaker's timbre and target emotion category based on each phoneme feature vector and each prosody information at the phoneme level.

[0200] Optionally, based on each phoneme feature vector and the prosodic information at the phoneme level, the source speech data at the frame level having the timbre of the source speaker and the target emotion category is determined, including: predicting the number of frames for each phoneme to obtain the number of speech frames for each phoneme; copying each phoneme feature vector according to the corresponding number of speech frames to obtain each frame feature vector at the frame level; copying each prosodic information at the phoneme level according to the corresponding number of speech frames to obtain each prosodic information at the frame level; and determining the source speech data at the frame level having the timbre of the source speaker and the target emotion category based on each frame feature vector and each frame prosodic information.

[0201] Optionally, prosody prediction is performed on each phoneme based on the emotional feature data to obtain prosody information at the phoneme level corresponding to each phoneme, including: performing fundamental frequency prediction and energy prediction on each phoneme based on the emotional feature data to obtain fundamental frequency values ​​and energy values ​​corresponding to each phoneme; embedding the fundamental frequency values ​​and energy values ​​to obtain fundamental frequency embedding vectors and energy embedding vectors; integrating the fundamental frequency embedding vectors and energy embedding vectors belonging to the same phoneme to obtain prosody information at the phoneme level corresponding to each phoneme. Through the above technical means, the prosody information at the phoneme level corresponding to each phoneme is obtained by performing fundamental frequency prediction and energy prediction on each phoneme, making the prediction of the prosody information at the phoneme level corresponding to each phoneme more efficient and accurate, and the obtained prosody information at the phoneme level accurately reflects the pitch and intensity of each phoneme.

[0202] Optionally, after obtaining the number of speech frames for each phoneme, the method also includes: obtaining the adjusted frame number for the first phoneme input in the target text; using the adjusted frame number as the number of speech frames for the first phoneme, returning to the step of copying each phoneme feature vector according to the corresponding number of speech frames to obtain each frame feature vector at the frame level, until the source speech data at the frame level is obtained.

[0203] Optionally, the method also includes: obtaining the adjusted fundamental frequency value and energy value for the second phoneme input in the target text; using the adjusted fundamental frequency value and energy value as the fundamental frequency value and energy value of the second phoneme, and returning to the step of embedding the fundamental frequency value and energy value respectively to obtain the fundamental frequency embedding vector and the energy embedding vector, until the source speech data at the frame level is obtained.

[0204] In the technical means provided in the above embodiments, by adjusting at least one aspect of the number of speech frames, prosody information and emotional intensity, the emotions, intonation and dwell time of each word corresponding to the generated source speech data are made more real and natural, and closer to real human speech.

[0205] Corresponding to the speech synthesis method provided in the first embodiment of the present disclosure, the fourth embodiment of the present disclosure provides a computer-readable storage medium storing a program of the speech synthesis method, which is executed by a processor to perform the following steps:

[0206] Obtain target text to be synthesized into speech and a target emotion category; process the target text into source speech data having the timbre of the source speaker and the target emotion category according to the source speaker category and the target emotion category, wherein the source speaker category has at least audio data under the target emotion category; determine phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; and process the phoneme feature data into a speech signal having the timbre of the target speaker and the target emotion category according to the source speech data and the target speaker category.

[0207] The speech synthesis method provided in this application, when it is desired to generate a speech signal with the timbre of a target speaker and a target emotional category, first generates source speech data with a source speaker category having audio data in various emotional categories, and then generates a relatively realistic and natural speech signal with the source speaker's timbre reflecting the target emotional category when the target text is presented. The speech synthesis method provided in this application can avoid serious timbre leakage, emotional transfer failure, and emotional flatness in the synthesized speech, ensuring that the synthesized speech retains the target emotional category and the timbre of the target speaker to the greatest extent possible.

[0208] Optionally, determining phoneme feature data corresponding to the target text from the source speech data includes: if the source speech data is an acoustic spectrum, converting the acoustic spectrum into a speech signal in the time domain; and removing information from the time domain speech signal that is irrelevant to the target text to obtain the phoneme feature data corresponding to the target text. This technical approach demonstrated above can effectively remove information irrelevant to the target text from the source speech data, effectively obtaining phoneme feature data relevant to the target text.

[0209] Optionally, based on the source speech data and the target speaker category, the phoneme feature data is processed into a speech signal having the target speaker's timbre and the target emotion category, including: extracting first prosodic information corresponding to the source speech data, the first prosodic information being used to characterize the target emotion category; determining second prosodic information when the target speaker's timbre is used to present each phoneme of the target emotion category based on the first prosodic information and the target speaker category; and fusing the second prosodic information with the phoneme feature data to generate a speech signal having the target speaker's timbre and the target emotion category. Through the above-described technical approach, the prosodic information of the target speaker's timbre when presenting the target text of the target emotion category is explicitly standardized using the prosodic information in the source speech data. Thus, the generated speech signal corresponds to the source speech data not only in terms of textual content but also in terms of prosodic information. Because the source speech data is relatively realistic and natural, and the prosodic information reflects the target emotion category through the pitch and energy of each phoneme, the generated speech signal is not only realistic and natural, but also accurately reflects the target emotion category.

[0210] Optionally, based on the source speaker category and the target emotion category, the target text is processed into source speech data having the source speaker's timbre and the target emotion category, including: inputting the source speaker category, the target emotion category, and the target text into a pre-trained emotion speech synthesis model, so that the emotion speech synthesis model outputs source speech data having the source speaker's timbre and the target emotion category, wherein the emotion speech synthesis model is trained based at least on the source speaker category. Through the technical means demonstrated above, the timbre vectors of the emotion speech synthesis model in the training phase and the inference phase come from the same speaker, and the combination of (timbre vector, emotion vector) encountered by the emotion speech synthesis model in the inference phase has been trained in the training phase, so that the emotion speech synthesis model can output real and natural source speech data in the inference phase.

[0211] Optionally, based on the source speaker category and the target emotion category, the target text is processed into source speech data having the source speaker timbre and the target emotion category, including: determining emotion feature data based on the target emotion category; determining a source speaker embedding vector based on the source speaker category; and processing the target text into source speech data having the source speaker timbre and the target emotion category based on the emotion feature data and the source speaker embedding vector.

[0212] Optionally, the source speech data is frame-level data, and the target text is processed into source speech data with the source speaker's timbre and target emotion category based on the emotional feature data and the source speaker embedding vector, including: determining each phoneme feature vector with the source speaker's timbre corresponding to each phoneme in the phoneme sequence corresponding to the target text based on the source speaker embedding vector; performing prosody prediction on each phoneme based on the emotional feature data to obtain each prosody information at the phoneme level corresponding to each phoneme; and determining the frame-level source speech data with the source speaker's timbre and target emotion category based on each phoneme feature vector and each prosody information at the phoneme level.

[0213] Optionally, based on each phoneme feature vector and the prosodic information at the phoneme level, the source speech data at the frame level having the timbre of the source speaker and the target emotion category is determined, including: predicting the number of frames for each phoneme to obtain the number of speech frames for each phoneme; copying each phoneme feature vector according to the corresponding number of speech frames to obtain each frame feature vector at the frame level; copying each prosodic information at the phoneme level according to the corresponding number of speech frames to obtain each prosodic information at the frame level; and determining the source speech data at the frame level having the timbre of the source speaker and the target emotion category based on each frame feature vector and each frame prosodic information.

[0214] Optionally, prosody prediction is performed on each phoneme based on the emotional feature data to obtain prosody information at the phoneme level corresponding to each phoneme, including: performing fundamental frequency prediction and energy prediction on each phoneme based on the emotional feature data to obtain fundamental frequency values ​​and energy values ​​corresponding to each phoneme; embedding the fundamental frequency values ​​and energy values ​​to obtain fundamental frequency embedding vectors and energy embedding vectors; integrating the fundamental frequency embedding vectors and energy embedding vectors belonging to the same phoneme to obtain prosody information at the phoneme level corresponding to each phoneme. Through the above technical means, the prosody information at the phoneme level corresponding to each phoneme is obtained by performing fundamental frequency prediction and energy prediction on each phoneme, making the prediction of the prosody information at the phoneme level corresponding to each phoneme more efficient and accurate, and the obtained prosody information at the phoneme level accurately reflects the pitch and intensity of each phoneme.

[0215] Optionally, after obtaining the number of speech frames for each phoneme, the method also includes: obtaining the adjusted frame number for the first phoneme input in the target text; using the adjusted frame number as the number of speech frames for the first phoneme, returning to the step of copying each phoneme feature vector according to the corresponding number of speech frames to obtain each frame feature vector at the frame level, until the source speech data at the frame level is obtained.

[0216] Optionally, the method also includes: obtaining the adjusted fundamental frequency value and energy value for the second phoneme input in the target text; using the adjusted fundamental frequency value and energy value as the fundamental frequency value and energy value of the second phoneme, and returning to the step of embedding the fundamental frequency value and energy value respectively to obtain the fundamental frequency embedding vector and the energy embedding vector, until the source speech data at the frame level is obtained.

[0217] In the technical means provided in the above embodiments, by adjusting at least one aspect of the number of speech frames, prosody information and emotional intensity, the emotions, intonation and dwell time of each word corresponding to the generated source speech data are made more real and natural, and closer to real human speech.

[0218] It should be noted that for the detailed description of the apparatus, electronic device and computer-readable storage medium provided in the second, third and fourth embodiments of the present disclosure, reference can be made to the relevant description of the first embodiment of the present disclosure, and no further details will be given here.

[0219] Although the present disclosure is disclosed as above in terms of preferred embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

[0220] In a typical configuration, a node device in a blockchain includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0221] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0222] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), random access memory (RAM) of other types, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage media, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0223] 2. Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] Although the present disclosure is disclosed as above in terms of preferred embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

Claims

1. A speech synthesis method, comprising: Obtain the target text to be synthesized into speech and the target emotion category; According to the source speaker category and the target emotion category, the target text is processed into source speech data having the source speaker timbre and the target emotion category, wherein the source speaker category at least has audio data under the target emotion category Determining phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to characterize the text content of the target text; According to the source speech data and the target speaker category, the phoneme feature data is processed into a speech signal having the timbre of the target speaker and the target emotion category.

2. The method according to claim 1, wherein The determining of the phoneme feature data corresponding to the target text from the source speech data includes: In a case where the source speech data is an acoustic spectrum, converting the acoustic spectrum into a speech signal in the time domain; Information irrelevant to the target text is removed from the speech signal in the time domain to obtain phoneme feature data corresponding to the target text.

3. The method according to claim 1, wherein The step of processing the phoneme feature data into a speech signal having the timbre of the target speaker and the target emotion category according to the source speech data and the target speaker category includes: Extracting first prosody information corresponding to the source speech data, where the first prosody information is used to represent the target emotion category; determining, based on the first prosody information and the target speaker category, second prosody information when the target speaker's timbre performs speech presentation for each of the phonemes of the target emotion category; The second prosody information and the phoneme feature data are fused to generate a speech signal having the timbre of a target speaker and the target emotion category.

4. The method according to claim 1, wherein The step of processing the target text into source speech data having the source speaker's timbre and the target emotion category according to the source speaker category and the target emotion category includes: The source speaker category, the target emotion category and the target text are input into a pre-trained emotion speech synthesis model so that the emotion speech synthesis model outputs source speech data with the source speaker's timbre and the target emotion category, wherein the emotion speech synthesis model is trained at least based on the source speaker category.

5. The method according to claim 1, wherein The step of processing the target text into source speech data having the source speaker's timbre and the target emotion category according to the source speaker category and the target emotion category includes: Determining emotional feature data according to the target emotional category; Determining a source speaker embedding vector according to the source speaker category; The target text is processed into source speech data having the timbre of the source speaker and the target emotion category according to the emotion feature data and the source speaker embedding vector.

6. The method according to claim 5, wherein: The source speech data is frame-level data, and the processing of the target text into source speech data having the source speaker's timbre and the target emotion category according to the emotion feature data and the source speaker embedding vector includes: Determining, based on the source speaker embedding vector, each phoneme feature vector having the source speaker's timbre corresponding to each phoneme in the phoneme sequence corresponding to the target text; Performing prosody prediction on each of the phonemes according to the emotional feature data to obtain prosody information at the phoneme level corresponding to each of the phonemes; According to each of the phoneme feature vectors and each of the prosody information at the phoneme level, source speech data at the frame level having the source speaker's timbre and the target emotion category is determined.

7. The method according to claim 6, wherein: The determining, based on each of the phoneme feature vectors and the prosodic information at the phoneme level, the frame-level source speech data having the source speaker's timbre and the target emotion category comprises: Predicting the number of frames for each of the phonemes to obtain the number of speech frames for each of the phonemes; Copying each of the phoneme feature vectors according to the corresponding number of speech frames to obtain each frame feature vector at the frame level; Copying each prosodic information at the phoneme level according to the corresponding number of speech frames to obtain each prosodic information at the frame level; According to each of the frame feature vectors and each of the frame-level prosody information, frame-level source speech data having the source speaker's timbre and the target emotion category is determined.

8. The method according to claim 6, wherein: The prosody prediction of each of the phonemes according to the emotional feature data is performed to obtain prosody information at the phoneme level corresponding to each of the phonemes, including: Performing fundamental frequency prediction and energy prediction on each of the phonemes according to the emotional feature data to obtain a fundamental frequency value and an energy value corresponding to each of the phonemes; Embedding the fundamental frequency value and the energy value respectively to obtain a fundamental frequency embedding vector and an energy embedding vector; The fundamental frequency embedding vector and the energy embedding vector belonging to the same phoneme are integrated to obtain the prosody information at the phoneme level corresponding to each phoneme.

9. The method according to claim 7, wherein: After obtaining the number of speech frames of each phoneme, the method further includes: Obtaining an adjusted frame number for a first phoneme input in the target text; The adjusted number of frames is used as the number of speech frames of the first phoneme, and the step of copying each of the phoneme feature vectors according to the corresponding number of speech frames to obtain each frame feature vector at the frame level is returned to, until the source speech data at the frame level is obtained.

10. The method according to claim 8, wherein The method further comprises: Obtaining an adjusted fundamental frequency value and energy value for a second phoneme input in the target text; The adjusted fundamental frequency value and energy value are used as the fundamental frequency value and energy value of the second phoneme, and the step of embedding the fundamental frequency value and the energy value respectively to obtain a fundamental frequency embedding vector and an energy embedding vector is returned to, until the source speech data at the frame level is obtained.

11. A speech synthesis device, comprising: An acquisition unit configured to acquire a target text to be synthesized into speech and a target emotion category; A first processing unit is configured to process the target text into source speech data having a source speaker's timbre and the target emotion category according to a source speaker category and the target emotion category, wherein the source speaker category at least has audio data under the target emotion category; a determining unit configured to determine phoneme feature data corresponding to the target text from the source speech data, wherein the phoneme feature data is used to represent the text content of the target text; The second processing unit is configured to process the phoneme feature data into a speech signal having the timbre of the target speaker and the target emotion category according to the source speech data and the target speaker category.

12. An electronic device comprising: processor; as well as The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 10 is executed.

13. A computer-readable storage medium storing a data processing program, wherein the program is executed by a processor to perform the method according to any one of claims 1 to 10.

Images