Speech synthesis method, apparatus and electronic device

Abstract

The application discloses a speech synthesis method and device, electronic equipment and a computer readable storage medium. The method comprises: obtaining a target text to be converted into speech, a target emotion category and a target emotion intensity; determining start end information and end information of an emotion intensity interval to which the target emotion intensity belongs from a plurality of emotion intensities preset for the target emotion category; generating emotion feature data corresponding to the target text according to the start end information and the end information; processing the target text into each phoneme vector of a phoneme level with emotion features according to the emotion feature data; performing prosody prediction and prediction of the number of speech frames occupied on each phoneme vector respectively to obtain first prosody information and a speech content vector at a frame level; and processing the speech content vector into a time-domain speech signal according to the first prosody information. The scheme provided by the application enables the synthesized speech to correctly reflect the emotion type and emotion intensity, thereby improving the human-computer interaction experience of the user.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a speech synthesis method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of artificial intelligence technology, speech synthesis technology has been widely applied. Speech synthesis is a technology that generates artificial speech through mechanical and electronic methods, capable of converting any text information into standard and fluent speech in real time. In human-computer interaction scenarios, speech synthesis technology enables machines to "speak" and interact with users. Users often hope that the synthesized speech is more realistic, thereby improving the user's human-computer interaction experience.

[0003] In related technologies, to make synthesized speech have a certain degree of realism, text is usually input into a pre-trained acoustic model to generate speech data of a certain emotion. Emotion categories include anger, happiness, sadness, and many others. The acoustic model is trained by inputting a certain amount of speech data samples of each emotion category, enabling the trained acoustic model to generate speech data of various different emotions.

[0004] However, human voices not only have different emotions, but also varying degrees of intensity. The voice data synthesized by the aforementioned voice synthesis schemes cannot reflect the intensity of emotions, resulting in a poor human-computer interaction experience for users. Summary of the Invention

[0005] This application provides a speech synthesis method, apparatus, electronic device, and computer-readable storage medium, enabling the synthesized speech to not only accurately reflect the type of emotion but also the intensity of emotion, thereby improving the user's human-computer interaction experience. The specific solution is as follows.

[0006] In a first aspect, embodiments of this application provide a speech synthesis method, the method comprising:

[0007] Obtain the target text to be converted into speech, the target emotion category, and the target emotion intensity;

[0008] From a plurality of preset emotion intensities for the target emotion category, determine the beginning and end information of the emotion intensity range to which the target emotion intensity belongs;

[0009] Based on the starting information and the ending information, generate the emotion feature data corresponding to the target text;

[0010] Based on the emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features;

[0011] Prosodic prediction and the number of speech frames occupied by each of the phoneme vectors are performed respectively to obtain the first prosodic information and speech content vector at the frame level.

[0012] Based on the first prosody information, the speech content vector is processed into a time-domain speech signal.

[0013] Secondly, embodiments of this application provide a speech synthesis device, the device comprising:

[0014] The acquisition unit is used to acquire the target text to be converted into speech, the target emotion category, and the target emotion intensity.

[0015] The determining unit is used to determine the beginning and end information of the emotional intensity range to which the target emotional intensity belongs from a plurality of preset emotional intensities for the target emotional category.

[0016] The generation unit is used to generate emotion feature data corresponding to the target text based on the starting information and the ending information.

[0017] The first processing unit is used to process the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data.

[0018] The prediction unit is used to predict the prosody and the number of speech frames occupied by each of the phoneme vectors, so as to obtain the first prosodic information and speech content vector at the frame level.

[0019] The second processing unit is used to process the speech content vector into a time-domain speech signal based on the prosody information.

[0020] Thirdly, this application also provides an electronic device, including:

[0021] Processor; and

[0022] A memory for storing a data processing program, which, when the electronic device is powered on and runs by the processor, performs the method as described in any of the first aspects.

[0023] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a data processing program that is executed by a processor to perform the method described in any of the first aspects.

[0024] Compared with the prior art, this application has the following advantages:

[0025] The speech synthesis method provided in this application pre-determines multiple emotion intensities for each emotion category, divides the emotion intensity into emotion intensity intervals, and then determines the corresponding emotion intensity interval based on the target emotion category and target emotion intensity of the target text. Emotion feature data corresponding to the target text is generated based on the beginning and end information of the target emotion intensity interval. Since the emotion intensity interval is obtained after further subdividing the emotion category, the generated emotion feature data corresponding to the target text not only correctly reflects the emotion category but also correctly reflects the emotion intensity. Then, based on the emotion feature data, the target text is processed into phoneme vectors at the phoneme level with emotion features. Here, the target text is split into individual phonemes, and each phoneme vector is obtained by processing the emotion feature data of the target text information. It can be seen that each phoneme vector is a phoneme-level vector integrating phoneme, emotion category, and emotion intensity. Next, prosodic prediction and the number of speech frames occupied by each phoneme vector are performed to obtain the prosodicity and the number of frames occupied by each phoneme, thus obtaining the first prosodic information containing the prosodic features of each frame. Based on the number of frames occupied by each phoneme, the phoneme-level phoneme vectors are processed into frame-level speech content vectors. Since each phoneme vector integrates phoneme, emotion category, and emotion intensity, the speech content vector is a frame-level vector that integrates each phoneme and the emotion category and emotion intensity of each frame. Then, based on the first prosodic information containing the prosodic features of each frame and the speech content vector based on the target text content containing the emotion features (emotion category and emotion intensity) of each frame, the speech signal corresponding to the target text is generated.

[0026] As can be seen, the speech synthesis method provided in this application divides an emotion category into more detailed emotion intensity intervals based on multiple emotion intensities. It then generates more accurate emotion feature data for the target emotion intensity according to its corresponding interval. This ensures that the synthesized speech not only accurately reflects the emotion type but also the required emotion intensity of the text. Furthermore, the speech synthesis method provided in this application also considers the prosodic features of the target text, making the synthesized speech realistic and natural, closer to human speech, and improving the user's human-computer interaction experience. Attached Figure Description

[0027] Figure 1 This is a flowchart of the speech synthesis method provided in the embodiments of this application;

[0028] Figure 2 These are detailed diagrams of the speech synthesis method provided in the embodiments of this application;

[0029] Figure 3 This is a structural block diagram of an example of the speech synthesis device provided in the embodiments of this application;

[0030] Figure 4This is an architecture diagram of the speech synthesis system provided in the embodiments of this application;

[0031] Figure 5 This is a structural block diagram of an example of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0033] It should be noted that the terms "first," "second," "third," etc., in the claims, specification, and drawings of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. Such data are interchangeable where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown or described herein. Furthermore, the terms "comprising," "having," and their variations are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0034] Speech synthesis is a technology that uses mechanical and electronic methods to generate artificial speech, capable of converting any text information into standard, fluent speech in real time. In human-computer interaction scenarios, speech synthesis technology enables machines to "speak" and interact with users. Users often hope that the synthesized speech is more realistic, thereby improving their human-computer interaction experience.

[0035] In related technologies, to make synthesized speech have a certain degree of realism, text is usually input into a pre-trained acoustic model to generate speech data of a certain emotion. Emotion categories include anger, happiness, sadness, and many others. The acoustic model is trained by inputting a certain amount of speech data samples of each emotion category, enabling the trained acoustic model to generate speech data of various different emotions.

[0036] However, human voices not only have different emotions, but also varying degrees of intensity. The voice data synthesized by the aforementioned voice synthesis schemes cannot reflect the intensity of emotions, resulting in a poor human-computer interaction experience for users.

[0037] For the reasons mentioned above, in order to ensure that the synthesized speech can not only accurately reflect the type of emotion, but also accurately reflect the intensity of the emotion, thereby improving the user's human-computer interaction experience, the first embodiment of this application provides a speech synthesis method. This method is applied to an electronic device, which may be a desktop computer, a laptop computer, a mobile phone, a tablet computer, a server, a terminal device, etc., or other electronic devices capable of data statistics. This application embodiment is not specifically limited.

[0038] The following describes the application scenarios of the speech synthesis method provided in this application. The speech synthesis method provided in this application can be applied to human-computer interaction in game scenarios. In games, there are non-player characters (NPCs) who can guide players through the game and interact with the virtual characters controlled by the players. Depending on the game scenario or the change in game pace, NPCs often emit different voices. The speech synthesis method provided in this application can synthesize voices for NPCs that fit various game scenarios. During player-NPC interaction, when the NPC's voice function is triggered, the NPC can "speak" according to the current game scenario. The speech synthesis method provided in this application can also be applied to robot voice. When a user "communicates" with a robot, the robot emits voices that fit the current context; this application does not specifically limit this application.

[0039] The following combination Figure 1 The speech synthesis method provided in the first embodiment of this application will be described in detail.

[0040] like Figure 1 The diagram shown is a flowchart of a speech synthesis method provided in an embodiment of this application. The method includes the following steps S101 to S106.

[0041] Step S101: Obtain the target text to be converted into speech, the target emotion category, and the target emotion intensity.

[0042] In this step, the target text can be a sentence or a paragraph, and its content can be Chinese, English, or any other language. The target emotion category is the emotion category corresponding to the speech to be generated from the target text. Emotion categories can be divided into seven categories according to Paul Ekman's basic emotion theory: anger, surprise, disgust, happiness, fear, sadness, and neutrality. Neutrality indicates no emotion. Emotion categories can also be categorized into four main types: joy, anger, sorrow, and happiness. The target emotion intensity is the intensity of the emotion to be generated from the target text. Emotion intensity represents the strength of each emotion category; different intensities reflect different degrees of the corresponding emotion category. For example, the emotion category of anger can correspond to intensities such as very angry and slightly angry. The specific division of emotion categories and intensities is determined based on the actual situation in specific applications; however, this application does not specifically limit this.

[0043] Typically, the emotion type and intensity corresponding to each word (or phoneme) in a text should be basically consistent. That is, in this step, for a target text, given an emotion category and an emotion intensity, we can obtain the emotion type and intensity corresponding to each word (or phoneme) in the target text.

[0044] Step S102: Determine the beginning and end information of the emotional intensity range to which the target emotional intensity belongs from a plurality of preset emotional intensities for the target emotional category.

[0045] In this step, an emotion category can have multiple preset emotion intensities. These preset intensities are then sorted in ascending or descending order, resulting in at least one emotion intensity interval. In other words, an emotion category can be divided into at least one emotion intensity interval. In this embodiment, the minimum emotion intensity for each emotion category can be set to 0, and the maximum to 1. Therefore, each emotion category can be divided into one emotion intensity interval [0, 1], or into three emotion intensity intervals: [0, 0.33], (0.33, 0.67], and (0.67, 1]. An emotion intensity of 0 represents the absence of that emotion; that is, the emotion category corresponding to an emotion intensity of 0 is neutral. The two endpoints of each emotion intensity interval are the start and end information. The initial information includes the initial emotion category and the initial emotion intensity, and the final information includes the final emotion category and the final emotion intensity. For example, the initial information corresponding to the emotion intensity range of anger [0, 0.33] is: emotion category - neutral, emotion intensity -0, and the corresponding final information is: emotion category - angry, emotion intensity -0.33; the initial information corresponding to another emotion intensity range of anger (0.33, 0.67] is: emotion category - angry, emotion intensity -0.33, and the corresponding final information is: emotion category - angry, emotion intensity -0.67.

[0046] Below are two examples of emotion categories divided into different intensity ranges based on emotion intensity.

[0047] Example 1: Each emotion category can be divided into 5 emotion intensity intervals, as shown in Table 1, which is an example table of emotion intensity intervals divided for each emotion category provided in the embodiments of this application.

[0048] Table 1. An example of the intensity ranges for each emotion category.

[0049]

[0050] Example 2: Each emotion category can be divided into 3 emotion intensity intervals, as shown in Table 2, which is another example table of emotion intensity intervals divided for each emotion category provided in the embodiments of this application.

[0051] Table 2. Another example of the emotion intensity ranges divided into each emotion category.

[0052]

[0053] It should be noted that the subsequent steps in the embodiments of this application are illustrated by the division of emotional intensity ranges shown in Table 2, and are not intended to limit this application.

[0054] In this way, after obtaining the target emotion category and target emotion intensity of the target text, the emotion intensity interval to which the target text belongs can be determined, and then the start information and end information of the emotion intensity interval to which it belongs can be determined.

[0055] Step S103: Generate emotion feature data corresponding to the target text according to the start information and the end information.

[0056] In this step, the start information and end information of the emotion intensity interval reflect the minimum emotion intensity and the maximum emotion intensity of this emotion intensity interval. Obtaining the emotion feature data of the target text according to the start information and end information can reflect the proportion of the emotion intensity of the target text in the emotion intensity interval to which it belongs, that is, the emotion feature data reflects the degree of emotion strength of the target text in the emotion interval divided under a certain emotion category. The emotion feature data is data integrating the emotion category and emotion intensity. In specific applications, the emotion feature data can be represented by a vector.

[0057] It can be understood that the more emotion intensities preset for the emotion category, the more corresponding emotion intervals there are. Correspondingly, the more accurate the degree of emotion strength reflected by the emotion feature data generated by the target text is, and it is closer to the degree of emotion strength represented by the target emotion intensity to be generated.

[0058] Step S104: Process the target text into each phoneme vector at the phoneme level with emotion features according to the emotion feature data.

[0059] In this step, the phoneme level refers to each phoneme in the target text, that is, each phoneme vector is generated for each phoneme in the target text. In specific applications, the target text can be split into a phoneme sequence. A phoneme is the smallest speech unit divided according to the natural attributes of speech, and is divided according to the pronunciation actions in a syllable. One pronunciation action constitutes one phoneme. For example, the Chinese syllable "啊(a1)" has only one phoneme, "代(dai)" has two phonemes d and ai4, etc. "a1" represents that the tone of "a" is the first tone, and "ai4" represents that the tone of "ai" is the fourth tone. Therefore, each text can be split into a corresponding phoneme sequence. For example, in a possible scenario, the target text is "欢迎来到这里", then the corresponding phonemes are: h, uan1, y, ing2, l, ai2, d, ao4, zh, e4, l, i3, and the phonemes are converted into a corresponding phoneme sequence according to the serial numbers of each phoneme.

[0060] Since a target text corresponds to an emotion category and an emotion intensity, each phoneme in the phoneme sequence corresponds to the same emotion category and the same emotion intensity. Therefore, the phoneme vectors obtained from the emotion feature data of the target text are vectors with the same emotion category and emotion intensity but different phoneme content. Because the emotion feature data reflects the intensity of emotion within an emotion interval defined by an emotion category, each phoneme vector reflects the intensity of emotion within that interval. Each phoneme vector is a vector integrating the corresponding phoneme, its corresponding emotion category, and its emotion intensity.

[0061] In this way, we obtain the phoneme vectors containing emotional features for each phoneme in the target text, which are divided according to the pronunciation action.

[0062] Step S105: Perform prosodic prediction and prediction of the number of speech frames occupied by each of the phoneme vectors to obtain the first prosodic information and speech content vector at the frame level.

[0063] In practical applications, in addition to emotional features, human voices also possess prosodic features. Emotional features represent the overall emotional category and intensity of the speech, while prosodic features represent the intonation and rhythm of the speech details. If prosodic features are not considered during speech synthesis, the generated speech will be unrealistic and unnatural, resulting in a poor human-computer interaction experience for users.

[0064] To make the synthesized speech more similar to real human speech, this step performs prosodic prediction on the phoneme vectors corresponding to each phoneme. Since the phoneme vectors integrate the phoneme, the corresponding emotion category, and the emotion intensity, prosodic information corresponding to each phoneme can be generated based on the phoneme vectors.

[0065] Furthermore, since speech is a quasi-stationary signal, meaning it is short-term stationary (typically 10-30 ms), speech signal processing involves frame segmentation to reduce the impact of overall non-stationary and time-varying characteristics. Accordingly, to generate realistic short-term stationary speech signals, the features of each frame can be obtained frame by frame, and then the corresponding speech can be generated based on these features.

[0066] The following introduces the relevant concepts of audio frames. The sampling rate (SFR) refers to the number of samples extracted from a continuous signal per second to form a discrete signal, measured in Hertz (Hz). To ensure smooth transitions between frames and maintain continuity, audio framing typically employs overlapping segmentation, guaranteeing that adjacent frames partially overlap. The number of sampling points collected between the starting positions of two adjacent frames is called the frame shift. The number of frames per second is calculated as sampling rate / frame shift. In this embodiment, the frame shift and sampling rate can be set (e.g., a sampling rate of 22050Hz and a frame shift of 256 sampling points). Based on the set frame shift and sampling rate, the characteristics of each frame are obtained, thereby generating a speech signal.

[0067] In this step, the number of speech frames occupied by each phoneme is predicted, and the first prosodic information and speech content vector at the frame level are obtained based on the predicted number of speech frames. The first prosodic information reflects the prosodic information of each frame in the generated speech, and the speech content vector is a vector reflecting the text content in the generated speech as well as the emotion category and emotion intensity of each frame.

[0068] Step S106: Based on the first prosody information, process the speech content vector into a time-domain speech signal.

[0069] In this step, based on the first prosodic vector reflecting the prosodic features of each frame in the speech to be generated, and the speech content vector based on the target text reflecting the emotion category and emotion intensity of each frame, the first prosodic information with the prosodic features of each frame and the speech content vector based on the target text content with the emotion features (emotion category and emotion intensity) of each frame can be generated. Then, after subsequent decoding and conversion processing, the speech signal corresponding to the target text is obtained.

[0070] The speech synthesis method provided in this application pre-determines multiple emotion intensities for each emotion category, divides the emotion intensity into emotion intensity intervals, and then determines the corresponding emotion intensity interval based on the target emotion category and target emotion intensity of the target text. Emotion feature data corresponding to the target text is generated based on the beginning and end information of the target emotion intensity interval. Since the emotion intensity interval is obtained after further subdividing the emotion category, the generated emotion feature data corresponding to the target text not only correctly reflects the emotion category but also correctly reflects the emotion intensity. Then, based on the emotion feature data, the target text is processed into phoneme vectors at the phoneme level with emotion features. Here, the target text is split into individual phonemes, and each phoneme vector is obtained by processing the emotion feature data of the target text information. It can be seen that each phoneme vector is a phoneme-level vector integrating phoneme, emotion category, and emotion intensity. Next, prosodic prediction and the number of speech frames occupied by each phoneme vector are performed to obtain the prosodicity and the number of frames occupied by each phoneme, thus obtaining the first prosodic information containing the prosodic features of each frame. Based on the number of frames occupied by each phoneme, the phoneme-level phoneme vectors are processed into frame-level speech content vectors. Since each phoneme vector integrates phoneme, emotion category, and emotion intensity, the speech content vector is a frame-level vector that integrates each phoneme and the emotion category and emotion intensity of each frame. Then, based on the first prosodic information containing the prosodic features of each frame and the speech content vector based on the target text content containing the emotion features (emotion category and emotion intensity) of each frame, the speech signal corresponding to the target text is generated.

[0071] As can be seen, the speech synthesis method provided in this application divides an emotion category into more detailed emotion intensity intervals based on multiple emotion intensities. It then generates more accurate emotion feature data for the target emotion intensity according to its corresponding interval. This ensures that the synthesized speech not only accurately reflects the emotion type but also the required emotion intensity of the text. Furthermore, the speech synthesis method provided in this application also considers the prosodic features of the target text, making the synthesized speech realistic and natural, closer to human speech, and improving the user's human-computer interaction experience.

[0072] Based on the above implementation methods, in order to make the emotional feature data corresponding to the generated target text more accurate, the speech synthesis method provided in this application can calculate the emotional feature data of the target text based on the emotional feature data of the starting information and the emotional feature data of the ending information.

[0073] The emotional feature data of the starting information and the emotional intensity data of the ending information can be preset data. After determining the emotional intensity range based on the target emotional intensity, the corresponding emotional feature data of the starting information and the emotional intensity data of the ending information can be selected from the preset data. The emotional feature data of the starting information and the emotional intensity data of the ending information can also be obtained by real-time embedding. This application does not specifically limit this.

[0074] Step S103 can be implemented through the following steps S201 to S204.

[0075] Step S201: Normalize the target emotion intensity according to the emotion intensity range to obtain the normalized target emotion intensity.

[0076] Normalization is a dimensionless processing method that transforms the absolute values ​​of a physical system into relative values. In this application, normalizing the target emotion intensity according to emotion intensity intervals means calculating the percentage of the target emotion intensity within a corresponding emotion intensity interval to obtain the normalized target emotion intensity. Thus, within the detailed division of emotion intensity intervals for an emotion category, the proportion of the target emotion intensity within its respective interval is obtained.

[0077] If the target emotion category corresponding to the target text "Welcome here" in the example shown in step S104 is happiness, the target emotion intensity is w, the emotion intensity interval is divided into the emotion intensity intervals in Table 2, the emotion intensity interval is determined according to the value of w, and the normalized target emotion intensity w′ is obtained.

[0078] Table 3 shows an example table of emotion intensity after normalizing the emotion intensity in each emotion intensity range in Table 2, as provided in the embodiments of this application.

[0079] Table 3. Examples of normalized emotion intensities within each emotion intensity range.

[0080]

[0081] Step S202: The normalized target emotion intensity is determined as the first weight, and the difference between the value 1 and the first weight is determined as the second weight.

[0082] Step S203: Embed the starting information to obtain the emotional feature data corresponding to the starting information, and embed the ending information to obtain the emotional feature data corresponding to the ending information.

[0083] Step S204: The emotion feature data corresponding to the starting information and the emotion feature data corresponding to the ending information are weighted and summed according to the second weight and the first weight, respectively, to obtain the emotion feature data corresponding to the target text.

[0084] In step S202, a first weight and a second weight are determined based on the normalized target emotion intensity, as shown in Table 4, which is an example table of the first weight and the second weight obtained based on the normalized emotion intensity in Table 3 provided in this application embodiment.

[0085] Table 4. Examples of the first and second weights obtained from the normalized emotion intensity.

[0086]

[0087] In step S203, the initial information of the emotion intensity range determined based on the target emotion intensity and the target emotion category is embedded to obtain the emotion feature data corresponding to the initial information, and the final information is also embedded to obtain the emotion feature data corresponding to the final information. The emotion feature data is the embedding vector obtained after embedding the numerical values. Specifically: the initial emotion category in the initial information is embedded to obtain the initial emotion category embedding vector, the initial emotion intensity in the initial information is embedded to obtain the initial emotion intensity embedding vector, and the initial emotion category embedding vector and the initial emotion intensity embedding vector are integrated to obtain the emotion feature data corresponding to the initial information; correspondingly, the final emotion category in the final information is embedded to obtain the final emotion category embedding vector, the final emotion intensity in the final information is embedded to obtain the final emotion intensity embedding vector, and the final emotion category embedding vector and the final emotion intensity embedding vector are integrated to obtain the emotion feature data corresponding to the initial information.

[0088] In step S204, the first weight is used as the weight of the emotional feature data of the end information in the emotional intensity interval, the second weight is used as the weight of the emotional feature data of the beginning information in the emotional intensity interval, and the emotional feature data of the end information and the emotional feature data of the beginning information are weighted and summed to obtain the emotional feature data of the target text.

[0089] As shown in Table 5, this application provides an example table of emotional feature data of the target text obtained by weighting and summing the emotional feature data of the end information and the emotional feature data of the beginning information according to the first weight and the second weight obtained in Table 4. In the table, a1 is the emotional intensity data when the emotional intensity is 0, a2 is the emotional intensity data when the emotional intensity is 0.33, a3 is the emotional intensity data when the emotional intensity is 0.67, and a4 is the emotional intensity data when the emotional intensity is 1.

[0090] Table 5. Examples of sentiment feature data for target text

[0091]

[0092]

[0093] In this way, the emotional intensity is normalized according to the emotional intensity range to obtain a more granular normalized emotional intensity within the range. Then, the emotional feature data from the beginning and end of each emotional intensity range are weighted and summed to obtain the emotional feature data of the target text. The emotional feature data from the beginning and end of the range accurately reflect the emotional characteristics at both ends of the emotional intensity range. Therefore, by weighting and summing a limited number of emotional feature data points, more accurate emotional feature data can be obtained, thus eliminating the need for overly detailed classification of emotional categories to obtain emotional feature data that accurately reflects the emotional intensity of the text. By presetting multiple emotional intensities for each emotional category, the granularity of controlling the emotional intensity of the text-to-speech conversion can be made more refined. The more emotional intensities preset for each emotional category, the better the final control effect of the speech emotion and the more refined the control granularity.

[0094] Optionally, step S105 can be implemented according to the following steps S205 to S208.

[0095] Step S205: Perform prosodic prediction on each of the phoneme vectors to obtain the second prosodic information of each phoneme.

[0096] Step S206: Predict the number of speech frames occupied by each phoneme vector to obtain the number of speech frames occupied by each phoneme.

[0097] Step S207: Copy the second prosodic information of each phoneme according to the corresponding predicted speech frame number to obtain the first prosodic information at the frame level.

[0098] Step S208: Copy each phoneme vector according to the corresponding predicted speech frame number to obtain a frame-level speech content vector.

[0099] Understandably, each phoneme vector is a vector that integrates phoneme, emotion category, and emotion intensity. Based on the phoneme, emotion category, and emotion intensity, the second prosodic information corresponding to each phoneme can be predicted. That is, the prosodic information predicted based on each phoneme vector is phoneme-level prosodic information.

[0100] The number of speech frames occupied by each phoneme can be predicted based on the phonemes, emotion categories, and emotion intensities integrated in each phoneme vector. According to the number of speech frames occupied by each phoneme, the first prosody information at the phoneme level and the phoneme vectors at the phoneme level are copied according to the corresponding predicted number of speech frames, obtaining the second prosody information at the frame level and the speech content vectors at the frame level.

[0101] It should be noted that there is no sequential order between the prosody prediction for each phoneme in step S205 and the prediction of the number of speech frames occupied by each phoneme in step S206. In practical applications, the two can be predicted simultaneously, or one can be predicted first and the other later. Correspondingly, there is no sequential order between step S207 and step S208, which will not be elaborated here.

[0102] Exemplarily, the phonemes corresponding to the target text "hao a" are: h, ao2, a1. The respective phoneme vectors at the phoneme level are (c1, c2, c3), the second prosody information of each phoneme predicted is (d, e, f), and the predicted number of speech frames occupied by each phoneme is 2 frames, 3 frames, and 2 frames respectively. The first prosody information at the frame level and the speech content vectors can be obtained according to the number of speech frames occupied by each phoneme. As shown in Table 6, it is an example table of obtaining the first prosody information at the frame level and the speech content vectors according to the second prosody information and phoneme vectors at the phoneme level provided by an embodiment of the present application.

[0103] Table 6. Example of the first prosody information and speech content vectors at the frame level

[0104]

[0105] In practical applications, prosody information usually includes fundamental frequency and energy. The fundamental frequency represents the pitch of the sound, and the energy represents the strength of the sound. In a piece of speech, the pitch change of the speech is determined by the fundamental frequency. The fundamental frequency is the frequency of the fundamental tone vibration, and the fundamental tone is generated by the periodic vibration of the vocal cords. Generally, when the emotion category is happy, the speech signal has a larger amplitude and the corresponding fundamental frequency value is larger; when the emotion category is sad, the speech signal has a smaller amplitude and the corresponding fundamental frequency value is smaller.

[0106] In the case where the prosody information includes fundamental frequency and energy, step S205 can be implemented through the following steps S209 - S211.

[0107] S209: Perform fundamental frequency prediction and energy prediction on each of the phoneme vectors respectively to obtain the fundamental frequency value and energy value corresponding to each phoneme.

[0108] S210: Convert the fundamental frequency value and the energy value into fundamental frequency embedding vectors and energy embedding vectors respectively.

[0109] S211: Integrate the fundamental frequency embedding vector and the energy embedding vector belonging to the same phoneme to obtain the second prosodic information of each phoneme.

[0110] In a specific implementation, the fundamental frequency and energy of each phoneme in the target text are predicted based on the phoneme vectors, resulting in the fundamental frequency and energy values ​​for each phoneme. These values ​​are then embedded separately, mapping them to a fundamental frequency embedding vector and an energy embedding vector. It is known that both the fundamental frequency embedding vector and the energy embedding vector are phoneme-level vectors, and they have the same dimension. Finally, the fundamental frequency embedding vector and the energy embedding vector are integrated to form the second prosodic information.

[0111] This technique uses the fundamental frequency and energy of each phoneme to obtain the prosodic information of each phoneme, making the prediction of prosodic information more efficient and accurate. The obtained second prosodic information accurately reflects the pitch and intensity of each phoneme.

[0112] The speech synthesis method provided in this application takes text data of the target text as input during data processing. The text data can be converted into a text embedding vector for subsequent data processing. In this application, the text embedding vector is generated based on the phoneme sequence. Therefore, step S104 can be implemented according to the following steps S212 to S214.

[0113] Step S212: Convert the target text into text embedding vectors corresponding to each phoneme;

[0114] Step S213: Copy the emotion feature data according to the number of phonemes in the target text to obtain the emotion feature data corresponding to each phoneme;

[0115] Step S214: For any of the phonemes, the text embedding vector corresponding to the phoneme is encoded according to the emotional feature data corresponding to the phoneme to obtain the phoneme vector with emotional features.

[0116] Understandably, when a target text corresponds to an emotion category and an emotion intensity, the emotion feature data of the target text is the emotion feature data of each phoneme. That is, in step S213, the emotion feature data of each phoneme is identical. The emotion feature data is copied according to the number of phonemes in the target text, aiming to obtain emotion feature data with the same length as the phonemes. Since the text embedding vector is the vector mapped to the phoneme sequence of the target text, the length of the text embedding vector is also the same as the phoneme length. In this way, the text embedding vector and the emotion feature data can be encoded to obtain a phoneme vector representing the emotion features with the same length as the phonemes.

[0117] In an optional implementation, when performing speech synthesis, users often want to generate speech with various different timbres as needed. Therefore, the speech synthesis method provided in this application can preset multiple speaker categories. When there are multiple speaker categories, before step S102, the speech synthesis method provided in this application further includes the following steps:

[0118] Obtain the target speaker category;

[0119] The target speaker category is converted into a speaker embedding vector at the phoneme level.

[0120] Accordingly, step S104 can be implemented in the following way:

[0121] Based on the emotional feature data and the speaker embedding vector, the target text is processed into phoneme vectors at the phoneme level, each with emotional features and speaker timbre.

[0122] This application pre-defines multiple speaker categories. When a user inputs target text, target emotion category, and target emotion intensity, they can input the target speaker category corresponding to the desired timbre of the generated speech. In specific implementations, different speaker categories correspond to different timbres. Then, similar to converting the target text into a text embedding vector, the target speaker category is mapped to a speaker embedding vector. The speaker embedding vector can be pre-defined and selected based on the input speaker category, or it can be an embedding vector obtained by embedding the speaker category in real time; this application does not specifically limit this.

[0123] When the input data includes speaker categories, the phoneme vectors for each phoneme can be generated based on emotion feature data and speaker embedding vectors. Thus, the resulting phoneme vectors not only possess emotion features (emotion category and intensity) but also speaker timbre features. Consequently, the timbre of the final generated speech corresponds to the timbre of the target speaker category.

[0124] This technology allows for the generation of speech signals with different timbres for different speaker categories, making the speech synthesis method provided in this application more flexible and further improving the user's human-computer interaction experience.

[0125] Optionally, step S106 can be implemented according to the following steps:

[0126] The first prosodic information and the speech content vector are decoded to obtain the acoustic spectrum corresponding to the target text; the acoustic spectrum is then converted into a time-domain speech signal.

[0127] In this step, the acoustic spectrum is the Mel spectrum, which can reflect the acoustic characteristics of the speech signal to be generated. In other words, decoding the first prosodic information and the speech content vector is essentially predicting the acoustic characteristics of each frame based on the first prosodic information and the speech content vector at the frame level.

[0128] Understandably, the Mel spectrogram represents the distribution of a speech signal across different frequencies. The Mel scale is based on the sensory perception of pitch by listeners at equal distances. Since the human ear is more sensitive to distinguishing low-frequency signals than high-frequency signals, it is easier for the human ear to distinguish low-frequency frequencies at equal distances from high-frequency frequencies within the normal frequency range. Therefore, the Mel scale was proposed, ensuring that frequencies at equal distances in the low-frequency and high-frequency ranges appear identical to each other on the new scale. Thus, the Mel spectrogram is a spectral image based on the characteristics of human hearing.

[0129] When prosodic information includes fundamental frequency and energy, predicting acoustic features based on the first prosodic information and speech content vector can be considered as predicting acoustic features based on the fundamental frequency embedding vector and energy embedding vector, obtaining the Mel spectrum in the frequency domain, and converting the Mel spectrum into a speech signal in the time domain.

[0130] In a specific implementation, the Mel spectrum can be converted into a time-domain speech signal using the HIFI-GAN model (a generative adversarial network model for efficient and high-fidelity speech synthesis). The input to HIFI-GAN is the Mel spectrum, which is upsampled through multiple convolutional layers until the output is a time-domain waveform with the same number of frames as the total number of predicted speech frames.

[0131] In practical applications, the generated speech signal for the target text may require adjustments to the emotional intensity and / or frame count and / or prosody for certain words. Therefore, users can listen to the synthesized speech signal and input the corresponding adjustment information for words that require adjustment of emotional intensity, duration (frame count), and prosody.

[0132] Therefore, in order to make the generated speech from the target text more realistic and natural, the speech synthesis method provided in this application may further include the following adjustments:

[0133] For frame number adjustment: obtain the adjusted frame number for the target phoneme input of the target text; based on the adjusted frame number, return to step S207 until the adjusted speech signal is obtained.

[0134] For prosody adjustment: obtain the adjusted fundamental frequency value and energy value of the target phoneme input for the target text; based on the adjusted fundamental frequency value and energy value, return to step S210 until the adjusted speech signal is obtained.

[0135] Adjustment of emotional intensity: Obtain the adjusted emotional intensity of the target word input for the target text; the target word includes at least one target phoneme; based on the adjusted emotional intensity, determine the adjusted emotional feature data corresponding to each target phoneme in the target word; based on the adjusted emotional feature data corresponding to each target phoneme, return to step S214 until the adjusted speech signal is obtained.

[0136] Specifically, regarding frame number adjustment, at least one character in the target text requires frame number adjustment. The user can input the adjusted frame number for that character. The adjusted frame number can be determined by the user's assessment after listening to the text and whether the duration prediction module predicts too many or too few frames for that character. Then, returning to step S210, the phoneme whose frame number needs adjustment is copied according to the adjusted frame number, along with its corresponding phoneme vector and second prosodic information. The copied first prosodic information and speech content vector at the frame level are then used to replace the phoneme's position in the first prosodic information and speech content vector of the target text, respectively. This yields the adjusted first prosodic information and speech content vector of the target text, and based on this, the adjusted frame number speech signal is obtained.

[0137] Specifically, for prosodic adjustment, at least one character in the target text requires prosodic adjustment. The user can input the adjusted prosodic information for that character, which can specifically be the fundamental frequency value and energy value. Then, returning to step S210, the adjusted second prosodic information is obtained. The phoneme requiring prosodic adjustment is copied from the adjusted second prosodic information according to the predicted speech frame number, resulting in the frame-level adjusted first prosodic information for that phoneme. This first prosodic information is then inserted into the first prosodic information of the target text to obtain the adjusted first prosodic information of the target text. Based on the adjusted first prosodic information of the target text and the speech content vector of the target text, the prosodic-adjusted speech signal is obtained.

[0138] Specifically, for adjusting the emotional intensity, at least one word in the target text needs to have its emotional intensity adjusted. The user can input the adjusted emotional intensity for that word, and then return to step S214 to obtain the adjusted emotional range. Then, the adjusted phoneme vector corresponding to that word is calculated, and the adjusted phoneme vector is replaced with the position of that phoneme in each phoneme vector of the target text to obtain each phoneme vector of the adjusted target text. Each phoneme vector of the adjusted target text is copied according to the predicted duration of each phoneme to obtain the frame-level adjusted speech content vector. Based on the first prosodic information of the target text and the adjusted speech content vector, the speech signal with adjusted emotional intensity is obtained.

[0139] It should be noted that, in response to the above-mentioned adjustments to the number of frames, rhythm, and emotional intensity at the character level, this application obtains the adjusted phoneme vector and the adjusted second rhythm information at the corresponding phoneme level according to the adjusted information for the phoneme corresponding to the character, and then replaces the adjusted phoneme vector and the adjusted second rhythm information in the corresponding positions, while the phoneme vector and the second rhythm information corresponding to other phonemes remain unchanged.

[0140] For example, the target text "Welcome here" in step S104 has a happy emotion category and an initial emotion intensity of 0.3. After listening to the synthesized speech, the user adjusts the emotion intensity of "welcome" to 0.5. Table 7 shows an example table of adjusted phoneme vectors after adjusting the emotion intensity of the target characters in the target text provided in this application embodiment. In this table, m1 is the emotion feature data when the happiness intensity is 0, m2 is the emotion feature data when the happiness intensity is 0.33, and m3 is the emotion feature data when the happiness intensity is 0.67.

[0141] Table 7. Examples of adjusted phoneme vectors after adjusting the emotional intensity of the target words in the target text.

[0142]

[0143] The above specific implementation methods achieve intra-sentence emotion control and prosodic control of a target text information, making the intra-sentence emotions and intonation more realistic and natural, and closer to human speech.

[0144] However, in practical applications, when a real person speaks a passage, the emotional category and intensity of the preceding and following sentences are often different. When there is a sudden emotional shift between two sentences—for example, the preceding sentence is happy and the following sentence is angry—the emotional transition between the two sentences synthesized using the aforementioned speech synthesis method will be abrupt and rather abrupt. The resulting speech will have a disjointed emotional flow and will not closely resemble a real person's voice. Therefore, to make the emotional transition between sentences smoother and more natural, when the target text contains a preceding sentence, the speech synthesis method provided in this application embodiment may further include the following steps before step S104:

[0145] Step S215: Based on at least some of the emotional feature data of the previous sentence text, adjust the emotional feature data of the transition interval in the target text to obtain the adjusted emotional feature data corresponding to the target text; the transition interval includes the text start portion of the target text that includes a preset number of phonemes.

[0146] Step S104 can be implemented according to the following steps:

[0147] Based on the adjusted emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features.

[0148] Specifically, step S215 can be implemented according to the following step S216.

[0149] Step S216: Weighted summation of at least a portion of the emotional feature data of the preceding text with the emotional feature data of each phoneme in the target text located within the transition interval, to obtain the adjusted emotional feature data corresponding to each phoneme in the transition interval; wherein, the third weight required for weighted summation of at least a portion of the emotional feature data of the preceding text gradually decreases according to the phoneme order in the transition interval, and the fourth weight required for weighted summation of emotional feature data of each phoneme in the target text located within the transition interval gradually increases according to the phoneme order in the transition interval.

[0150] Understandably, to make the emotional transition between sentences more natural, the following sentence often needs to consider the emotional type and intensity of the preceding sentence, adjusting the emotional type and intensity of the following sentence accordingly. In this embodiment, when generating speech for the target text, a transition interval can be selected in the target text. The transition interval includes the beginning portion of the text containing a preset number of phonemes, and the emotional feature data corresponding to the transition interval is adjusted according to the emotional feature data of the preceding sentence.

[0151] In practical applications, there may be multiple sentiment feature data corresponding to the preceding text. For example, the sentiment intensity of some words in the preceding text may be adjusted so that the sentiment feature data corresponding to each word in the preceding text are not exactly the same. In this case, the sentiment feature data corresponding to the phonemes in the second half of the preceding text can be selected. Specifically, the sentiment feature data corresponding to the last phoneme in the preceding text can be selected to adjust the sentiment feature data in the transition interval of the target text.

[0152] Specifically, within the transition interval, the emotional feature data of the phonemes in the latter half of the preceding sentence can be weighted and summed with the emotional feature data of each phoneme within the transition interval. Within the transition interval, following the phoneme order, the weights of the emotional feature data of the latter half of the preceding sentence gradually decrease from 1 to 0, while the weights of the emotional feature data of each phoneme within the transition interval gradually increase from 0 to 1. If multiple emotional feature data correspond to the phonemes in the latter half of the preceding sentence, the emotional feature data of the last phoneme in the preceding sentence can be selected and weighted together with the emotional feature data of each phoneme in the transition interval of the target text according to their respective weights to obtain the emotional feature data of each phoneme in the transition interval.

[0153] In practice, the transition interval can be selected based on the target text and the degree of emotional change. The number of phonemes in the transition interval affects the weight change trend of at least some phonemes in the preceding text and the emotional feature data of each phoneme in the transition interval of the target text. If there are fewer phonemes in the transition interval, the weight change trend is steeper, and the emotional transition is more intense; if there are more phonemes in the transition interval, the weight change trend is gentler, and the emotional transition is also gentler.

[0154] For example, the preceding text to the target text "Welcome Here" is "Long Time No See," and the emotion type of the preceding text "Long Time No See" is sadness with an emotion intensity of 0.46, while the emotion type of the target text "Welcome Here" is happiness with an emotion intensity of 0.3. The first six phonemes "h, uan1, y, ing2, l, ai2" from the phonemes "h, uan1, y, ing2, l, ai2" corresponding to the target text can be selected as the transition interval. The emotion feature data of the last phoneme in the preceding text is A, and the emotion feature data of each phoneme in the transition interval of the following text is B. Table 8 shows an example table of the emotion feature data of each phoneme in the transition interval provided in this application embodiment.

[0155] Table 8. Examples of emotional characteristic data for each phoneme in the transition interval.

[0156]

[0157] This technique allows for a smoother transition in the emotional characteristics of two sentences, resulting in a more natural and smoother emotional transition between the two sentences generated from the preceding and following texts.

[0158] Furthermore, in order to make the rhythmic transitions between sentences more natural, after step S205, the speech synthesis method provided in this application embodiment may further include the following steps:

[0159] Step S217: Based on the second prosodic information of the previous sentence text, adjust the second prosodic information of each phoneme in the target text to obtain the adjusted second prosodic information of each phoneme in the target text.

[0160] Step S207 can be achieved by following these steps:

[0161] The adjusted second prosodic information of each phoneme in the target text is copied according to the corresponding predicted speech frame number to obtain the first prosodic information at the frame level.

[0162] Specifically, step S217 can be implemented according to the following steps S218 to S219.

[0163] Step S218: Based on the fundamental frequency average value of the preceding sentence text, adjust the fundamental frequency value of at least a portion of the target text so that the difference between the fundamental frequency average value of the preceding sentence and the fundamental frequency average value of the target text is within a preset range.

[0164] Step S219: Adjust the energy values ​​of at least a portion of the target text based on the average energy value of the preceding sentence, so that the difference between the average energy value of the preceding sentence and the average energy value of the target text is within a preset range.

[0165] Understandably, to make the rhythmic transition between sentences more natural, the rhythmic features of the preceding sentence are often considered in the subsequent sentence. The rhythmic features of the subsequent sentence are adjusted based on the rhythmic features of the preceding sentence. In this embodiment, when generating speech for the target text, the rhythmic features of each phoneme in the target text can be adjusted according to the rhythmic features of the preceding sentence; alternatively, a rhythmic transition interval can be selected in the target text. This interval includes the beginning portion of the text containing a preset number of phonemes. The rhythmic transition interval can be the same as or different from the transition interval selected during the previous emotional transition. The emotional feature data corresponding to the rhythmic transition interval is then adjusted according to the emotional feature data of the preceding sentence.

[0166] Specifically, prosodic features include fundamental frequency and energy. Therefore, the average fundamental frequency of each phoneme in the previous sentence can be calculated. Based on the average fundamental frequency of each phoneme in the previous sentence, the fundamental frequency values ​​of each phoneme in the next sentence or in the prosodic transition interval of the next sentence can be adjusted so that the difference between the average fundamental frequency of the previous sentence and the average fundamental frequency of the next sentence is within a preset range, ensuring that the average fundamental frequency of the previous sentence and the average fundamental frequency of the next sentence are basically the same.

[0167] Correspondingly, the adjustment of the energy values ​​of each phoneme in the following sentence or in the prosodic transition interval of the following sentence is similar to the adjustment of the fundamental frequency value, so that the average energy of the two sentences is basically the same, which will not be elaborated here.

[0168] This technique ensures that the average fundamental frequency and average energy of the two sentences are roughly equal, meaning that the rhythmic features of the two sentences are roughly equal. The fundamental frequency represents the pitch of the sound, and the energy represents the strength of the sound. This makes the transition of the rhythm (pitch and strength of the sound) between the two sentences generated from the text more natural.

[0169] Corresponding to the speech synthesis method provided in the first embodiment of this application, the second embodiment of this application also provides a speech synthesis apparatus, such as... Figure 3 As shown, the device includes:

[0170] Acquisition unit 301 is used to acquire the target text to be converted into speech, the target emotion category, and the target emotion intensity;

[0171] The determining unit 302 is used to determine the beginning and end information of the emotion intensity range to which the target emotion intensity belongs from a plurality of preset emotion intensities for the target emotion category.

[0172] The generation unit 303 is used to generate emotion feature data corresponding to the target text based on the starting information and the ending information;

[0173] The first processing unit 304 is used to process the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data.

[0174] The prediction unit 305 is used to predict the prosody and the number of speech frames occupied by each of the phoneme vectors, so as to obtain the first prosodic information and speech content vector at the frame level.

[0175] The second processing unit 306 is used to process the speech content vector into a time-domain speech signal based on the prosody information.

[0176] Optionally, the generation unit 303 is specifically used to: normalize the target emotion intensity according to the emotion intensity range to obtain a normalized target emotion intensity; determine the normalized target emotion intensity as a first weight, and determine the difference between the value 1 and the first weight as a second weight; and perform weighted summation of the emotion feature data corresponding to the starting information and the emotion feature data corresponding to the ending information according to the second weight and the first weight, respectively, to obtain the emotion feature data corresponding to the target text.

[0177] Optionally, the prediction unit 305 is specifically used for: performing prosodic prediction on each of the phoneme vectors to obtain second prosodic information for each phoneme; predicting the number of speech frames occupied by each of the phoneme vectors to obtain the number of speech frames occupied by each phoneme; copying the second prosodic information of each phoneme according to the corresponding predicted number of speech frames to obtain first prosodic information at the frame level; and copying each phoneme vector according to the corresponding predicted number of speech frames to obtain a speech content vector at the frame level.

[0178] Optionally, the prediction unit 305 is further configured to: perform fundamental frequency prediction and energy prediction on each phoneme vector to obtain the fundamental frequency value and energy value corresponding to each phoneme; convert the fundamental frequency value and the energy value into a fundamental frequency embedding vector and an energy embedding vector, respectively; and integrate the fundamental frequency embedding vector and the energy embedding vector belonging to the same phoneme to obtain the second prosodic information of each phoneme.

[0179] Optionally, the first processing unit 304 is specifically used to: convert the target text into text embedding vectors corresponding to each phoneme; copy the emotion feature data according to the number of phonemes in the target text to obtain emotion feature data corresponding to each phoneme; for any phoneme, encode the text embedding vector corresponding to the phoneme according to the emotion feature data corresponding to the phoneme to obtain the phoneme vector with emotion features corresponding to the phoneme.

[0180] Optionally, the second processing unit 306 is specifically used to: decode the first prosodic information and the speech content vector to obtain the acoustic spectrum corresponding to the target text; and convert the acoustic spectrum into a time-domain speech signal.

[0181] Optionally, the speech synthesis apparatus provided in the second embodiment of this application further includes:

[0182] The first adjustment unit is used to adjust the emotional feature data of the transition interval in the target text according to at least part of the emotional feature data of the preceding text, so as to obtain the adjusted emotional feature data corresponding to the target text; the transition interval includes the text start part of the target text that includes a preset number of phonemes.

[0183] The first processing unit 304 is specifically used to: process the target text into phoneme vectors at the phoneme level with emotional features based on the adjusted emotional feature data.

[0184] Optionally, the first adjustment unit is specifically used to: perform a weighted summation of the emotional feature data of at least some phonemes in the preceding text with the emotional feature data of each phoneme in the target text located within the transition interval, to obtain the adjusted emotional feature data corresponding to each phoneme in the transition interval; wherein, the third weight required for the weighted summation of the emotional feature data of at least some phonemes in the preceding text gradually decreases according to the phoneme order in the transition interval, and the fourth weight required for the weighted summation of the emotional feature data of each phoneme in the target text located within the transition interval gradually increases according to the phoneme order in the transition interval.

[0185] Optionally, the speech synthesis apparatus provided in the second embodiment of this application further includes:

[0186] The first adjustment unit is used to adjust the second prosodic information of each phoneme in the target text according to the second prosodic information of the preceding text, so as to obtain the adjusted second prosodic information of each phoneme in the target text.

[0187] The prediction unit 305 is specifically used to: copy the adjusted second prosodic information of each phoneme in the target text according to the corresponding predicted speech frame number to obtain frame-level first prosodic information.

[0188] Optionally, the second adjustment unit is specifically used to: adjust the fundamental frequency value of at least a portion of the text in the target text according to the fundamental frequency average value of the preceding text, so that the difference between the fundamental frequency average value of the preceding text and the fundamental frequency average value of the target text is within a preset range; and adjust the energy value of at least a portion of the text in the target text according to the energy average value of the preceding text, so that the difference between the energy average value of the preceding text and the energy average value of the target text is within a preset range.

[0189] Optionally, the acquisition unit 301 is also used to acquire the target speaker category.

[0190] The first processing unit 304 is further configured to: convert the target speaker category into a speaker embedding vector at the phoneme level; and process the target text into phoneme vectors at the phoneme level with emotional features and speaker timbre based on the emotion feature data and the speaker embedding vector.

[0191] Optionally, the speech synthesis apparatus provided in the second embodiment of this application further includes:

[0192] The third adjustment unit is used to obtain the adjusted frame number of the target phoneme input for the target text; based on the adjusted frame number, it returns to the step of copying the second prosodic information of each phoneme according to the corresponding predicted speech frame number to obtain the first prosodic information at the frame level, until the adjusted speech signal is obtained.

[0193] Optionally, the speech synthesis apparatus provided in the second embodiment of this application further includes:

[0194] The fourth adjustment unit is used to obtain the adjusted fundamental frequency value and energy value of the target phoneme input for the target text; based on the adjusted fundamental frequency value and energy value, it returns to the step of converting the fundamental frequency value and the energy value into a fundamental frequency embedding vector and an energy embedding vector, respectively, until the adjusted speech signal is obtained.

[0195] Optionally, the speech synthesis apparatus provided in the second embodiment of this application further includes:

[0196] The fifth adjustment unit is used to acquire the adjusted emotional intensity of the target character input for the target text; the target character includes at least one target phoneme; based on the adjusted emotional intensity, it determines the adjusted emotional feature data corresponding to each target phoneme in the target character; based on the adjusted emotional feature data corresponding to each target phoneme, it returns to the step of encoding the text embedding vector corresponding to any phoneme according to the emotional feature data corresponding to the phoneme to obtain the phoneme vector with emotional features, until the adjusted speech signal is obtained.

[0197] like Figure 4The diagram shown is an architecture diagram of the speech synthesis system provided in this application embodiment. First, based on the target emotion category and target emotion intensity, the starting and ending information of the corresponding emotion intensity range are determined. The embedding layer is used to embed the target text to obtain a text embedding vector, embed the speaker category to obtain a speaker embedding vector, embed the starting information to obtain the emotion feature data corresponding to the starting information (i.e., the embedding vector of the starting information), and embed the ending information to obtain the emotion feature data corresponding to the ending information (i.e., the embedding vector of the ending information). Then, the emotion feature data of the starting information and the emotion feature data of the ending information are weighted and mixed to obtain the emotion feature data of the target emotion intensity. The encoder is used to process the emotion feature data and the speaker embedding vector. The input vector and text embedding vector are encoded to obtain phoneme vectors at the phoneme level; the prosody prediction module is used to predict the prosody of each phoneme based on the phoneme vectors; the duration prediction module is used to predict the number of speech frames occupied by each phoneme based on the phoneme vectors, and then copy the second prosody information at the phoneme level to the first prosody information at the frame level, and copy each phoneme vector at the phoneme level to the speech content vector at the frame level; the prosody adjustment module is used to perform prosody adjustment, and the duration adjustment module is used to perform frame number adjustment; the decoder is used to decode the first prosody information and the speech content vector to obtain the acoustic spectrum; the vocoder is used to convert the acoustic spectrum into a speech signal in the time domain.

[0198] Corresponding to the speech synthesis method provided in the first embodiment of this application, the third embodiment of this application also provides an electronic device for speech synthesis. For example... Figure 5 As shown, the electronic device includes: a processor 501; and a memory 502 for storing a program for a speech synthesis method. After the device is powered on and the program for the speech synthesis method is run by the processor, the following steps are performed:

[0199] Obtain the target text to be converted into speech, the target emotion category, and the target emotion intensity;

[0200] From a plurality of preset emotion intensities for the target emotion category, determine the beginning and end information of the emotion intensity range to which the target emotion intensity belongs;

[0201] Based on the starting information and the ending information, generate the emotion feature data corresponding to the target text;

[0202] Based on the emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features;

[0203] Prosodic prediction and the number of speech frames occupied by each of the phoneme vectors are performed respectively to obtain the first prosodic information and speech content vector at the frame level.

[0204] Based on the first prosody information, the speech content vector is processed into a time-domain speech signal.

[0205] Corresponding to the speech synthesis method provided in the first embodiment of this application, the fourth embodiment of this application provides a computer-readable storage medium storing a program for a speech synthesis method, which is executed by a processor to perform the following steps:

[0206] Obtain the target text to be converted into speech, the target emotion category, and the target emotion intensity;

[0207] From a plurality of preset emotion intensities for the target emotion category, determine the beginning and end information of the emotion intensity range to which the target emotion intensity belongs;

[0208] Based on the starting information and the ending information, generate the emotion feature data corresponding to the target text;

[0209] Based on the emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features;

[0210] Prosodic prediction and the number of speech frames occupied by each of the phoneme vectors are performed respectively to obtain the first prosodic information and speech content vector at the frame level.

[0211] Based on the first prosody information, the speech content vector is processed into a time-domain speech signal.

[0212] It should be noted that for a detailed description of the apparatus, electronic device and computer-readable storage medium provided in the second, third and fourth embodiments of this application, please refer to the relevant description of the first embodiment of this application, which will not be repeated here.

[0213] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0214] In a typical configuration, a node device in a blockchain includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0215] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0216] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage media, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0217] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0218] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the target text to be converted into speech, the target emotion category, and the target emotion intensity; From a plurality of preset emotion intensities for the target emotion category, determine the beginning and end information of the emotion intensity range to which the target emotion intensity belongs; Based on the starting information and the ending information, generate the emotion feature data corresponding to the target text; Based on the emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features; Prosodic prediction and the number of speech frames occupied by each of the phoneme vectors are performed respectively to obtain the first prosodic information and speech content vector at the frame level. Based on the first prosody information, the speech content vector is processed into a time-domain speech signal; When the target text has a preceding sentence, before processing the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data, the method further includes: Based on at least some of the emotional feature data of the preceding text, the emotional feature data of the transition interval in the target text is adjusted to obtain the adjusted emotional feature data corresponding to the target text; the transition interval includes the text beginning portion of the target text that includes a preset number of phonemes; The step of processing the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data includes: Based on the adjusted emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features.

2. The method according to claim 1, characterized in that, The step of generating emotion feature data corresponding to the target text based on the starting information and the ending information includes: The target emotion intensity is normalized according to the emotion intensity range to obtain the normalized target emotion intensity. The normalized target emotion intensity is determined as the first weight, and the difference between the value 1 and the first weight is determined as the second weight. The starting information is embedded to obtain the emotional feature data corresponding to the starting information, and the ending information is embedded to obtain the emotional feature data corresponding to the ending information. The emotional feature data corresponding to the starting information and the emotional feature data corresponding to the ending information are weighted and summed according to the second weight and the first weight, respectively, to obtain the emotional feature data corresponding to the target text.

3. The method according to claim 1, characterized in that, The step of predicting prosody and the number of speech frames occupied by each phoneme vector to obtain frame-level first prosodic information and speech content vector includes: Prosody prediction is performed on each of the phoneme vectors to obtain the second prosody information for each phoneme. The number of speech frames occupied by each of the phoneme vectors is predicted to obtain the number of speech frames occupied by each phoneme. The second prosodic information of each phoneme is copied according to the corresponding predicted speech frame number to obtain the first prosodic information at the frame level; Each phoneme vector is copied according to the corresponding predicted number of speech frames to obtain a frame-level speech content vector.

4. The method according to claim 3, characterized in that, The step of performing prosodic prediction on each of the phoneme vectors to obtain the second prosodic information for each phoneme includes: For each phoneme vector, fundamental frequency prediction and energy prediction are performed respectively to obtain the fundamental frequency value and energy value corresponding to each phoneme; The fundamental frequency value and the energy value are respectively converted into a fundamental frequency embedding vector and an energy embedding vector; The fundamental frequency embedding vector and the energy embedding vector belonging to the same phoneme are integrated to obtain the second prosodic information of each phoneme.

5. The method according to claim 1, characterized in that, The step of processing the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data includes: The target text is converted into a text embedding vector corresponding to each phoneme; The emotional feature data is copied according to the number of phonemes in the target text to obtain the emotional feature data corresponding to each phoneme. For any given phoneme, the text embedding vector corresponding to the phoneme is encoded based on the emotional feature data corresponding to the phoneme to obtain the phoneme vector with emotional features.

6. The method according to claim 1, characterized in that, The step of processing the speech content vector into a time-domain speech signal based on the first prosodic information includes: The first prosodic information and the speech content vector are decoded to obtain the acoustic spectrum corresponding to the target text; The acoustic spectrum is converted into a time-domain speech signal.

7. The method according to claim 1, characterized in that, The step of adjusting the emotional feature data of the transition interval in the target text based on at least a portion of the emotional feature data of the preceding sentence to obtain the adjusted emotional feature data corresponding to the target text includes: The emotional feature data of at least some phonemes in the preceding text are weighted and summed with the emotional feature data of each phoneme in the target text located in the transition interval to obtain the adjusted emotional feature data corresponding to each phoneme in the transition interval. The third weight required for weighted summation of the emotional feature data of at least some phonemes in the preceding text gradually decreases according to the phoneme order within the transition interval, and the fourth weight required for weighted summation of the emotional feature data of each phoneme in the target text located within the transition interval gradually increases according to the phoneme order within the transition interval.

8. The method according to claim 3, characterized in that, When the target text has a preceding sentence, after performing prosodic prediction on each phoneme vector to obtain the second prosodic information of each phoneme, the method further includes: Based on the second prosodic information of the preceding text, the second prosodic information of each phoneme in the target text is adjusted to obtain the adjusted second prosodic information of each phoneme in the target text; The step of copying the second prosodic information of each phoneme according to the corresponding predicted speech frame number to obtain frame-level first prosodic information includes: The adjusted second prosodic information of each phoneme in the target text is copied according to the corresponding predicted speech frame number to obtain the first prosodic information at the frame level.

9. The method according to claim 8, characterized in that, When the second prosodic information of the target text is obtained based on the fundamental frequency value and energy value corresponding to each phoneme in the target text, the step of adjusting the second prosodic information of each phoneme in the target text according to the second prosodic information of the previous sentence text to obtain the adjusted second prosodic information of each phoneme in the target text includes: Based on the fundamental frequency average value of the preceding text, adjust the fundamental frequency value of at least a portion of the target text so that the difference between the fundamental frequency average value of the preceding text and the fundamental frequency average value of the target text is within a preset range; Based on the average energy value of the preceding sentence, adjust the energy value of at least a portion of the target text so that the difference between the average energy value of the preceding sentence and the average energy value of the target text is within a preset range.

10. The method according to claim 1, characterized in that, Before determining the start and end information of the emotion intensity interval to which the target emotion intensity belongs from a plurality of preset emotion intensities for the target emotion category, the method further includes: Obtain the target speaker category; Convert the target speaker category into a phoneme-level speaker embedding vector; The step of processing the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data includes: Based on the emotional feature data and the speaker embedding vector, the target text is processed into phoneme vectors at the phoneme level, each with emotional features and speaker timbre.

11. The method according to claim 3, characterized in that, The method further includes: Obtain the adjusted frame number of the target phoneme input for the target text; Based on the adjusted frame number, return to the step of copying the second prosodic information of each phoneme according to the corresponding predicted speech frame number to obtain the first prosodic information at the frame level, until the adjusted speech signal is obtained.

12. The method according to claim 4, characterized in that, The method further includes: Obtain the adjusted fundamental frequency and energy values ​​of the target phonemes for the target text; Based on the adjusted fundamental frequency value and energy value, the process returns to the step of converting the fundamental frequency value and the energy value into a fundamental frequency embedding vector and an energy embedding vector, respectively, until the adjusted speech signal is obtained.

13. The method according to claim 5, characterized in that, The method further includes: Obtain the adjusted emotional intensity of the target word input for the target text; the target word includes at least one target phoneme; Based on the adjusted emotional intensity, determine the adjusted emotional feature data corresponding to each target phoneme in the target word; Based on the adjusted emotional feature data corresponding to each target phoneme, the step of encoding the text embedding vector corresponding to the phoneme according to the emotional feature data corresponding to the phoneme is returned, until the adjusted speech signal is obtained.

14. A speech synthesis device, characterized in that, The device includes: The acquisition unit is used to acquire the target text to be converted into speech, the target emotion category, and the target emotion intensity. The determining unit is used to determine the beginning and end information of the emotional intensity range to which the target emotional intensity belongs from a plurality of preset emotional intensities for the target emotional category. The generation unit is used to generate emotion feature data corresponding to the target text based on the starting information and the ending information. The first processing unit is used to process the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data. The prediction unit is used to predict the prosody and the number of speech frames occupied by each of the phoneme vectors, so as to obtain the first prosodic information and speech content vector at the frame level. The second processing unit is used to process the speech content vector into a time-domain speech signal based on the prosody information. When the target text has a preceding sentence, before processing the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data, the first processing unit is further configured to: Based on at least some of the emotional feature data of the preceding text, the emotional feature data of the transition interval in the target text is adjusted to obtain the adjusted emotional feature data corresponding to the target text; the transition interval includes the text beginning portion of the target text that includes a preset number of phonemes; The step of processing the target text into phoneme vectors at the phoneme level with emotional features based on the emotional feature data includes: Based on the adjusted emotional feature data, the target text is processed into phoneme vectors at the phoneme level with emotional features.

15. An electronic device, characterized in that, include: processor; as well as A memory for storing a data processing program, which, when the electronic device is powered on and runs through the processor, executes the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The system contains a data processing program that is executed by a processor to perform the method as described in any one of claims 1-13.

Images