Voice generation method and device, medium and electronic equipment
By constructing the emotional space of emotional roulette theory, decomposing and combining basic emotions and neutral emotions, the problem of TTS system showing single emotions in the emotional synthesis task is solved, and the voice of multiple emotions is generated is realized, reducing the difficulty of data acquisition and Improves expression richness.
Patent Information
- Application Number
- CN202310604158.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-26
AI Technical Summary
The existing TTS system has only targeted several common emotions in the emotional synthesis task, and cannot effectively express multiple emotions that humans can experience, resulting in the single expression of generated voice emotions and the inability to generate emotions that have not been seen in the training stage.
Based on the emotional roulette theory, emotional space is constructed, and by decomposing the target emotional identification as the basis of emotion and neutral emotions, using global representations to combine to generate rich emotional representations, thereby generating voices containing multiple emotions.
It realizes the generation of emotions that have not been seen in the training stage, reduces the difficulty of obtaining emotional data, enriches the emotional expression of voice, and only requires basic emotions and partial compound emotional data, improving the flexibility and richness of emotional synthesis .
Smart Images

Figure CN119028315A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of speech generation technology, and in particular, to a speech generation method, device, medium and electronic device. Background Art
[0002] TTS (Text To Speech) systems can automatically generate corresponding speech based on a piece of text. The emotion synthesis task can help generate speech with more emotional expression. However, it is rare to see a complete emotion theory system in existing TTS systems. Most TTS systems only model emotions for a few common emotions, such as sadness, anger, happiness, etc. Studies have shown that humans can experience more than 30,000 different emotions. If only a few emotions are modeled to perform the emotion synthesis task of the TTS system, the generated speech will be relatively simple in terms of emotional expression. Summary of the invention
[0003] This section is provided to introduce the concepts in a brief form, which will be described in detail in the detailed implementation section below. This section is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to be used to limit the scope of the technical solution claimed for protection.
[0004] In a first aspect, the present disclosure provides a speech generation method, comprising:
[0005] Obtaining target text and first target sentiment identifier;
[0006] Determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion;
[0007] Determine, according to a preset emotion space, a global representation corresponding to each of the at least one basic emotion, wherein the emotion space includes a global representation corresponding to each basic emotion;
[0008] Obtaining a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion;
[0009] Generate corresponding speech according to the target text and the first emotion representation.
[0010] In a second aspect, the present disclosure provides a speech generating device, comprising:
[0011] An information acquisition module, used to acquire a target text and a first target emotion identifier;
[0012] A basic emotion determination module, configured to determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion;
[0013] A global representation determination module, used to determine the global representation corresponding to the at least one basic emotion according to a preset emotion space, wherein the emotion space includes the global representation corresponding to each basic emotion;
[0014] A first emotion representation determination module, configured to obtain a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion;
[0015] A speech generation module is used to generate corresponding speech according to the target text and the first emotion representation.
[0016] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the speech generation method provided in the first aspect of the present disclosure.
[0017] In a fourth aspect, the present disclosure provides an electronic device, including:
[0018] a storage device having at least one computer program stored thereon;
[0019] At least one processing device is used to execute the at least one computer program in the storage device to implement the steps of the speech generation method provided in the first aspect of the present disclosure.
[0020] In the above technical solution, an emotional space is pre-constructed, which includes multiple defined basic emotions and the global representation corresponding to each basic emotion. In the emotion synthesis task, according to the input first target emotion identifier, the first target emotion identifier is decomposed into at least one corresponding basic emotion, and the final first emotion representation is obtained according to the global representation corresponding to the at least one basic emotion. In this way, through various combinations between different basic emotions, emotional representations of different emotions can be obtained, and then speech containing various emotions can be generated, which is richer in emotional expression. Moreover, this technical solution can generate emotions that have not been seen in the training stage. For example, in the training stage, some emotional data of compound emotion categories are not involved in the training, but only after modeling the global representation of various basic emotions in the emotional space, some compound emotions that have not been seen in the training stage can be combined through the combination of basic emotions. Therefore, only some emotional data of basic emotions and a small amount of emotional data of compound emotions are needed in the training stage, which reduces the difficulty of obtaining emotional data.
[0021] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:
[0023] Figure 1 A flow chart of a speech generation method provided by an exemplary embodiment is shown;
[0024] Figure 2 A schematic diagram showing a preset emotional space;
[0025] Figure 3 A flowchart of constructing an emotional space provided by an exemplary embodiment is shown;
[0026] Figure 4 A schematic diagram of obtaining a second emotion representation in an exemplary embodiment is shown;
[0027] Figure 5 A flowchart showing an implementation of step S306 provided by an exemplary embodiment;
[0028] Figure 6 A block diagram of a speech generating device provided by an exemplary embodiment is shown;
[0029] Figure 7 A block diagram of an electronic device provided by an exemplary embodiment is shown. DETAILED DESCRIPTION
[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0034] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0035] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0036] All actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.
[0037] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0038] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0039] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0040] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0041] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0042] Existing TTS systems generally use emotion labels to model emotions in speech data. For example, pre-record speech containing different emotions as emotion data of different emotion categories. These emotion categories are usually several common emotions, such as sadness, anger, happiness, etc. The emotion is modeled according to the emotion data corresponding to each emotion category, and the emotion label of the corresponding emotion category is obtained. In this way, in the emotion synthesis task, the speech of the corresponding emotion can be synthesized according to the emotion label of the required emotion category. This emotion modeling method generally only models several common emotions. If you want to model more kinds of emotions, you need to obtain emotion data of more emotion categories. On the one hand, it takes a lot of cost, and on the other hand, it is very difficult to obtain rich emotion data, which makes the speech generated by the TTS system relatively simple in terms of emotion expression. In addition, the emotion modeling method of the prior art can only generate emotions seen in the training stage. For example, if the emotion data of the expected emotion category is not included in the training stage, then when the speech is generated later, it will not be possible to generate speech containing the expected emotion. Therefore, the existing emotion modeling method has great limitations.
[0043] In view of the problems existing in the prior art, the embodiment of the present disclosure provides a speech generation method based on the emotion wheel. To facilitate understanding of the speech generation method, the embodiment of the present disclosure first describes the emotion wheel before introducing the method.
[0044] The emotion roulette is a theory for defining emotions in the field of psychology. Eight basic emotions are defined in the emotion roulette, namely happiness, trust, fear, surprise, sadness, disgust, anger and expectation. The remaining emotions can be defined by the combination of these basic emotions. For example, liking can be composed of the two basic emotions of happiness and trust. Emotions composed of at least two basic emotions are called compound emotions. For more definitions of emotion combinations, please refer to Table 1 below. It should be noted that the basic emotions shown in Table 1 and the definition of emotion combinations are only used as an example and do not constitute a limitation on the basic emotions and their combinations in the embodiments of the present disclosure. In actual implementation, the basic emotions and the combinations of basic emotions can be flexibly set. For example, some compound emotions can also be composed of more than three basic emotions.
[0045] hapiness trust fear surprise sad disgust angry expect hapiness ecstasy favorite Guilt Happy Bittersweet Pathological pride optimism trust respect yield curious Sentimental Love and Hate Dominate hope fear Thriller Awe despair ashamed freeze anxiety surprise Surprise be opposed to No indignation Puzzled sad Grief regret envy pessimistic disgust hatred Contempt Cynicism angry fury Aggressive expect alert
[0046] Table 1
[0047] The embodiment of the present disclosure models an emotional space based on the theory of the emotional wheel, and the emotional space includes a plurality of predefined basic emotions and a global representation corresponding to each basic emotion, and the global representation is understood as an emotional representation of the basic emotion. On this basis, the speech generation method provided by the embodiment of the present disclosure is described in detail below.
[0048] Figure 1 A flowchart of a speech generation method provided by an exemplary embodiment is shown. Figure 1 , the speech generation method comprises the following steps:
[0049] S101, obtaining a target text and a first target emotion identifier.
[0050] In this step, a target text for generating speech and a first target emotion identifier for representing the target emotion are obtained. In some embodiments, the target emotion represented by the first target emotion identifier may belong to basic emotions, such as happiness, trust, anger, etc. in Table 1, or may belong to a composite emotion composed of multiple basic emotions, such as love, guilt, etc. in Table 1.
[0051] S102, determining at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion.
[0052] In some embodiments, the first target emotion identifier can be represented by a combination of the IDs of the corresponding basic emotions. For example, for the eight basic emotions mentioned above, their IDs are: happiness: 0; trust: 1; fear: 2; surprise: 3; sadness: 4; disgust: 5; anger: 6; expectation: 7. Assuming that the target emotion represented by the first target emotion identifier belongs to a basic emotion, such as happiness, the first target emotion identifier is 0. Assuming that the target emotion represented by the first target emotion identifier belongs to a composite emotion composed of at least two basic emotions, such as liking, since liking is composed of two basic emotions, happiness and trust, the first target emotion identifier is [0,1].
[0053] Furthermore, the emotion roulette lacks a definition of emotion intensity. In some embodiments, in order to express the emotion category while also being able to express the required emotion intensity, the disclosed embodiment expands the basic emotions of the emotion roulette by adding an additional neutral basic emotion, thereby expanding the basic emotions from eight to nine. Neutral basic emotions can be considered as meaningless emotions, and the various basic emotions defined above are called non-neutral basic emotions. All emotions are considered to be a combination of at least one non-neutral basic emotion and one neutral basic emotion. For example, happiness can be considered to be a combination of happiness and neutral basic emotions, and liking can be considered to be a combination of happiness, trust, and neutral basic emotions. Therefore, the at least one basic emotion corresponding to the first target emotion identifier includes at least one non-neutral basic emotion and one neutral basic emotion.
[0054] It should be noted that if the target emotion is composed of a non-neutral basic emotion and a neutral basic emotion, the target emotion is still considered to be a basic emotion. If the target emotion is composed of at least two non-neutral basic emotions and one neutral basic emotion, the target emotion is considered to be a compound emotion.
[0055] It should also be noted that the neutral basic emotion is used to control the emotional intensity of the target emotion. The emotional intensity can be controlled by the proportion of the neutral basic emotion. The higher the proportion of the neutral basic emotion, the weaker the emotional intensity of the target emotion. The lower the proportion of the neutral basic emotion, the stronger the emotional intensity of the target emotion.
[0056] S103, determining global representations corresponding to the at least one basic emotion according to a preset emotional space, wherein the emotional space includes a global representation corresponding to each basic emotion.
[0057] Before this step, it is necessary to construct an emotion space, which includes multiple defined basic emotions and a global representation corresponding to each basic emotion. The global representation can be understood as an emotion representation of the basic emotion. In some embodiments, the global representation can be in the form of a vector. Figure 2 A schematic diagram showing a preset emotional space.
[0058] S104: Obtain a first emotion representation according to the global representation corresponding to the at least one basic emotion.
[0059] In some embodiments, when obtaining the target text and the first target emotion identifier, the first weight can be obtained at the same time. The first weight includes the weight values corresponding to the at least one basic emotion corresponding to the first target emotion identifier, and these weight values are used to indicate the proportional distribution between the at least one basic emotion. It is worth noting that the proportional distribution is not fixed, but can be flexibly determined according to the emotion to be generated. In this way, even for the same emotion category, you only need to change the proportional distribution between its basic emotions to obtain a variety of different emotions, so that the target emotion obtained is richer. In this step, the first emotion representation can be obtained by performing a multiplication operation based on the global representations corresponding to the first weight and the at least one basic emotion.
[0060] Exemplarily, assuming that the target emotion represented by the first target emotion identifier is liking, the first weight obtained includes a weight value of happiness, a weight value of trust, and a weight value of neutrality, and a multiplication operation is performed based on the first weight and the global representations corresponding to happiness, trust, and neutrality, respectively.
[0061] Exemplarily, the multiplication operation can be a matmul calculation, which represents matrix multiplication, which means multiplying the elements at the same position in the two matrices and then adding them together. Specifically, the first weight can be one of the matrices, the first weight includes the weight values corresponding to the n basic emotions used to constitute the target emotion, and the other matrix includes the global representations corresponding to the n basic emotions. The two matrices have the same dimension n. Therefore, according to the matmul calculation, the weight value corresponding to the same basic emotion and the global representation are multiplied, and then the results are added to obtain the first emotion representation.
[0062] In the above process, the intensity of liking can be controlled by controlling the weight value of the neutral basic emotion. The larger the weight value of the neutral basic emotion, the weaker the intensity of liking, and the smaller the weight value of the neutral basic emotion, the stronger the intensity of liking. Thus, by setting the neutral basic emotion, the intensity of emotion can be controlled, so that the generated speech can express richer and more layered emotions.
[0063] In some embodiments, the first weight may be determined based on user input. For example, for the target emotion of liking, three basic emotions (happiness, trust, and neutrality) used to constitute liking and their weight value control slides may be displayed on the page, and the user may freely adjust the weight value control slides of the three basic emotions and click a confirmation button to obtain the first weight.
[0064] S105, generating corresponding speech according to the target text and the first emotion representation.
[0065] In the above technical solution, an emotional space is constructed in the TTS system based on the theory of the emotional roulette. The emotional space includes multiple defined basic emotions and the global representation corresponding to each basic emotion. In the emotional synthesis task, according to the input first target emotional identifier, the first target emotional identifier is decomposed into at least one corresponding basic emotion, and the final first emotional representation is obtained according to the global representation corresponding to the at least one basic emotion. In this way, through various combinations between different basic emotions, emotional representations of different emotions can be obtained, and then speech containing various different emotions can be generated, which is richer in emotional expression. Moreover, this technical solution can generate emotions that have not been seen in the training stage. For example, in the training stage, some emotional data of compound emotional categories are not involved in the training, but only after modeling the global representation of various basic emotions in the emotional space, some compound emotions that have not been seen in the training stage can be combined through the combination of basic emotions. Therefore, only some emotional data of basic emotions and a small amount of emotional data of compound emotions are needed in the training stage, which reduces the difficulty of obtaining emotional data.
[0066] In addition, the technical solution can adjust the weight value corresponding to each basic emotion in the first weight, that is, adjust the proportion distribution between each basic emotion. For example, for a compound emotion such as liking, 0.4 proportion of happiness plus 0.6 proportion of trust, 0.8 proportion of happiness plus 0.2 proportion of trust, these two proportions express different levels of liking, that is, the expected various emotions can be generated by adjusting the first weight. In addition, the technical solution can adjust the weight value of neutral basic emotions to achieve control of emotional intensity.
[0067] In some embodiments, step S105 includes: determining a first phoneme sequence corresponding to the target text, obtaining a corresponding first phoneme representation based on the first phoneme sequence, obtaining a first target representation based on the first phoneme representation and the first emotion representation, and generating corresponding speech based on the first target representation.
[0068] In the above steps, the target text is processed into a phoneme sequence to obtain a first phoneme sequence, and then the first phoneme sequence is encoded to obtain a corresponding first phoneme representation. It should be noted that the step of obtaining the first phoneme representation according to the target text can be performed after obtaining the target text, and the execution order between the first phoneme representation and steps S102 to S104 is not limited.
[0069] Then, the first phoneme representation and the first emotion representation can be concatenated to obtain a first target representation, and then decoding processing is performed according to the first target representation to obtain a corresponding predicted Mel spectrum, so as to generate corresponding speech according to the predicted Mel spectrum.
[0070] In some embodiments, the method further includes: obtaining a first timbre identifier representing the timbre. In the process of generating the corresponding speech according to the first target representation, the speech of the corresponding timbre can be generated according to the first timbre identifier and the first target representation, such as performing decoding processing according to the first timbre identifier and the first target representation to obtain the corresponding predicted Mel spectrum, and then generating the speech of the corresponding timbre according to the predicted Mel spectrum.
[0071] It is understandable that in order to construct the emotion space, a certain amount of training data is needed to train the emotion encoder, and the training data may include training data for each basic emotion and training data for some compound emotions. The process of constructing the emotion space is described below.
[0072] Figure 3 A flowchart of constructing an emotional space provided by an exemplary embodiment is shown. Please refer to Figure 3 , the emotional space is constructed through the following steps:
[0073] S301, determining training data and an emotion encoder to be trained, wherein the training data includes a second phoneme sequence corresponding to a training text, a reference mel spectrum, and a second target emotion identifier.
[0074] In this step, some training data need to be obtained, and each training data corresponds to a training text. For a training text, it is necessary to obtain speech data recorded with a specific emotion, process the training text into a second phoneme sequence, process the speech data into a reference mel spectrum, and determine the second target emotion identifier corresponding to the specific emotion, so as to obtain training data corresponding to the training text.
[0075] S302, input the reference mel spectrum to the emotion encoder to be trained, and the emotion encoder outputs a set of reference weights and a set of global representations based on the emotion information in the reference mel spectrum, wherein the set of reference weights includes weight values corresponding to each basic emotion, and the set of global representations includes global representations corresponding to each basic emotion.
[0076] In some embodiments, the sentiment encoder may use a GST (Global Style Token) model.
[0077] Exemplarily, following the example in the previous text, nine basic emotions are predefined. The reference mel spectrum is input into the emotion encoder to be trained. The emotion encoder outputs a set of reference weights and a set of global representations based on the emotion information in the reference mel spectrum. The set of reference weights includes weight values corresponding to each of the nine defined basic emotions. These weight values are used to indicate the proportion of each basic emotion to the target emotion constituting the second target emotion identification representation. The proportion distribution is automatically learned and output by the emotion encoder based on the emotion information in the reference mel spectrum. The set of global representations includes a global representation of each of the nine defined basic emotions.
[0078] S303: Determine at least one corresponding basic emotion according to the second target emotion identifier, and obtain a second emotion representation according to the global representation and weight value corresponding to the at least one basic emotion.
[0079] Among them, the process of determining at least one corresponding basic emotion according to the second target emotion identifier is consistent with the aforementioned step S102 and will not be repeated here.
[0080] In some embodiments, after determining the corresponding at least one basic emotion, a multiplication operation may be performed based on the global representation corresponding to the at least one basic emotion and the weight value to obtain a second emotion representation.
[0081] In some embodiments, the weight values of other basic emotions in the output group of reference weights may be masked, i.e., masked, based on the at least one basic emotion to obtain a group of masked weights, and then the group of masked weights may be normalized, such as Softmax, to obtain a group of second weights, and then the second weights may be multiplied with the output group of global representations to obtain a second emotion representation. For example, the group of second weights and the group of global representations may be matmuled.
[0082] Figure 4 A schematic diagram showing a second emotion representation in an exemplary embodiment is shown, please refer to Figure 4 , assuming that the target emotion represented by the second target emotion identifier is liking, and its corresponding at least one basic emotion includes happiness, trust and neutrality, then the weight values of the other basic emotions except happiness, trust and neutrality in the output group of reference weights are masked to obtain the masked weights, and then the masked weights are Softmax processed to obtain the second weights, and finally matmul calculation is performed based on the second weights and the output group of global representations to obtain the second emotion representation.
[0083] S304, obtaining a corresponding second phoneme representation according to the second phoneme sequence, and obtaining a second target representation according to the second phoneme representation and the second emotion representation.
[0084] In this step, the second phoneme sequence is encoded to obtain a corresponding second phoneme representation. The second phoneme representation and the second emotion representation are concatenated to obtain a second target representation.
[0085] It should be noted that the step of obtaining the corresponding second phoneme representation according to the second phoneme sequence can be performed after obtaining the training data, and the execution order of the step of obtaining the corresponding second phoneme representation according to the second phoneme sequence is not limited.
[0086] S305 , performing decoding processing according to the second target representation to obtain a corresponding predicted mel spectrum.
[0087] S306, calculating a loss based on the predicted mel spectrum, and training the emotion encoder based on the loss.
[0088] Among them, the predicted Mel spectrum and the reference Mel spectrum represent the predicted value and the true value respectively, so the loss can be calculated based on the predicted Mel spectrum and the reference Mel spectrum, and the emotion encoder can be trained based on the loss.
[0089] S307, after the training is completed, construct an emotion space according to the global representation finally corresponding to each basic emotion.
[0090] It is easy to understand that after the training is completed, the global representation corresponding to each basic emotion is constructed as follows Figure 2 The emotional space shown.
[0091] It is easy to understand that the combination ratio between basic emotions, such as the ratio between happiness and trust in liking, is not defined in the emotion wheel. Therefore, the embodiment of the present disclosure automatically learns the component ratio between basic emotions and the corresponding global representation through the emotion encoder in the TTS system.
[0092] Furthermore, in the process of training the emotion encoder in the above step S306, in order to help improve the stability of the emotion space, a loss calculation strategy is proposed, that is, first determine whether the target emotion represented by the second target emotion identifier belongs to the basic emotion or the compound emotion. If the represented target emotion belongs to the basic emotion, the emotion classification loss can be calculated to help the training of the emotion encoder.
[0093] Figure 5 FIG. 1 is a flowchart showing an implementation of step S306 provided by an exemplary embodiment. Figure 5 , step S306 includes:
[0094] S501, calculating the prediction loss according to the predicted Mel spectrum and the reference Mel spectrum.
[0095] S502, when the target emotion represented by the second target emotion identifier belongs to the basic emotion, the emotion classification loss is calculated according to the group of reference weights output by the emotion encoder and the corresponding basic emotion.
[0096] Exemplarily, if the target emotion represented by the second target emotion identifier is happiness, it is determined to be a basic emotion, and then the emotion classification loss is calculated using a cross entropy loss function based on the set of reference weights output by the emotion encoder and the basic emotion of happiness.
[0097] S503, training the sentiment encoder according to the prediction loss and the sentiment classification loss.
[0098] In some embodiments, when the target emotion represented by the second target emotion identifier belongs to a complex emotion, the emotion encoder may be trained only based on the prediction loss.
[0099] According to the above scheme, the emotion encoder is iteratively trained so that the emotion encoder continuously learns according to the emotion information in the input reference mel spectrum, so that the global representation of each basic emotion output becomes more and more accurate until the training is completed.
[0100] Based on the same technical concept, Figure 6 A block diagram of a speech generating device provided by an exemplary embodiment is shown. Figure 6 , the speech generating device 600 comprises:
[0101] Information acquisition module 601, used to acquire target text and a first target emotion identifier;
[0102] A basic emotion determination module 602 is used to determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion;
[0103] A global representation determination module 603 is used to determine the global representation corresponding to the at least one basic emotion according to a preset emotion space, wherein the emotion space includes the global representation corresponding to each basic emotion;
[0104] A first emotion representation determination module 604, configured to obtain a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion;
[0105] The speech generation module 605 is used to generate corresponding speech according to the target text and the first emotion representation.
[0106] In some embodiments, the speech generation module 605 is used to determine a first phoneme sequence corresponding to the target text, obtain a corresponding first phoneme representation based on the first phoneme sequence, and obtain a first target representation based on the first phoneme representation and the first emotion representation, and generate corresponding speech based on the first target representation.
[0107] In some embodiments, the speech generation device 600 further includes a timbre identification acquisition module for acquiring a first timbre identification representing the timbre. The speech generation module 605 is configured to generate a speech of corresponding timbre according to the first timbre identification and the first target representation.
[0108] In some embodiments, the speech generation device 600 further includes a first weight acquisition module for acquiring a first weight, wherein the first weight includes a weight value corresponding to the at least one basic emotion. The first emotion representation determination module 604 is used to perform a multiplication operation based on the first weight and the global representation corresponding to the at least one basic emotion to obtain the first emotion representation.
[0109] In some embodiments, the at least one basic emotion includes at least one non-neutral basic emotion and one neutral basic emotion; the neutral basic emotion is used to control the emotional intensity of the target emotion, and the weight value of the neutral basic emotion is negatively correlated with the emotional intensity of the target emotion.
[0110] In some embodiments, the emotional space is constructed by the following modules:
[0111] A training data determination module, used to determine training data and an emotion encoder to be trained, wherein the training data includes a second phoneme sequence corresponding to a training text, a reference mel spectrum, and a second target emotion identifier;
[0112] An emotion encoding module, used for inputting the reference Mel spectrum into the emotion encoder to be trained, wherein the emotion encoder outputs a set of reference weights and a set of global representations according to the emotion information in the reference Mel spectrum, wherein the set of reference weights includes a weight value corresponding to each basic emotion, and the set of global representations includes a global representation corresponding to each basic emotion;
[0113] A second emotion representation determination module, configured to determine at least one corresponding basic emotion according to the second target emotion identifier, and obtain a second emotion representation according to a global representation and a weight value corresponding to the at least one basic emotion;
[0114] a target representation determination module, configured to obtain a corresponding second phoneme representation according to the second phoneme sequence, and obtain a second target representation according to the second phoneme representation and the second emotion representation;
[0115] A mel spectrum prediction module, configured to perform decoding processing according to the second target representation to obtain a corresponding predicted mel spectrum;
[0116] An encoder training module, used for calculating a loss according to the predicted Mel spectrum, and training the emotion encoder according to the loss;
[0117] The emotion space construction module is used to construct the emotion space according to the global representation finally corresponding to each basic emotion after the training is completed.
[0118] It is worth noting that the above-mentioned training data determination module, emotion encoding module, second emotion representation determination module, target representation determination module, mel spectrum prediction module, encoder training module and emotion space construction module may be configured in another electronic device, that is, the various modules for constructing the emotion space and the various modules for generating speech according to the target text may be configured in different electronic devices.
[0119] In some embodiments, the second emotion representation determination module includes:
[0120] A masking processing module, configured to perform masking processing on the weight values of other basic emotions in the outputted set of reference weights according to the at least one basic emotion, so as to obtain a set of masking weights;
[0121] A normalization processing module, used for normalizing the masking weight to obtain a second weight;
[0122] An operation module is used to perform a multiplication operation according to the second weight and the output group of global representations to obtain the second emotion representation.
[0123] In some embodiments, the encoder training module includes:
[0124] A prediction loss calculation module, used to calculate the prediction loss according to the predicted Mel spectrum and the reference Mel spectrum;
[0125] A classification loss calculation module, used for calculating the emotion classification loss according to the output group of reference weights and the corresponding basic emotion when the target emotion represented by the second target emotion identifier belongs to the basic emotion;
[0126] A training module is used to train the emotion encoder according to the prediction loss and the emotion classification loss.
[0127] Regarding the speech generating device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0128] Reference below Figure 7, which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0129] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0130] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0131] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0132] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0133] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0134] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the target text and the first target emotion identifier; determines at least one corresponding basic emotion according to the first target emotion identifier, and the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion; determines the global representation corresponding to the at least one basic emotion according to a preset emotion space, and the emotion space includes the global representation corresponding to each basic emotion; obtains the first emotion representation according to the global representation corresponding to the at least one basic emotion; generates the corresponding speech according to the target text and the first emotion representation.
[0135] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0136] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0137] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the information acquisition module may also be described as a "module for acquiring the target text and the first target emotion identifier".
[0138] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0139] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0140] According to one or more embodiments of the present disclosure, Example 1 provides a speech generation method, including:
[0141] Obtaining target text and first target sentiment identifier;
[0142] Determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion;
[0143] Determine, according to a preset emotion space, a global representation corresponding to each of the at least one basic emotion, wherein the emotion space includes a global representation corresponding to each basic emotion;
[0144] Obtaining a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion;
[0145] Generate corresponding speech according to the target text and the first emotion representation.
[0146] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein generating corresponding speech according to the target text and the first emotion representation includes:
[0147] Determine a first phoneme sequence corresponding to the target text, and obtain a corresponding first phoneme representation according to the first phoneme sequence;
[0148] A first target representation is obtained according to the first phoneme representation and the first emotion representation, and corresponding speech is generated according to the first target representation.
[0149] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein the method further includes:
[0150] Obtaining a first timbre identifier representing a timbre;
[0151] Generating corresponding speech according to the first target representation includes:
[0152] Generate speech of corresponding timbre according to the first timbre identifier and the first target representation.
[0153] According to one or more embodiments of the present disclosure, Example 4 provides a method of any one of Examples 1 to 3, wherein the method further includes:
[0154] Obtaining a first weight, wherein the first weight includes a weight value corresponding to each of the at least one basic emotion;
[0155] The obtaining of a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion comprises:
[0156] A multiplication operation is performed on the first weight and the global representations corresponding to the at least one basic emotion to obtain the first emotion representation.
[0157] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, wherein the at least one basic emotion includes at least one non-neutral basic emotion and one neutral basic emotion; the neutral basic emotion is used to control the emotion intensity of the target emotion, and the weight value of the neutral basic emotion is negatively correlated with the emotion intensity of the target emotion.
[0158] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 4, wherein the emotion space is constructed by the following steps:
[0159] Determine training data and an emotion encoder to be trained, wherein the training data includes a second phoneme sequence corresponding to a training text, a reference mel spectrum, and a second target emotion identifier;
[0160] The reference Mel spectrum is input into the emotion encoder to be trained, and the emotion encoder outputs a set of reference weights and a set of global representations according to the emotion information in the reference Mel spectrum, wherein the set of reference weights includes a weight value corresponding to each basic emotion, and the set of global representations includes a global representation corresponding to each basic emotion;
[0161] Determine at least one corresponding basic emotion according to the second target emotion identifier, and obtain a second emotion representation according to the global representation and weight value corresponding to the at least one basic emotion;
[0162] Obtain a corresponding second phoneme representation according to the second phoneme sequence, and obtain a second target representation according to the second phoneme representation and the second emotion representation;
[0163] Performing decoding processing according to the second target representation to obtain a corresponding predicted mel spectrum;
[0164] Calculating a loss according to the predicted mel spectrum, and training the emotion encoder according to the loss;
[0165] After the training is completed, the emotion space is constructed according to the global representation finally corresponding to each basic emotion.
[0166] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 6, wherein obtaining a second emotion representation according to the global representation and weight value corresponding to the at least one basic emotion includes:
[0167] According to the at least one basic emotion, masking is performed on the weight values of other basic emotions in the outputted set of reference weights to obtain a set of masking weights;
[0168] Normalizing the masking weight to obtain a second weight;
[0169] A multiplication operation is performed on the second weight and the output group of global representations to obtain the second emotion representation.
[0170] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 6, wherein the loss is calculated according to the predicted mel spectrum, and the emotion encoder is trained according to the loss, including:
[0171] Calculating a prediction loss according to the predicted Mel spectrum and the reference Mel spectrum;
[0172] When the target emotion represented by the second target emotion identifier belongs to the basic emotion, calculating the emotion classification loss according to the output group of reference weights and the corresponding basic emotion;
[0173] The sentiment encoder is trained according to the prediction loss and the sentiment classification loss.
[0174] According to one or more embodiments of the present disclosure, Example 9 provides a speech generating device, including:
[0175] An information acquisition module, used to acquire a target text and a first target emotion identifier;
[0176] A basic emotion determination module, configured to determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion;
[0177] A global representation determination module, used to determine the global representation corresponding to the at least one basic emotion according to a preset emotion space, wherein the emotion space includes the global representation corresponding to each basic emotion;
[0178] A first emotion representation determination module, configured to obtain a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion;
[0179] A speech generation module is used to generate corresponding speech according to the target text and the first emotion representation.
[0180] According to one or more embodiments of the present disclosure, Example 10 provides the apparatus of Example 9, wherein the speech generation module is used to determine a first phoneme sequence corresponding to the target text, obtain a corresponding first phoneme representation based on the first phoneme sequence, and obtain a first target representation based on the first phoneme representation and the first emotion representation, and generate corresponding speech based on the first target representation.
[0181] According to one or more embodiments of the present disclosure, Example 11 provides the device of Example 10, which also includes a timbre identification acquisition module for acquiring a first timbre identification representing the timbre; and a speech generation module for generating a speech of corresponding timbre according to the first timbre identification and the first target representation.
[0182] According to one or more embodiments of the present disclosure, Example 12 provides an apparatus of any one of Examples 9 to 11, wherein the apparatus further includes a first weight acquisition module for acquiring a first weight, wherein the first weight includes a weight value corresponding to each of the at least one basic emotions; and a first emotion representation determination module is used to obtain the first emotion representation by performing a multiplication operation on the first weight and a global representation corresponding to each of the at least one basic emotion.
[0183] According to one or more embodiments of the present disclosure, Example 13 provides the device of Example 12, wherein the at least one basic emotion includes at least one non-neutral basic emotion and one neutral basic emotion; the neutral basic emotion is used to control the emotion intensity of the target emotion, and the weight value of the neutral basic emotion is negatively correlated with the emotion intensity of the target emotion.
[0184] According to one or more embodiments of the present disclosure, Example 14 provides the apparatus of Example 12, wherein the emotion space is constructed by the following modules:
[0185] A training data determination module, used to determine training data and an emotion encoder to be trained, wherein the training data includes a second phoneme sequence corresponding to a training text, a reference mel spectrum, and a second target emotion identifier;
[0186] An emotion encoding module, used for inputting the reference Mel spectrum into the emotion encoder to be trained, wherein the emotion encoder outputs a set of reference weights and a set of global representations according to the emotion information in the reference Mel spectrum, wherein the set of reference weights includes a weight value corresponding to each basic emotion, and the set of global representations includes a global representation corresponding to each basic emotion;
[0187] A second emotion representation determination module, configured to determine at least one corresponding basic emotion according to the second target emotion identifier, and obtain a second emotion representation according to a global representation and a weight value corresponding to the at least one basic emotion;
[0188] a target representation determination module, configured to obtain a corresponding second phoneme representation according to the second phoneme sequence, and obtain a second target representation according to the second phoneme representation and the second emotion representation;
[0189] A mel spectrum prediction module, configured to perform decoding processing according to the second target representation to obtain a corresponding predicted mel spectrum;
[0190] An encoder training module, used for calculating a loss according to the predicted Mel spectrum, and training the emotion encoder according to the loss;
[0191] The emotion space construction module is used to construct the emotion space according to the global representation finally corresponding to each basic emotion after the training is completed.
[0192] According to one or more embodiments of the present disclosure, Example 15 provides the apparatus of Example 14, wherein the second emotion representation determination module includes:
[0193] A masking processing module, configured to perform masking processing on the weight values of other basic emotions in the outputted set of reference weights according to the at least one basic emotion, so as to obtain a set of masking weights;
[0194] A normalization processing module, used for normalizing the masking weight to obtain a second weight;
[0195] An operation module is used to perform a multiplication operation according to the second weight and the output group of global representations to obtain the second emotion representation.
[0196] According to one or more embodiments of the present disclosure, Example 16 provides the apparatus of Example 14, wherein the encoder training module includes:
[0197] A prediction loss calculation module, used to calculate the prediction loss according to the predicted Mel spectrum and the reference Mel spectrum;
[0198] A classification loss calculation module, used for calculating the emotion classification loss according to the output group of reference weights and the corresponding basic emotion when the target emotion represented by the second target emotion identifier belongs to the basic emotion;
[0199] A training module is used to train the emotion encoder according to the prediction loss and the emotion classification loss.
[0200] According to one or more embodiments of the present disclosure, Example 17 provides a computer-readable medium having a computer program stored thereon, which implements the method of any one of Examples 1 to 8 when executed by a processing device.
[0201] According to one or more embodiments of the present disclosure, Example 18 provides an electronic device, including:
[0202] a storage device having at least one computer program stored thereon;
[0203] At least one processing device is used to execute the at least one computer program in the storage device to implement the method of any one of Examples 1 to 8.
[0204] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0205] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0206] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A method for generating speech, It is characterized in that include: Obtaining target text and first target sentiment identifier; Determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion; Determine, according to a preset emotion space, a global representation corresponding to each of the at least one basic emotion, wherein the emotion space includes a global representation corresponding to each basic emotion; Obtaining a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion; Generate corresponding speech according to the target text and the first emotion representation.
2. The method according to claim 1, It is characterized in that The generating corresponding speech according to the target text and the first emotion representation includes: Determine a first phoneme sequence corresponding to the target text, and obtain a corresponding first phoneme representation according to the first phoneme sequence; A first target representation is obtained according to the first phoneme representation and the first emotion representation, and corresponding speech is generated according to the first target representation.
3. The method according to claim 2, It is characterized in that The method further comprises: Obtaining a first timbre identifier representing a timbre; Generating corresponding speech according to the first target representation includes: Generate speech of corresponding timbre according to the first timbre identifier and the first target representation.
4. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: Obtaining a first weight, wherein the first weight includes a weight value corresponding to each of the at least one basic emotion; The obtaining of a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion comprises: A multiplication operation is performed on the first weight and the global representations corresponding to the at least one basic emotion to obtain the first emotion representation.
5. The method according to claim 4, It is characterized in that The at least one basic emotion includes at least one non-neutral basic emotion and one neutral basic emotion; the neutral basic emotion is used to control the emotional intensity of the target emotion, and the weight value of the neutral basic emotion is negatively correlated with the emotional intensity of the target emotion.
6. The method according to claim 4, It is characterized in that The emotional space is constructed through the following steps: Determine training data and an emotion encoder to be trained, wherein the training data includes a second phoneme sequence corresponding to a training text, a reference mel spectrum, and a second target emotion identifier; The reference Mel spectrum is input into the emotion encoder to be trained, and the emotion encoder outputs a set of reference weights and a set of global representations according to the emotion information in the reference Mel spectrum, wherein the set of reference weights includes a weight value corresponding to each basic emotion, and the set of global representations includes a global representation corresponding to each basic emotion; Determine at least one corresponding basic emotion according to the second target emotion identifier, and obtain a second emotion representation according to the global representation and weight value corresponding to the at least one basic emotion; Obtain a corresponding second phoneme representation according to the second phoneme sequence, and obtain a second target representation according to the second phoneme representation and the second emotion representation; Performing decoding processing according to the second target representation to obtain a corresponding predicted mel spectrum; Calculating a loss according to the predicted mel spectrum, and training the emotion encoder according to the loss; After the training is completed, the emotion space is constructed according to the global representation finally corresponding to each basic emotion.
7. The method according to claim 6, It is characterized in that The step of obtaining a second emotion representation according to the global representation and the weight value corresponding to the at least one basic emotion comprises: According to the at least one basic emotion, masking the weight values of other basic emotions in the outputted set of reference weights to obtain a set of masking weights; Normalizing the masking weight to obtain a second weight; A multiplication operation is performed on the second weight and the output group of global representations to obtain the second emotion representation.
8. The method according to claim 6, It is characterized in that The calculating the loss according to the predicted Mel spectrum and training the emotion encoder according to the loss includes: Calculating a prediction loss according to the predicted Mel spectrum and the reference Mel spectrum; When the target emotion represented by the second target emotion identifier belongs to the basic emotion, calculating the emotion classification loss according to the output group of reference weights and the corresponding basic emotion; The sentiment encoder is trained according to the prediction loss and the sentiment classification loss.
9. A speech generating device, It is characterized in that include: An information acquisition module, used to acquire a target text and a first target emotion identifier; A basic emotion determination module, configured to determine at least one corresponding basic emotion according to the first target emotion identifier, wherein the target emotion represented by the first target emotion identifier is composed of the at least one basic emotion; A global representation determination module, used to determine the global representation corresponding to the at least one basic emotion according to a preset emotion space, wherein the emotion space includes the global representation corresponding to each basic emotion; A first emotion representation determination module, configured to obtain a first emotion representation according to the global representations respectively corresponding to the at least one basic emotion; A speech generation module is used to generate corresponding speech according to the target text and the first emotion representation.
10. A computer readable medium having a computer program stored thereon, It is characterized in that When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 8 are implemented.
11. An electronic device, It is characterized in that include: a storage device having at least one computer program stored thereon; At least one processing device, configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.
