Speech synthesis model training method and device, emotion speech synthesis method and device and medium

Abstract

The invention discloses a speech synthesis model training method, an emotional speech synthesis method and device and a medium, relates to the technical field of audio processing, can be applied to financial science and technology and medical health business scenarios, and comprises the steps: obtaining sample speech data containing text information, emotional intensity information and speech information, and corresponding reference speech; determining a text token, an emotion intensity token and a voice token corresponding to the sample voice data; performing multi-modal feature fusion on the three tokens by using a language model (LLM) to obtain semantic representation; performing acoustic feature decoding and continuous voice signal synthesis processing on the semantic representation by using a dual-coder-decoder to obtain a sample synthesis voice; and constructing a reward function under multiple dimensions, and based on the sample synthesis voice and the reference voice, performing joint iterative training on the voice synthesis model by adopting a mode of combining supervised fine tuning and group relative strategy optimization to obtain a trained voice synthesis model.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a speech synthesis model training method, an emotional speech synthesis method, a device, and a medium. Background Technology

[0002] Text-to-Speech (TTS), a key supporting technology in the field of human-computer interaction, has undergone years of development and has completed a core leap from ensuring intelligibility to pursuing naturalness. Currently, the industry's research focus is shifting entirely to enhancing expressiveness. Among these technologies, Emotional TTS, due to its ability to directly enhance the realism and emotional resonance of interactive scenarios, is widely used in fields such as fintech and healthcare, becoming a core driving force for upgrading TTS technology from a functional tool to an experiential carrier. As the demand for emotional expression in various interactive scenarios becomes increasingly sophisticated, how to achieve more nuanced and controllable emotional expression has become the core goal of current emotional TTS technology research and development.

[0003] Current emotional speech synthesis technologies are mainly limited to discrete emotion category control schemes. These schemes use pre-set fixed emotion labels such as happiness, sadness, anger, and surprise, and train the model by associating these labels with acoustic features such as pitch, energy, and duration. This allows the model to generate speech with a corresponding style based on the input emotion category. However, this approach lacks the ability to continuously model the intensity and subtle emotional changes of emotions, resulting in synthesized speech that is coarse and lacks continuity in emotional expression. It fails to effectively address the core need for fine-grained emotion control and cannot improve the realism and diversity of expression while maintaining the quality of speech synthesis. Summary of the Invention

[0004] In view of this, this application provides a speech synthesis model training method, an emotional speech synthesis method, an apparatus and a medium, which can improve the realism and diversity of expression while maintaining the quality of speech synthesis.

[0005] According to a first aspect of this application, a method for training a speech synthesis model is provided, the speech synthesis model including a language model LLM and a dual encoder-decoder, the method comprising: Acquire sample speech data and the corresponding reference speech, wherein the sample speech data includes at least text information, emotion intensity information and speech information, and the reference speech is the expected synthesized speech corresponding to the sample speech data; The text information, the emotion intensity information, and the voice information are discretized and encoded respectively to obtain the corresponding text token, emotion intensity token, and voice token; The text token, the emotion intensity token, and the voice token are input into the language model LLM for multimodal feature fusion to obtain a semantic representation that integrates multimodal information; The semantic representation is input into the dual encoder-decoder, and the semantic representation is subjected to acoustic feature decoding and continuous speech signal synthesis processing to obtain sample synthesized speech; The synthesized speech samples are compared with the baseline speech in multiple dimensions. A reward function is constructed based on three dimensions: emotion classification, global emotion intensity, and local emphasis control. The speech synthesis model is jointly iteratively trained by combining supervised fine-tuning and group relative strategy optimization until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression indicators. The speech synthesis model is then judged to have completed training.

[0006] According to a second aspect of this application, an emotional speech synthesis method is provided, comprising: Determine the target speech data to be processed by speech synthesis, wherein the target speech data includes at least target text information, target emotion intensity information, and target speech information; The target text information, the target emotion intensity information, and the target speech information are discretized and encoded respectively to obtain the corresponding target text token, target emotion intensity token, and target speech token; The target text token, the target emotion intensity token, and the target speech token are input into the speech synthesis model that has been trained as described in the first aspect to obtain the emotional synthesized speech corresponding to the target speech data.

[0007] According to a third aspect of this application, a speech synthesis model training apparatus is provided, the speech synthesis model including a language model LLM and a dual encoder-decoder, the method comprising: The acquisition module is used to acquire sample speech data and the corresponding reference speech. The sample speech data includes at least text information, emotion intensity information and speech information. The reference speech is the expected synthesized speech corresponding to the sample speech data. The first processing module is used to perform discretization encoding on the text information, the emotion intensity information and the voice information respectively to obtain the corresponding text token, emotion intensity token and voice token; The fusion module is used to input the text token, the emotion intensity token, and the voice token into the language model LLM to perform multimodal feature fusion and obtain a semantic representation with fused multimodal information; The first synthesis module is used to input the semantic representation into the dual encoder-decoder, perform acoustic feature decoding and continuous speech signal synthesis processing on the semantic representation, and obtain sample synthesized speech; The training module is used to compare the sample synthesized speech with the benchmark speech in multiple dimensions, construct a reward function based on three dimensions: emotion classification, global emotion intensity, and local emphasis control, and perform joint iterative training on the speech synthesis model by combining supervised fine-tuning and group relative strategy optimization until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression index, at which point the speech synthesis model is judged to have completed training.

[0008] According to a fourth aspect of this application, an emotional speech synthesis apparatus is provided, comprising: The determination module is used to determine the target speech data to be processed by speech synthesis, wherein the target speech data includes at least target text information, target emotion intensity information and target speech information; The second processing module is used to perform discretization encoding on the target text information, the target emotion intensity information and the target speech information respectively to obtain the corresponding target text token, target emotion intensity token and target speech token; The second synthesis module is used to input the target text token, the target emotion intensity token, and the target speech token into the trained speech synthesis model described in the first aspect, so as to obtain the emotional synthesized speech corresponding to the target speech data.

[0009] According to a fifth aspect of this application, a storage medium is provided having a computer program stored thereon, the computer program causing a computer to perform the methods as described in the first or second aspect.

[0010] According to a sixth aspect of this application, an electronic device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, the processor being configured to invoke and run the computer program stored in the storage medium to perform the methods as described in the first or second aspect.

[0011] The technical solution provided by this invention introduces multimodal sample inputs containing text information, emotional intensity information, and speech information, and performs discretization encoding on the three types of information to generate corresponding tokens. Combined with the Language Modeling (LLM), it achieves deep fusion of multimodal features, which can not only break through the limitations of traditional fixed emotional labels, but also achieve fine-grained emotional intensity adjustment based on emotional intensity information, and ensure the naturalness of synthesized speech through prosodic reference of speech information. Then, it completes acoustic feature decoding and speech synthesis through dual encoders and decoders, and combines a reward function based on three dimensions of emotion classification, global emotional intensity, and local emphasis control, as well as a joint iterative training method that combines supervised fine-tuning and group relative strategy optimization. This can significantly enhance the realism and diversity of emotional expression in synthesized speech while steadily improving the basic sound quality of speech synthesis. Ultimately, it achieves synergistic optimization of fine-grained emotional control and high-quality speech synthesis, meeting the core requirements of delicate and natural emotional speech expression in practical applications.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a speech synthesis model training method provided in an embodiment of this application is shown. Figure 2 A flowchart illustrating a speech synthesis model training method according to another embodiment of this application is shown; Figure 3 A flowchart illustrating an emotional speech synthesis method provided in an embodiment of this application is shown; Figure 4 This paper shows a schematic diagram of the structure of a speech synthesis model training device provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of an emotional speech synthesis device provided in an embodiment of this application is shown. Detailed Implementation

[0014] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0015] Current emotional speech synthesis technologies are mainly limited to discrete emotion category control schemes. These schemes use pre-set fixed emotion labels such as happiness, sadness, anger, and surprise, and train the model by associating these labels with acoustic features such as pitch, energy, and duration. This allows the model to generate speech with a corresponding style based on the input emotion category. However, this approach lacks the ability to continuously model the intensity and subtle emotional changes of emotions, resulting in synthesized speech that is coarse and lacks continuity in emotional expression. It fails to effectively address the core need for fine-grained emotion control and cannot improve the realism and diversity of expression while maintaining the quality of speech synthesis.

[0016] To address the aforementioned technical problems, embodiments of the present invention provide a method for training a speech synthesis model. This speech synthesis model may include a Large Language Model (LLM) and a dual encoder-decoder, such as... Figure 1 As shown, the method includes: Step 110: Obtain sample speech data and the corresponding reference speech. The sample speech data includes at least text information, emotion intensity information and speech information. The reference speech is the expected synthesized speech corresponding to the sample speech data.

[0017] The sample speech data consists of a multimodal input data set used to train the speech synthesis model, containing at least text information, emotion intensity information, and speech information. Text information (such as the specific phrase "The sunset is beautiful today") carries semantic content and determines the linguistic content of the synthesized speech, forming the semantic basis of speech synthesis. Emotion intensity information represents the emotional attributes of the sample speech data, including emotion categories (such as joy, sadness, and anger) and corresponding intensity values ​​(such as 0-10 quantization values ​​based on the VAD emotion space), serving as a key basis for achieving fine-grained emotion control. Speech information comprises speech signals with prosodic features (such as a segment of speech recorded by a real person), covering acoustic features such as fundamental frequency (pitch), energy (volume), and duration (pronunciation duration), providing a reference for the prosodic naturalness of the synthesized speech. The benchmark speech corresponds one-to-one with the sample speech data, representing the expected synthesized speech that the model should generate after training on the sample speech data. It must meet preset emotional expression requirements, sound quality standards, and prosodic logic, serving as a benchmark for comparing the quality and emotional expression of the synthesized speech.

[0018] By acquiring multimodal sample speech data containing text, emotional intensity, and speech, the model can be provided with three key types of information: semantics, emotion, and prosody. This avoids problems such as the disconnect between emotional expression and semantics, and prosodic distortion caused by traditional single-modal input. At the same time, by clearly defining the baseline speech as the expected synthesized speech, a clear reference standard can be provided for the multi-dimensional comparison of subsequent sample synthesized speech with the baseline speech, ensuring that the model training has a clear optimization direction. This lays a data foundation for improving the fine-grained control of emotion, the naturalness of sound quality, and the realism of emotional expression of synthesized speech through reward functions and joint training.

[0019] Step 120: Discretize and encode the text information, sentiment intensity information, and speech information respectively to obtain the corresponding text token, sentiment intensity token, and speech token.

[0020] Discretization encoding refers to the process of converting different types of input information (text, sentiment intensity, and speech) from unstructured / continuous form into structured / discrete coded units (tokens). The core is to preserve the core features of the information through standardized encoding while adapting to the model's input format requirements. Text tokens are discrete units formed after text information is discretized, containing identifiers of sentiment-focused words and the start and end markers of the text sequence, used to convey semantic information with sentiment emphasis to the model. Sentiment intensity tokens are discrete units formed after sentiment intensity information is discretized, generated by binding sentiment intensity levels with unique codes, used to convey precise sentiment intensity information to the model. Speech tokens are discrete units formed after speech information is discretized, containing discretized acoustic feature encoding, semantic identifiers (associated text semantic units), and type identifiers (annotated speech modalities), used to convey prosodic information with semantic association to the model.

[0021] In this embodiment of the disclosure, to enable the three types of heterogeneous information—text, emotional intensity, and speech—to adapt to the multimodal feature fusion requirements of a Language Model (LLM), each type of information needs to be transformed into structured discrete units. Specifically, for text information, semantic parsing can be used to distinguish between emotional focus words (such as "excited" and "sad" expressing core emotions) and ordinary text words carrying basic semantics. Sub-word segmentation and encoding are performed on both types of words, and a unique identifier is added to the encoding result of the emotional focus words. At the same time, sequence start and end identifiers are added to clarify the semantic boundaries of the text, forming a text token. For emotional intensity information, based on a preset VAD (pleasure-arousal-dominance) emotional space level division rule, continuous emotional intensity values ​​can be matched to the corresponding discrete level, and a unique encoding unit pre-configured for that level can be retrieved to form an emotional intensity token. For speech information, its acoustic feature sequence containing fundamental frequency, energy, and duration can be extracted first, and then the continuous acoustic features can be converted into an encoded form through a discretization algorithm (such as vector quantization). Semantic identifiers associated with text semantic units and type identifiers annotating modal attributes are added to form a speech token.

[0022] By discretizing and encoding text, emotion intensity, and speech information separately, unstructured text semantics, continuous emotion intensity values, and continuous acoustic feature sequences can be uniformly transformed into a discrete token format that LLM can efficiently parse, avoiding the problems of low fusion efficiency or information loss caused by heterogeneous data types. On the other hand, text tokens retain the emotional focus in the semantics through emotion focus identification; emotion intensity tokens achieve fine-grained emotional information discretization through VAD level association, which can break through the limitations of traditional fixed emotion labels; speech tokens ensure the binding of prosodic features with text semantics and clear modal boundaries through semantic and type identification. The three work together to ensure that the three key types of information—semantic focus, emotion intensity, and prosodic features—are fully preserved during subsequent multimodal feature fusion. This effectively avoids the problems of emotion and semantics being disconnected and prosody and text being misaligned in traditional encoding methods, providing key support for the model to generate synthetic speech with accurate emotional expression and natural prosody.

[0023] Step 130: Input the text token, emotion intensity token and speech token into the language model LLM in the speech synthesis model to perform multimodal feature fusion and obtain a semantic representation that integrates multimodal information.

[0024] Multimodal feature fusion refers to the process of integrating token features from different modalities—text (semantic dimension), emotional intensity (emotional dimension), and speech (prosodic dimension)—into a unified feature representation through LLM's attention mechanism and feature interaction. The core is to achieve complementarity and synergy between different modal information. Semantic representation refers to the unified feature vector output by LLM through multimodal feature fusion. It integrates three key types of information: the semantic content of the text, the emotional attributes of the emotional intensity, and the prosodic logic of the speech. It is the core intermediate feature connecting multimodal input and subsequent acoustic feature decoding.

[0025] In this embodiment of the disclosure, after inputting the text token carrying semantic information and emotional focus identifier, the emotional intensity token associated with the VAD emotional space level, and the speech token containing prosodic feature encoding and semantic / type identifier into the language model LLM of the speech synthesis model, the LLM first distinguishes the attributes of the three types of tokens (text semantics, emotional intensity, and speech prosody) through the built-in modality recognition mechanism, and then establishes the association between modalities with the help of the cross-attention mechanism: on the one hand, the emotional focus word identifier in the text token is deeply bound to the quantitative emotional information of the emotional intensity token (such as the encoding corresponding to "highly pleasant") to ensure that the emotional expression matches the semantic focus; on the other hand, combined with the semantic identifier of the speech token (associated text unit), the prosodic features carried by the speech token are aligned with the semantic order of the text token and the emotional logic of the emotional intensity token. Through multi-layer feature interaction and integration, the information barriers between different modalities are eliminated, and finally a unified feature vector that simultaneously contains text semantic integrity, accurate emotional intensity, and adapted prosodic logic is output, that is, a semantic representation that integrates multi-modal information.

[0026] By fusing multimodal features of text, sentiment intensity, and speech tokens using a Language Modeling (LLM), the problem of isolated modal information in traditional multimodal processing can be effectively solved. This avoids the deviation of emotional expression from semantics caused by the disconnect between text semantics and sentiment intensity (e.g., synthesizing neutral words with a strong emotional tone), and also prevents prosody from not conforming to emotional logic due to misalignment between speech rhythm and text / emotion (e.g., a pleasant emotion corresponding to a low-pitched rhythm). This ensures that the output semantic representation simultaneously possesses the synergy of semantic, sentiment, and prosody information. Furthermore, this fusion method allows subsequent dual encoders and decoders to directly generate acoustic parameters based on the integrated information during acoustic feature decoding, eliminating the need for additional handling of intermodal adaptation issues. This provides crucial intermediate feature support for improving the semantic accuracy, emotional nuance, and prosodic naturalness of synthesized speech.

[0027] Step 140: Input the semantic representation into the dual encoder-decoder in the speech synthesis model, perform acoustic feature decoding and continuous speech signal synthesis processing on the semantic representation, and obtain sample synthesized speech.

[0028] Among them, acoustic feature decoding refers to the process of converting semantic representation into a continuous sequence of acoustic parameters containing fundamental frequency (pitch), energy (volume), and duration (speech duration), and it is necessary to ensure that the parameters match the emotional and prosodic information in the semantic representation; continuous speech signal synthesis processing refers to the process of reconstructing the decoded continuous acoustic parameter sequence into an initial speech signal through a vocoder, and then processing it through noise reduction, clarity optimization, etc., to finally generate audible speech; sample synthesized speech is the speech result corresponding to the input semantic representation after the dual codecs are processed.

[0029] In this embodiment of the disclosure, after the semantic representation integrating text semantics, emotional intensity, and prosodic logic is input into the dual encoder-decoder of the speech synthesis model, the dual encoder-decoder first performs dimensional transformation on the semantic representation through the built-in feature mapping layer, adjusting it into an intermediate feature vector that is adapted to acoustic feature decoding. Subsequently, the acoustic decoding layer of the decoder, based on the intermediate feature vector, combines the emotional intensity (such as the parameter tendency corresponding to "moderate pleasure") and prosodic logic (such as the duration allocation corresponding to "light and fast rhythm") implied in the semantic representation, and outputs a continuous acoustic parameter sequence containing fundamental frequency, energy, and duration (for example, "pleasant emotion" corresponds to a higher mean fundamental frequency, and "light and fast rhythm" corresponds to a shorter pronunciation duration). Next, the decoder calls the vocoder to reconstruct the continuous speech signal from the continuous acoustic parameter sequence, converting the abstract acoustic parameters into initial sample synthesized speech. Finally, the initial speech is denoised (to eliminate background noise) and its clarity optimized (to enhance syllable recognition) through the built-in sound quality optimization module of the decoder, ultimately obtaining a sample synthesized speech that meets the basic sound quality standards and whose emotional and prosodic attributes fit the semantic representation.

[0030] By using dual codecs to perform acoustic feature decoding and continuous speech signal synthesis on semantic representations, the multimodal information integrated in semantic representations can be accurately converted into concrete speech signals. On the one hand, the acoustic feature decoding stage can generate matching fundamental frequency, energy, and duration parameters based on the emotional intensity and prosodic logic in the semantic representation, avoiding the problem of emotional and prosodic disconnect in traditional synthesis schemes. On the other hand, the vocoder reconstruction and sound quality optimization in the continuous speech signal synthesis process can effectively improve the naturalness and clarity of the synthesized speech samples, solving the problem of the stiffness of parameterized speech.

[0031] Step 150: Compare the synthesized speech samples with the baseline speech in multiple dimensions. Construct a reward function based on three dimensions: emotion classification, global emotion intensity, and local emphasis control. Use a combination of supervised fine-tuning and group relative strategy optimization to jointly iteratively train the speech synthesis model until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression indicators. Then, determine that the speech synthesis model training is complete.

[0032] The reward function is a model optimization objective function constructed based on multi-dimensional comparison results. It integrates emotion classification reward, global emotion intensity reverse reward, and local emphasis control reward to guide the model parameters to update in a direction that fits the benchmark speech. Supervised fine-tuning uses the acoustic feature difference between the sample synthesized speech and the benchmark speech as the loss function to conduct supervised training on the language model LLM and dual encoder-decoder, enabling the model to initially grasp the basic mapping relationship between multimodal input and speech output. Group Relative Policy Optimization (GRPO) generates multiple sets of candidate synthesized speech based on the reward function on the basis of the parameters of supervised fine-tuning. By calculating the difference in reward value between each set of candidate speech and the benchmark speech, the model policy parameters are updated with the group relative advantage, focusing on optimizing the subtlety and controllability of emotion expression. The emotion expression index is a preset standard for evaluating the model's emotion expression effect (such as emotion classification accuracy ≥90%, global emotion intensity difference ≤0.1, and local emphasis matching degree ≥0.85), which, together with the loss function, serves as the basis for judging the termination of training.

[0033] In this embodiment of the disclosure, the sample synthesized speech output by the dual encoder-decoder can be compared with a preset benchmark speech in multiple dimensions: In the emotion classification dimension, a pre-trained emotion classification model is used to identify the emotion categories of the two (such as "pleasure" and "sadness") and calculate the category matching accuracy; in the global emotion intensity dimension, the emotion intensity values ​​of the two are quantified using acoustic feature tools (such as pleasure rating based on VAD space) and the intensity difference is calculated; in the local emphasis control dimension, the repetition features (fundamental frequency peak, energy peak, duration percentage) of the corresponding words are extracted by combining the emotion focus word localization results in the text information and the matching score is calculated; a reward function is constructed based on the comparison results of these three dimensions, converting the category matching accuracy into an emotion classification reward item and the intensity difference into a global emotion intensity. The reverse reward term and the repetition feature matching score are transformed into a local emphasis control reward term, and the three reward terms are integrated into a total reward function. Then, joint iterative training is initiated: In the first stage, supervised fine-tuning is performed, using the difference in acoustic features between the sample synthesized speech and the benchmark speech as the loss, to adjust the model parameters so that the basic synthesis effect is close to the benchmark speech. In the second stage, based on the parameters of supervised fine-tuning, multiple sets of candidate synthesized speech are generated through group relative strategy optimization. The reward value of each set is calculated according to the reward function, and the model strategy is updated with the group relative advantage to enhance the subtlety of emotional expression. The above process is iterated repeatedly, and the loss function value and multi-dimensional emotional expression index of the model are continuously monitored. When the loss function is less than the preset convergence threshold or the multi-dimensional reward meets the preset emotional expression index, the iteration stops, and the speech synthesis model training is judged to be complete.

[0034] By comparing synthesized speech samples with benchmark speech across multiple dimensions, the problem of one-sided or distorted emotional expression caused by traditional single-dimensional evaluation can be avoided. The reward function constructed based on multi-dimensional results can accurately guide the model to optimize multiple core dimensions of emotional expression simultaneously, rather than focusing on only a single indicator. The joint iterative training combining supervised fine-tuning and group relative strategy optimization can not only ensure the basic synthesis capability of the model through supervised fine-tuning, but also overcome the limitations of single supervised training in optimizing emotional subtlety through group relative strategy optimization, achieving a synergy between stable basic performance and refined emotional expression. Finally, through the dual termination conditions of loss function and emotional expression index, the model is ensured to achieve the preset emotional expression standard while possessing high-quality speech synthesis capability. This effectively solves the core problems of coarse-grained emotional control and the difficulty in balancing sound quality and emotion in existing technologies, enabling the trained speech synthesis model to generate speech that is semantically accurate, has clear sound quality, and expresses emotions subtly while adhering to real language logic.

[0035] In summary, the speech synthesis model training method provided by this invention introduces multimodal sample inputs containing text information, emotional intensity information, and speech information, and performs discretization encoding on the three types of information to generate corresponding tokens. Combined with the Language Model LLM, it achieves deep fusion of multimodal features, which can not only break through the limitations of traditional fixed emotional labels, but also achieve fine-grained emotional intensity adjustment based on emotional intensity information, and ensure the naturalness of synthesized speech through prosodic reference of speech information. Then, the acoustic feature decoding and speech synthesis are completed by dual encoders and decoders. Combined with a reward function constructed based on three dimensions of emotional classification, global emotional intensity, and local emphasis control, and a joint iterative training method that combines supervised fine-tuning and group relative strategy optimization, it can significantly enhance the realism and diversity of emotional expression in synthesized speech while steadily improving the basic sound quality of speech synthesis. Finally, it achieves synergistic optimization of fine-grained emotional control and high-quality speech synthesis, meeting the core requirements of delicate and natural emotional speech expression in practical applications.

[0036] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the implementation of this embodiment, this embodiment also provides another method for training a speech synthesis model, such as... Figure 2 As shown, the method includes: Step 210: Obtain sample speech data and the corresponding reference speech. The sample speech data includes at least text information, emotion intensity information and speech information. The reference speech is the expected synthesized speech corresponding to the sample speech data.

[0037] For the specific implementation steps of the embodiments disclosed herein, please refer to the relevant description in step 110 of the embodiment, which will not be repeated here.

[0038] Step 220: Discretize and encode the text information, sentiment intensity information, and speech information respectively to obtain the corresponding text token, sentiment intensity token, and speech token.

[0039] Among them, the emotional intensity information includes at least the emotional intensity value.

[0040] For embodiments of this disclosure, step 220 may include the following steps: Step 220-1: Perform semantic parsing on the text information, identify the emotional focus words and ordinary text words, perform word segmentation and encoding on the emotional focus words and ordinary text words respectively, generate text tokens with emotional focus identifiers, and add sequence start identifiers and sequence end identifiers to the text tokens to mark the boundaries of the text tokens.

[0041] In this embodiment of the disclosure, the input text information can first be semantically parsed. Through language logic analysis and sentiment identification, emotional focus words expressing core emotions (such as "super beautiful") and ordinary text words carrying basic semantics (such as "the scenery of this trip") can be distinguished. Then, sub-word segmentation encoding is performed on the two types of words respectively: emotional focus words are split into semantically related sub-word units (such as "super beautiful" split into "super" and "beautiful"), and ordinary text words are also split into sub-word units (such as "the scenery of this trip" split into "this time", "trip", "of", and "scenery"). Then, a unique token ID is assigned to each sub-word unit (such as "super" corresponding to ID "123" and "beautiful" corresponding to ID "456"). Subsequently, a unique emotional focus identifier is added to all sub-word token IDs of emotional focus words (such as marking "123" and "456" as "123_Emo_Focus" and "456_Emo_Focus"). According to the original semantic order of the text information (such as "this time → trip → of → scenery → super → beautiful"), the tokens with emotional focus identifiers are... The IDs are combined sequentially with the token IDs of ordinary text words to form a continuous token sequence. Finally, a sequence start identifier (such as "[CLS]") and a sequence end identifier (such as "[SEP]") are added to the beginning and end of the token sequence to clarify the semantic boundaries of the text tokens, and finally a complete text token containing emotional focus distinctions and boundary markers is generated.

[0042] Accordingly, when performing word segmentation and encoding on emotional focus words and ordinary text words respectively to generate text tokens with emotional focus identifiers, the implementation steps may include: performing sub-word segmentation and encoding on emotional focus words and ordinary text words respectively, and converting the segmented sub-words into token IDs; after adding emotional focus identifiers to the token IDs corresponding to emotional focus words, combining the token IDs with emotional focus identifiers with the token IDs corresponding to ordinary text words according to the original semantic order of the text to form a text token containing emotional focus identifiers.

[0043] Among them, emotion focus words are words in the text that directly express core emotions or emotional tendencies (such as "super happy", "sad", "excited"), which are the core objects of local emphasis control in emotional speech synthesis and need to highlight their emotional attributes through exclusive identifiers; ordinary text words are words in the text that carry basic semantics and have no obvious emotional tendencies (such as "today", "go to the park", "eat lunch"), which are mainly used to construct the semantic logic of the text and do not require additional emotion identifiers; sub-word segmentation encoding is a specific implementation of word segmentation encoding, which splits the complete word into semantically related sub-word units (such as "super happy" split into "super" and "happy"), and then assigns a unique token ID to each sub-word, taking into account both encoding universality and semantic integrity; the token ID is the discrete numerical code corresponding to the sub-word unit, which is the basic data format for the model to process text information and transmits the semantic information of the sub-word in numerical form; the emotion focus identifier is an exclusive mark added to the token ID corresponding to the emotion focus word (such as "Emo_Focus"), which is used to clearly inform the model of the token. IDs carry core emotional information and should be given special attention in subsequent multimodal fusion and speech synthesis. Sequence start identifiers / sequence end identifiers are special markers added to the beginning and end of text tokens (such as "[CLS]" and "[SEP]") to clarify the semantic boundaries of text tokens, prevent the model from confusing tokens from different texts, and ensure the integrity of semantic parsing.

[0044] By accurately distinguishing between emotional focus words and ordinary text words through semantic parsing, the model can avoid misjudging non-emotional words as emotional cores, ensuring the relevance of emotional expression. Adding exclusive identifiers to emotional focus words allows the subsequent language model LLM to quickly locate the emotional core during multimodal fusion, accurately binding emotional intensity information with semantic emphasis, and avoiding the disconnect between emotion and semantics.

[0045] Step 220-2: Determine the emotional intensity level corresponding to the emotional intensity value, and obtain the emotional intensity token corresponding to the emotional intensity level. Each emotional intensity level is pre-configured with a unique emotional intensity token based on the VAD emotional space.

[0046] Among them, the emotional intensity value is a numerical value that quantifies the degree of emotional expression. It is generated based on the three core dimensions of the VAD emotional space, such as "pleasure 7 points, arousal 6 points, and dominance 5 points", which is the basic data for describing the subtlety of emotions. The emotional intensity level is a category obtained by discretizing the continuous emotional intensity values. Each level corresponds to a specific range of emotional intensity values. The VAD emotional space is a multi-dimensional emotional description framework. It constructs an emotional coordinate system through three orthogonal dimensions of pleasure (Valence), arousal (Arousal), and dominance (Dominance). It can comprehensively and accurately quantify various emotions and is the core basis for classifying emotional intensity levels.

[0047] In this embodiment of the disclosure, the rules for classifying emotional intensity levels can be predefined based on the three dimensions of the VAD emotional space (pleasure, arousal, and dominance): the continuous numerical range of each dimension (e.g., 0-10 points) is divided into sub-intervals such as "low, medium, and high", and then different emotional intensity levels are formed by combining the sub-intervals (e.g., "high pleasure - medium arousal - medium dominance", "medium pleasure - low arousal - high dominance"). A unique emotional intensity token is configured for each level, and a preset mapping table of emotional intensity levels and emotional intensity tokens is established. When a specific emotional intensity value is obtained, the combination of the sub-intervals to which the intensity value belongs in the three dimensions of VAD is first determined according to the preset classification rules, and then the corresponding emotional intensity level is determined. Finally, the preset mapping table is queried to retrieve the unique emotional intensity token corresponding to the emotional intensity level, completing the conversion from continuous emotional intensity value to discrete encoded token.

[0048] By dividing emotional intensity levels based on the VAD emotional space, we can fully capture the subtle differences in emotions by utilizing the multi-dimensional characteristics of VAD, avoiding the one-sidedness of emotional description caused by traditional single-dimensional division. Discretizing continuous emotional intensity values ​​into levels and configuring unique tokens can solve the problems of unstable optimization and high computational cost when the model processes continuous numerical values. At the same time, the unique token can ensure the encoding consistency of the same emotional intensity level and avoid the model from having ambiguous understanding of the same emotional intensity.

[0049] Step 220-3: Convert the speech information into an acoustic feature sequence containing fundamental frequency, energy, and duration. Discretize and encode the acoustic feature sequence to generate a speech token corresponding to the speech information, and add semantic and type identifiers.

[0050] The acoustic feature sequence is a continuous set of features extracted from speech information to characterize the physical properties of speech. Its core components include fundamental frequency (Pitch, which measures pitch, e.g., a higher fundamental frequency in pleasant speech), energy (Energy, which measures volume, e.g., stronger energy in emphatic speech), and duration (Duration, which measures the length of pronunciation, e.g., longer duration for emotional focus words), arranged in temporal order to form a sequence. Semantic identifiers are tags added to speech tokens that associate them with corresponding text semantic units (e.g., tags indicating the 2nd-5th characters of the text 'Today I'm very happy'), used to establish a binding relationship between speech prosody and text semantics, avoiding misalignment of sound and meaning. Type identifiers are tags added to speech tokens that label their modal attributes (e.g., "Speech_Modal"), used to allow the model to distinguish between speech tokens and text tokens, as well as emotion intensity tokens, avoiding modal confusion during multimodal input.

[0051] In this embodiment of the disclosure, the speech signal can be segmented by a signal processing tool according to a fixed time window (e.g., 10ms / frame). The fundamental frequency value (e.g., extracted by autocorrelation) and energy value (e.g., calculated by short-time energy algorithm) of each frame are calculated frame by frame. At the same time, the duration of each word pronunciation segment is counted (e.g., the start and end times of the word are determined by endpoint detection and the time difference is calculated). The frame-by-frame values ​​of fundamental frequency and energy and the duration statistics are arranged in chronological order to form a continuous acoustic feature sequence. Then, a discretization coding algorithm (e.g., vector quantization VQ) is used to process the continuous sequence. Specifically, the continuous values ​​of fundamental frequency and energy can be mapped to a preset discrete codebook, and the duration is discretized into corresponding codes according to fixed intervals. Finally, a coding sequence containing only discrete values ​​is generated. Finally, an identifier is added to the coding sequence: the semantic identifier needs to be associated with the text unit corresponding to the speech information (e.g., the speech “Today I am very happy” corresponds to the text “Today I am very happy”, so it is marked “Semantic_Link: Today I am very happy”), and the type identifier is uniformly marked as a speech modality-specific marker (e.g., “Modal_Type: Speech”). The two are integrated with the discrete coding sequence to form a complete speech token.

[0052] By extracting three core acoustic features—fundamental frequency, energy, and duration—the key prosodic attributes of speech information can be fully preserved, providing an accurate reference for the naturalness of synthesized speech and avoiding the prosodic distortion caused by relying solely on a single feature in traditional methods. Discretization coding converts continuous acoustic features into model-friendly discrete units, which can reduce the complexity of the model in processing continuous features, reduce feature redundancy, and improve the efficiency of multimodal fusion.

[0053] Step 230: Input the text token, sentiment intensity token and voice token into the language model LLM to perform multimodal feature fusion and obtain a semantic representation that integrates multimodal information.

[0054] For the specific implementation process of the embodiments disclosed herein, please refer to the relevant description in step 130 of the embodiment, which will not be repeated here.

[0055] Step 240: Input the semantic representation into the dual encoder-decoder. The feature mapping layer of the dual encoder-decoder performs dimensional transformation on the semantic representation to generate an intermediate feature vector adapted for acoustic feature decoding.

[0056] Among them, the core processing action of dimensionality conversion to feature mapping layer refers to the process of adjusting the original dimension of semantic representation (such as 1024 dimensions) to perfectly match the input dimension of acoustic decoding layer of dual encoder-decoder (such as 512 dimensions). This is a prerequisite for ensuring that subsequent acoustic feature decoding can be executed stably and accurately. The intermediate feature vector is the feature carrier output by the feature mapping layer after dimensionality conversion. It can not only retain the collaborative information of semantics, emotion and prosody in semantic representation, but also adapt to the processing needs of acoustic decoding layer in terms of dimension.

[0057] In this embodiment of the present disclosure, after the semantic representation (assuming the original dimension is 1024) is input into the dual encoder-decoder, the feature mapping layer of the dual encoder-decoder first identifies, through a built-in dimension detection mechanism, that the input dimension requirement of the subsequent acoustic decoding layer is 512 dimensions. Then, the feature mapping layer calls a preset linear transformation matrix to perform dimension compression and feature reorganization on the 1024-dimensional semantic representation. At the same time, it retains key features related to emotional expression and prosodic logic in the semantic representation through activation functions such as ReLU, avoiding the loss of core information due to dimension conversion. After the above processing, the feature mapping layer can output an intermediate feature vector (512 dimensions) adapted to acoustic feature decoding. This intermediate feature vector can be directly input into the acoustic decoding layer of the dual encoder-decoder for subsequent processing.

[0058] By performing dimensional transformation on semantic representation through the feature mapping layer, the core problem of mismatch between the semantic representation dimension and the input dimension of the acoustic decoding layer can be effectively solved, avoiding decoding errors or information gaps caused by dimensional differences and ensuring the continuity of the dual encoder-decoder processing flow.

[0059] Step 250: Decode the intermediate feature vector through the acoustic decoding layer of the dual encoder-decoder to output a continuous acoustic parameter sequence containing fundamental frequency, energy, and duration.

[0060] The acoustic decoding layer is the core submodule of the dual encoder-decoder. Through attention mechanisms, temporal modeling and other techniques, it extracts and analyzes information related to acoustic expression from intermediate feature vectors, transforming abstract features into concrete acoustic parameters that can be directly used for speech synthesis. The continuous acoustic parameter sequence is the output of the acoustic decoding layer, a continuous set of values ​​arranged in temporal order. The core includes three key parameters: fundamental frequency (pitch, such as the average fundamental frequency of pleasant speech = 220Hz), energy (volume, such as the average energy of emphatic speech -12dB), and duration (pronunciation time, such as the duration of emotional focus words 0.8 seconds). These parameters directly determine the prosody and emotional expression of the synthesized speech.

[0061] In this embodiment of the present disclosure, after the intermediate feature vector is input into the acoustic decoding layer of the dual encoder-decoder, the acoustic decoding layer first focuses on key information in the intermediate feature vector that is strongly related to acoustic expression (such as the feature dimension corresponding to "moderate pleasure" emotion and the feature dimension corresponding to "high fundamental frequency" prosody) through a multi-head attention mechanism. At the same time, it captures the temporal correlation in the intermediate feature vector (matching the order of text semantics with the pronunciation rhythm of speech) through temporal modeling structures such as recurrent neural networks (RNN) or Transformers. Then, based on the pre-trained acoustic parameter prediction model, the feature information after attention mechanism and temporal modeling is converted into continuous values ​​of fundamental frequency and energy frame by frame, and combined with the boundary information of text semantic units, the pronunciation duration of each semantic unit is predicted synchronously. Finally, the fundamental frequency and energy values ​​of each frame and the temporally sequenced duration parameters are integrated according to the pronunciation order to output a continuous acoustic parameter sequence containing fundamental frequency, energy, and duration.

[0062] Step 260: Smooth the continuous acoustic parameter sequence, and use a vocoder to reconstruct the continuous speech signal from the smoothed continuous acoustic parameter sequence to generate the initial sample synthesized speech.

[0063] The initial sample synthesized speech is a preliminary speech product generated by the vocoder based on a smoothed acoustic parameter sequence. It retains the emotional and prosodic features in the parameter sequence, but may have slight background noise or insufficient syllable clarity.

[0064] In this embodiment of the present disclosure, smoothing processing can first be performed on the continuous acoustic parameter sequence output by the acoustic decoding layer. Specifically, a sliding window averaging algorithm can be used to calculate the local mean of the fundamental frequency and energy values ​​frame by frame, eliminating abrupt changes in single-frame parameters. At the same time, missing frames of parameters can be supplemented by linear interpolation to ensure that the temporal changes of fundamental frequency and energy are continuous and smooth curves. For the duration parameter, if there is a large difference in duration between adjacent semantic units (e.g., the previous word is 0.3 seconds and the next word is 1.5 seconds), the ratio can be adjusted to make it more in line with the rhythmic logic of natural human pronunciation. Then, the smoothed acoustic parameter sequence is input into the vocoder. The vocoder first divides the pronunciation frame segments of each semantic unit according to the duration parameter, then simulates the vocal cord vibration frequency based on the fundamental frequency value to generate a basic pitch waveform, and adjusts the waveform amplitude in combination with the energy value to match the volume intensity. Finally, the waveforms of each frame segment are spliced ​​in time to generate a speech signal with complete semantic rhythm and emotional prosody, i.e., the initial sample synthesized speech.

[0065] By smoothing out abrupt changes and errors in acoustic parameters, it is possible to effectively avoid rhythmic distortion problems such as pitch jumps and fluctuating volume in synthesized speech, making the speech rhythm more in line with the natural pronunciation habits of humans. The vocoder accurately simulates the human vocal mechanism, transforming abstract acoustic parameters into concrete and audible speech signals. It can fully preserve the emotional features and semantic rhythm contained in the smoothed parameters, solving the core problem that parameters cannot be directly converted into speech.

[0066] Step 270: Perform noise reduction and clarity optimization on the initial sample synthesized speech to finally obtain sample synthesized speech that meets the preset sound quality standards.

[0067] Among them, the preset sound quality standard is a predefined quantitative indicator of the quality of the synthesized speech of the evaluation sample. It usually includes requirements such as signal-to-noise ratio (e.g., ≥35dB), clarity (e.g., speech clarity index ≥0.85), and no obvious noise or distortion. It is the basis for judging whether the optimized speech is qualified. The sample synthesized speech is the final speech product obtained after the initial sample synthesized speech has been processed by noise reduction and clarity optimization. It is the core evaluation object for comparison with the benchmark speech in model training.

[0068] In this embodiment of the disclosure, initial sample synthesized speech generated by a vocoder can be obtained first. The noise type (such as background white noise, high-frequency current noise) and intensity are identified using a noise detection tool, and the speech signal is denoised using spectral subtraction. Specifically, noise features in the speech spectrum can be extracted and a noise model can be constructed. The noise spectrum is subtracted from the original speech spectrum, and the clean speech spectrum is retained and inversely transformed into a time-domain speech signal. Subsequently, the clarity of the denoised speech is detected to locate segments with unclear syllables. The high-frequency components corresponding to consonants are enhanced using a speech enhancement algorithm, and the transition parameters between syllables are adjusted to avoid problems such as swallowing words or dragging out sounds. Finally, the optimized speech is compared with a preset sound quality standard. If it meets the standard, it is output as the final sample synthesized speech. If it does not meet the standard, the parameter adjustment process for noise reduction and clarity optimization is repeated until it meets the standard.

[0069] By precisely separating and removing noise components from the original speech through noise reduction processing, the problem of background noise interfering with the main body of the speech can be effectively solved, the purity of the speech can be improved, and the emotional details can be avoided by noise masking them. Clarity optimization processing can solve the defects of unclear syllables and unclear pronunciation by enhancing the consonant frequency and correcting syllable connection, ensuring that the listener can accurately recognize the semantics of the speech.

[0070] Step 280: Compare the synthesized speech samples with the benchmark speech in multiple dimensions. Construct a reward function based on three dimensions: emotion classification, global emotion intensity, and local emphasis control. Use a combination of supervised fine-tuning and group relative strategy optimization to jointly iteratively train the speech synthesis model until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression indicators. Then, determine that the speech synthesis model training is complete.

[0071] For embodiments of this disclosure, when performing multi-dimensional comparisons between the synthesized sample speech and the benchmark speech, and constructing a reward function based on three dimensions—emotion classification, global emotion intensity, and local emphasis control—step 280 of the embodiment may include: Step 280-1: Extract emotional features from the synthesized speech sample and the baseline speech sample respectively. Use a pre-trained emotional classification model to identify the emotional category of the synthesized speech sample and the emotional category of the baseline speech sample. Calculate the matching accuracy between the emotional category of the synthesized speech sample and the emotional category of the baseline speech sample. Construct an emotional classification reward item based on the matching accuracy.

[0072] Among them, the pre-trained emotion classification model is a model that has been trained in advance on a large-scale emotion speech dataset. It has the ability to identify emotion categories from speech emotion features (such as distinguishing between "joy", "sadness", "anger" etc.) and can be used directly for emotion category determination without retraining. The emotion classification reward item is a component of the reward function built based on the matching accuracy. The core logic is: the higher the accuracy, the larger the reward value (e.g., 100% accuracy corresponds to a reward value of +5, and 0% accuracy corresponds to a reward value of -1), which is used to guide the model to optimize the emotion category matching degree.

[0073] In this embodiment of the disclosure, emotional features can first be extracted from the synthesized sample speech and the reference speech to obtain their respective emotional feature vectors. Then, the two sets of emotional feature vectors are input into a pre-trained emotional classification model. The model, through its built-in feature recognition and category mapping mechanism, outputs the emotional category of the synthesized sample speech (e.g., "mildly pleasant") and the emotional category of the reference speech (e.g., "moderately pleasant"). Next, the accuracy rate of matching the emotional categories is calculated. If the categories match (e.g., both are "moderately pleasant"), the accuracy rate is 100%; if they do not match (e.g., the sample is "mildly pleasant" and the reference is "moderately pleasant"), the accuracy rate is 0%. Finally, based on a preset reward rule... (In the formula, For reward items categorized as emotional, Synthesize speech emotion categories for samples. Using the baseline voice emotion category, the calculated matching accuracy is converted into a specific numerical reward, namely the emotion classification reward item.

[0074] Step 280-2: Calculate the intensity difference between the emotional intensity of the sample synthesized speech and the emotional intensity of the reference speech, and construct a global emotional intensity reverse reward item based on the intensity difference.

[0075] Among them, intensity difference is an objective indicator that quantifies the difference in emotional intensity between the synthesized speech and the benchmark speech. It is usually achieved by calculating the Euclidean distance, Manhattan distance, etc. of the three-dimensional VAD values ​​of the two. The smaller the difference, the closer the emotional intensity of the two is. The global emotional intensity reverse reward term is a component of the reward function built based on intensity difference. The core logic is that the smaller the intensity difference, the larger the reward value (i.e., "reverse" association). It is used to guide the model to narrow the overall emotional intensity gap between the synthesized speech and the benchmark speech, so as to achieve fine-grained emotional control.

[0076] In this embodiment of the present disclosure, the emotional intensity of the sample synthesized speech and the reference speech can be extracted separately. Specifically, an acoustic feature analysis tool can be used to extract three-dimensional quantitative values ​​(such as pleasantness, arousal, and dominance, with each dimension ranging from 0 to 10 points) based on the VAD emotional space from both, to obtain the emotional intensity of the sample synthesized speech (such as "pleasure 6, arousal 5, dominance 4") and the emotional intensity of the reference speech (such as "pleasure 8, arousal 5, dominance 4"). Then, the intensity difference between the two can be calculated using the Euclidean distance formula. Specifically, the three-dimensional values ​​can be regarded as spatial coordinates, and the straight-line distance between the coordinates can be calculated (such as in this example, difference = √[(8-6)²+(5-5)²+(4-4)²]=2). Finally, based on the preset reverse mapping rule (such as "reward value = 1 / (1+intensity difference)"), the calculated difference is converted into a specific reward value (in this example, reward value = 1 / (1+2)≈0.33), which is the global emotional intensity reverse reward item.

[0077] Step 280-3: Combining the results of locating emotional focus words in the text information, extract the repetition features of corresponding emotional focus words in the sample synthesized speech and the baseline speech, calculate the matching score of the two sets of repetition features, and construct a local emphasis control reward item based on the matching score.

[0078] Among them, the local emphasis control reward item is a component of the reward function built based on the matching score. The core logic is "the higher the matching score, the greater the reward value" (e.g., a score of 1 corresponds to a reward of 5, and a score of 0.5 corresponds to a reward of 2.5). It is used to guide the model to optimize the emphasis effect of emotional focus words and ensure that the emotional focus of the synthesized speech is consistent with the logic of natural language.

[0079] In this embodiment of the disclosure, the emotional focus word localization results obtained from the semantic parsing of the text information can be called first. Based on the localization results, speech segments corresponding to the emotional focus words are extracted from the sample synthesized speech and the reference speech, respectively. The repetition features of the two segments are extracted by acoustic signal processing tools, namely the fundamental frequency peak, energy peak, and pronunciation duration of the sample synthesized speech segment, and the fundamental frequency peak, energy peak, and pronunciation duration of the reference speech segment. Then, the matching score of the two sets of repetition features is calculated by using a feature similarity algorithm (such as cosine similarity). Finally, based on the preset reward mapping rule (such as "reward value = matching score × reward coefficient, with the coefficient set to 5"), the matching score is converted into a specific reward value (such as 0.93 × 5 ≈ 4.65), which is the local emphasis control reward item.

[0080] Step 280-4: Construct a reward function that includes an emotion classification reward, a global emotion intensity inverse reward, and a local emphasis control reward.

[0081] In this embodiment of the disclosure, the sentiment classification reward, the global sentiment intensity inverse reward, and the local emphasis control reward can be integrated by summing or weighted summation to obtain the final reward function. This provides a clear optimization direction for model parameter updates, namely, maximizing the total reward value of the reward function by adjusting the parameters.

[0082] In this embodiment of the present disclosure, when the speech synthesis model is jointly iteratively trained using a combination of supervised fine-tuning and group relative policy optimization until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotional expression indicators, and the speech synthesis model training is deemed complete, step 280 of the embodiment may include: using the acoustic feature difference between the sample synthesized speech and the reference speech as the loss function, supervising fine-tuning training of the language model LLM and the dual encoder-decoder; based on the supervised fine-tuning training, using the reward function as the optimization objective, performing reinforcement learning training of the language model LLM and the dual encoder-decoder using a group relative policy optimization algorithm; iteratively executing the above joint training process until the loss function of the speech synthesis model is less than a preset threshold and the multi-dimensional rewards meet the preset emotional expression indicators.

[0083] In summary, the technical solution of this application introduces multimodal sample inputs containing text information, emotional intensity information, and speech information, and performs discretization encoding on the three types of information to generate corresponding tokens. Combined with the Language Model (LLM) to achieve deep fusion of multimodal features, it can not only break through the limitations of traditional fixed emotional labels, but also achieve fine-grained emotional intensity adjustment based on emotional intensity information, and ensure the naturalness of synthesized speech through prosodic reference of speech information. Then, the acoustic feature decoding and speech synthesis are completed by dual encoders and decoders. Combined with a reward function constructed based on three dimensions of emotion classification, global emotional intensity, and local emphasis control, and a joint iterative training method that combines supervised fine-tuning and group relative strategy optimization, it can significantly enhance the realism and diversity of emotional expression of synthesized speech while steadily improving the basic sound quality of speech synthesis. Ultimately, it achieves synergistic optimization of fine-grained emotional control and high-quality speech synthesis, meeting the core requirements of delicate and natural emotional speech expression in practical applications.

[0084] Figure 3 A flowchart of an emotional speech synthesis method according to an embodiment of the present invention is provided, as follows: Figure 3 As shown, it may include the following steps: Step 310: Determine the target speech data to be processed by speech synthesis. The target speech data shall include at least the target text information, the target emotion intensity information, and the target speech information.

[0085] Among them, target speech data refers to the pre-determined set of input data to meet specific speech synthesis needs, and is the core processing object of the speech synthesis model; target text information is the text data (such as sentences and phrases) in the target speech data that carries semantic content, which clarifies "what the synthesized speech says" and is the basis for semantic accuracy, such as "I'm going on a spring outing with my family tomorrow"; target emotion intensity information is the quantitative information in the target speech data that defines the emotional subtlety of the synthesized speech, which is constructed based on the VAD emotion space and clarifies "what emotion the synthesized speech carries and how strong the emotion is", such as "moderate pleasure (pleasure score 7, arousal score 5, dominance score 4)"; target speech information is the speech signal in the target speech data that provides prosodic reference (such as real-person recorded speech in the same scene / with the same emotion), which includes prosodic features such as fundamental frequency, energy, and duration, and clarifies "how the synthesized speech is said" (such as rhythm speed and pitch fluctuations), ensuring that the prosody of the synthesized speech conforms to the natural pronunciation habits of humans.

[0086] Step 320: Discretize and encode the target text information, target emotion intensity information, and target speech information respectively to obtain the corresponding target text token, target emotion intensity token, and target speech token.

[0087] Among them, the target text token is a discrete unit after the target text information is discretized and encoded, which includes emotional focus word identifiers (such as “Emo_Focus” for “super happy”) and sequence start and end identifiers (such as “[CLS]” and “[SEP]”), conveying semantic information with emotional emphasis; the target emotional intensity token is a discrete unit after the target emotional intensity information is discretized and encoded, which is generated by binding the VAD emotional intensity level with a preset unique code (such as “Emo_VAD_07” for “moderately happy”), conveying precise emotional intensity; the target speech token is a discrete unit after the target speech information is discretized and encoded, which includes discretized acoustic features (coded values ​​of fundamental frequency, energy, and duration), semantic identifiers (associated with target text units), and type identifiers (labeled “Speech_Modal”), conveying prosodic information with semantic association.

[0088] For the specific implementation process of the embodiments disclosed herein, please refer to the relevant descriptions in step 120 and step 220 of the embodiments, which will not be repeated here.

[0089] Step 330: Input the target text token, target emotion intensity token, and target speech token into the trained speech synthesis model to obtain the emotional synthesized speech corresponding to the target speech data.

[0090] The trained speech synthesis model is a model that has been jointly iteratively trained through supervised fine-tuning and group relative policy optimization (GRPO), and meets the requirements of loss function being less than a preset threshold and multi-dimensional rewards conforming to emotional expression indicators. It has a built-in language model LLM and dual encoder-decoder, and has the ability to stably generate high-quality emotional speech. The emotional synthesized speech is the final speech product generated by the trained speech synthesis model based on the three target tokens input. It has the characteristics of semantic consistency with the target text, emotional intensity matching the target requirements, and prosody matching the target speech reference. It is the output result that directly meets the user's speech synthesis needs.

[0091] In this embodiment of the disclosure, the preprocessed target text token, target emotion intensity token, and target speech token can be simultaneously input into the trained speech synthesis model. The model first establishes modal associations of the three tokens through the built-in language model LLM and utilizes a cross-attention mechanism to complete multimodal feature fusion and generate a unified semantic representation integrating semantics, emotion, and prosody. Subsequently, this semantic representation is input into a dual encoder-decoder, which is first converted into an intermediate feature vector adapted to acoustic decoding by a feature mapping layer, and then a continuous acoustic parameter sequence containing fundamental frequency, energy, and duration is generated by the acoustic decoding layer. Then, parameter abrupt changes are eliminated through smoothing processing. Finally, the speech signal is reconstructed by a vocoder and subjected to noise reduction and clarity optimization to generate speech that simultaneously satisfies semantic accuracy, emotional subtlety, and natural prosody, i.e., the emotional synthesized speech corresponding to the target speech data.

[0092] The trained model, leveraging the parameter advantages accumulated during previous joint training, can accurately parse the core information of the three target tokens, avoiding semantic misunderstandings, emotional biases, and prosodic distortions caused by insufficient model performance. The multimodal fusion mechanism of the language model LLM ensures the coordination of semantics, emotion, and prosody, solving the problems of emotional and semantic disconnect and prosodic and emotional misalignment in traditional models. The standardized processing flow of dual encoders and decoders ensures the clarity and naturalness of the synthesized emotional speech, avoiding the defects of stiff parameterized speech and obvious background noise. The final generated synthesized emotional speech can fully meet the user's preset requirements for semantic content, emotional intensity, and prosodic style, effectively making up for the shortcomings of coarse-grained emotional expression and poor multi-dimensional coordination in existing speech synthesis technologies, and achieving high-quality personalized emotional speech generation.

[0093] In summary, the emotional speech synthesis method provided in this application first determines multimodal target speech data containing target text information, target emotional intensity information, and target speech information. Then, it performs discretization encoding on the three types of target information respectively. Finally, it inputs the encoded target text token, target emotional intensity token, and target speech token into the speech synthesis model. This allows the speech synthesis model to fully combine the three key types of information: semantics, emotion, and prosody. Ultimately, it generates emotional synthesized speech that highly matches the target semantics, has delicate and controllable emotional expression, and whose prosody conforms to the logic of real language. This effectively overcomes the problems of coarse-grained emotion control and the disconnect between synthesized speech and semantics / prosody in existing technologies. While ensuring the basic sound quality of speech synthesis, it significantly improves the realism and relevance of emotional expression, meeting the demand for personalized, high-quality emotional speech in practical applications.

[0094] It should be noted that the speech synthesis model training method and emotional speech synthesis method provided in this application can be applied to, but are not limited to, various scenarios requiring emotional speech interaction. For example, in financial scenarios, they can be adapted to needs such as personal financial affairs consultation assistance, financial business consultation guidance, financial knowledge popularization and risk warning, and notification of personal fund changes; in health-related scenarios, they can support voice interaction scenarios such as health consultation guidance, chronic disease health maintenance guidance, postoperative health recovery advice, and health care for the elderly, providing technical support for the transmission of emotional information in different scenarios.

[0095] Based on the above Figure 1 , Figure 2 A detailed description of the provided speech synthesis model training method, such as Figure 4 As shown, Figure 4 This is a block diagram illustrating a speech synthesis model training apparatus according to an exemplary embodiment. Figure 4 As shown, the device includes: The acquisition module 41 can be used to acquire sample speech data and the corresponding reference speech. The sample speech data includes at least text information, emotion intensity information and speech information. The reference speech is the expected synthesized speech corresponding to the sample speech data. The first processing module 42 can be used to perform discretization encoding on text information, emotion intensity information and voice information respectively to obtain the corresponding text token, emotion intensity token and voice token; The fusion module 43 can be used to input text tokens, sentiment intensity tokens and voice tokens into the language model LLM for multimodal feature fusion to obtain a semantic representation of fused multimodal information; The first synthesis module 44 can be used to input semantic representations into a dual encoder-decoder, perform acoustic feature decoding and continuous speech signal synthesis processing on the semantic representations, and obtain sample synthesized speech. Training module 45 can be used to compare the synthesized speech samples with the benchmark speech in multiple dimensions. It constructs a reward function based on three dimensions: emotion classification, global emotion intensity, and local emphasis control. It uses a combination of supervised fine-tuning and group relative strategy optimization to jointly iteratively train the speech synthesis model until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression indicators, at which point the speech synthesis model training is considered complete.

[0096] In some embodiments of this application, the emotional intensity information includes at least an emotional intensity value; the first processing module 42 is specifically used to perform semantic parsing on the text information, identify the emotional focus words and ordinary text words therein, perform word segmentation and encoding on the emotional focus words and ordinary text words respectively, generate a text token with an emotional focus identifier, and add a sequence start identifier and a sequence end identifier to the text token to mark the boundary of the text token; determine the emotional intensity level corresponding to the emotional intensity value, obtain the emotional intensity token corresponding to the emotional intensity level, wherein different emotional intensity levels are pre-configured with a unique emotional intensity token based on the VAD emotional space; convert the speech information into an acoustic feature sequence containing fundamental frequency, energy, and duration, perform discretization encoding on the acoustic feature sequence, generate a speech token corresponding to the speech information, and add a semantic identifier and a type identifier.

[0097] In some embodiments of this application, when performing word segmentation and encoding on emotional focus words and ordinary text words respectively to generate text tokens with emotional focus identifiers, the first processing module 42 can be specifically used to perform sub-word segmentation and encoding on emotional focus words and ordinary text words respectively, and convert the segmented sub-words into token IDs; after adding emotional focus identifiers to the token IDs corresponding to emotional focus words, the token IDs with emotional focus identifiers are combined with the token IDs corresponding to ordinary text words according to the original semantic order of the text to form a text token containing emotional focus identifiers.

[0098] In some embodiments of this application, the first synthesis module 44 is specifically used to input semantic representations into a dual encoder-decoder, where the feature mapping layer of the dual encoder-decoder performs dimensional transformation on the semantic representations to generate intermediate feature vectors adapted for acoustic feature decoding; the acoustic decoding layer of the dual encoder-decoder decodes the intermediate feature vectors to output a continuous acoustic parameter sequence containing fundamental frequency, energy, and duration; the continuous acoustic parameter sequence is smoothed, and a vocoder is used to reconstruct the smoothed continuous acoustic parameter sequence into a continuous speech signal to generate initial sample synthesized speech; the initial sample synthesized speech is subjected to noise reduction and clarity optimization processing to finally obtain sample synthesized speech that meets preset sound quality standards.

[0099] In some embodiments of this application, when comparing the synthesized sample speech with the reference speech in multiple dimensions and constructing a reward function based on three dimensions—emotion classification, global emotion intensity, and local emphasis control—the training module 45 can specifically be used to extract emotion features from the synthesized sample speech and the reference speech respectively, use a pre-trained emotion classification model to identify the emotion category of the synthesized sample speech and the emotion category of the reference speech, calculate the matching accuracy between the emotion category of the synthesized sample speech and the emotion category of the reference speech, and construct an emotion classification reward item based on the matching accuracy; calculate the intensity difference between the emotion intensity of the synthesized sample speech and the emotion intensity of the reference speech, and construct a global emotion intensity reverse reward item based on the intensity difference; combine the emotion focus word localization results in the text information to extract the repetition features of the corresponding emotion focus words in the synthesized sample speech and the reference speech, calculate the matching score of the two sets of repetition features, and construct a local emphasis control reward item based on the matching score; and construct a reward function that includes the emotion classification reward item, the global emotion intensity reverse reward item, and the local emphasis control reward item.

[0100] In some embodiments of this application, a combination of supervised fine-tuning and group relative policy optimization is used to jointly iteratively train the speech synthesis model until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotional expression indicators. When the speech synthesis model training is considered complete, the training module 45 can be used to perform supervised fine-tuning training on the language model LLM and dual encoder-decoder using the acoustic feature difference between the sample synthesized speech and the reference speech as the loss function. Based on the supervised fine-tuning training, the language model LLM and dual encoder-decoder are trained using a group relative policy optimization algorithm with the reward function as the optimization objective. The above joint training process is iteratively executed until the loss function of the speech synthesis model is less than the preset threshold and the multi-dimensional rewards meet the preset emotional expression indicators.

[0101] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0102] The embodiments of this application introduce multimodal sample inputs containing text information, emotional intensity information, and speech information, and perform discretization encoding on the three types of information to generate corresponding tokens. Combined with the Language Model LLM, deep fusion of multimodal features is achieved. This not only breaks through the limitations of traditional fixed emotional labels, but also enables fine-grained emotional intensity adjustment based on emotional intensity information, and ensures the naturalness of synthesized speech through prosodic reference of speech information. Then, acoustic feature decoding and speech synthesis are completed through dual encoders and decoders. Combined with a reward function constructed based on three dimensions of emotional classification, global emotional intensity, and local emphasis control, and a joint iterative training method that combines supervised fine-tuning and group relative strategy optimization, it can significantly enhance the realism and diversity of emotional expression in synthesized speech while steadily improving the basic sound quality of speech synthesis. Ultimately, it achieves synergistic optimization of fine-grained emotional control and high-quality speech synthesis, meeting the core requirements of delicate and natural emotional speech expression in practical applications.

[0103] Based on the above Figure 3 The provided method for emotional speech synthesis details how, after training the speech synthesis model, its use can be referenced to that of a normal deep learning model. Specifically, an emotional speech synthesis device is provided, such as... Figure 5 As shown, the device includes: The determination module 51 can be used to determine the target speech data to be processed by speech synthesis. The target speech data includes at least target text information, target emotion intensity information and target speech information. The second processing module 52 can be used to perform discretization encoding on the target text information, target emotion intensity information and target speech information respectively to obtain the corresponding target text token, target emotion intensity token and target speech token; The second synthesis module 53 can be used to input the target text token, the target emotion intensity token, and the target speech token into the trained speech synthesis model to obtain the emotional synthesized speech corresponding to the target speech data.

[0104] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0105] The embodiments of this application first determine multimodal target speech data containing target text information, target emotion intensity information, and target speech information, then perform discretization encoding on the three types of target information respectively, and finally input the encoded target text token, target emotion intensity token, and target speech token into the speech synthesis model. This allows the speech synthesis model to fully combine the three key types of information: semantics, emotion, and prosody, and ultimately generate emotional synthesized speech that highly matches the target semantics, has delicate and controllable emotional expression, and whose prosody conforms to the logic of real language. This effectively overcomes the problems of coarse-grained emotion control and the disconnect between synthesized speech and semantics / prosody in the prior art. While ensuring the basic sound quality of speech synthesis, it significantly improves the realism and relevance of emotional expression, meeting the demand for personalized, high-quality emotional speech in practical applications.

[0106] Based on the above, Figures 1 to 3 Accordingly, this embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method. Figure 1 and Figure 2 The speech synthesis model training method shown and such Figure 3 The emotional speech synthesis method shown.

[0107] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause an electronic device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0108] Based on the above, Figures 1 to 3 The method shown, and Figure 4 and Figure 5 To achieve the above objectives, the present application also provides an electronic device, specifically a personal computer, tablet computer, server, or other network device, as shown in the virtual device embodiment. This device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 1 and Figure 2 The speech synthesis model training method shown and such Figure 3 The emotional speech synthesis method shown.

[0109] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0110] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0111] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.

[0113] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0114] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for training a speech synthesis model, characterized in that, The speech synthesis model includes a language model LLM and a dual encoder-decoder, and the method includes: Acquire sample speech data and the corresponding reference speech, wherein the sample speech data includes at least text information, emotion intensity information and speech information, and the reference speech is the expected synthesized speech corresponding to the sample speech data; The text information, the emotion intensity information, and the voice information are discretized and encoded respectively to obtain the corresponding text token, emotion intensity token, and voice token; The text token, the emotion intensity token, and the voice token are input into the language model LLM for multimodal feature fusion to obtain a semantic representation that integrates multimodal information; The semantic representation is input into the dual encoder-decoder, and the semantic representation is subjected to acoustic feature decoding and continuous speech signal synthesis processing to obtain sample synthesized speech; The synthesized speech samples are compared with the baseline speech in multiple dimensions. A reward function is constructed based on three dimensions: emotion classification, global emotion intensity, and local emphasis control. The speech synthesis model is jointly iteratively trained by combining supervised fine-tuning and group relative strategy optimization until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression indicators. The speech synthesis model is then judged to have completed training.

2. The method according to claim 1, characterized in that, The emotional intensity information includes at least an emotional intensity value; The text information, the emotion intensity information, and the speech information are each discretized and encoded to obtain corresponding text tokens, emotion intensity tokens, and speech tokens, including: The text information is semantically parsed to identify emotional focus words and ordinary text words. The emotional focus words and ordinary text words are segmented and encoded to generate text tokens with emotional focus identifiers. Sequence start identifiers and sequence end identifiers are added to the text tokens to mark the boundaries of the text tokens. Determine the emotional intensity level corresponding to the emotional intensity value, and obtain the emotional intensity token corresponding to the emotional intensity level. Each emotional intensity level is pre-configured with a unique emotional intensity token based on the VAD emotional space. The speech information is converted into an acoustic feature sequence containing fundamental frequency, energy, and duration. The acoustic feature sequence is then discretized and encoded to generate a speech token corresponding to the speech information. Semantic and type identifiers are then added.

3. The method according to claim 2, characterized in that, The emotional focus words and the ordinary text words are segmented and encoded to generate text tokens with emotional focus identifiers, including: Sub-word segmentation and encoding are performed on the emotional focus words and the ordinary text words respectively, and the segmented sub-words are converted into token IDs; After adding an emotional focus identifier to the token ID corresponding to the emotional focus words, the token ID with the emotional focus identifier is combined with the token ID corresponding to the ordinary text words in the original semantic order of the text to form a text token containing the emotional focus identifier.

4. The method according to claim 1, characterized in that, The semantic representation is input into the dual encoder-decoder, and acoustic feature decoding and continuous speech signal synthesis are performed on the semantic representation to obtain sample synthesized speech, including: The semantic representation is input into a dual encoder-decoder, and the feature mapping layer of the dual encoder-decoder performs dimensionality transformation on the semantic representation to generate an intermediate feature vector adapted for acoustic feature decoding. The intermediate feature vector is decoded by the acoustic decoding layer of the dual encoder-decoder to output a continuous sequence of acoustic parameters containing fundamental frequency, energy, and duration. The continuous acoustic parameter sequence is smoothed, and a vocoder is used to reconstruct the continuous speech signal from the smoothed continuous acoustic parameter sequence to generate initial sample synthesized speech. The initial sample synthesized speech is subjected to noise reduction and clarity optimization processing to finally obtain sample synthesized speech that meets the preset sound quality standards.

5. The method according to claim 1, characterized in that, The synthesized speech from the sample is compared with the baseline speech in multiple dimensions. A reward function is constructed based on three dimensions: emotion classification, global emotion intensity, and local emphasis control, including: Emotional features are extracted from the synthesized speech sample and the baseline speech sample, and a pre-trained emotion classification model is used to identify the emotion category of the synthesized speech sample and the emotion category of the baseline speech sample. The matching accuracy between the emotion category of the synthesized speech sample and the emotion category of the baseline speech sample is calculated, and an emotion classification reward item is constructed based on the matching accuracy. Calculate the intensity difference between the emotional intensity of the synthesized speech sample and the emotional intensity of the reference speech, and construct a global emotional intensity reverse reward term based on the intensity difference. Based on the location results of the emotional focus words in the text information, the repetition features of the corresponding emotional focus words in the sample synthesized speech and the reference speech are extracted, the matching score of the two sets of repetition features is calculated, and a local emphasis control reward item is constructed based on the matching score. Construct a reward function that includes the emotion classification reward, the global emotion intensity inverse reward, and the local emphasis control reward.

6. The method according to claim 1, characterized in that, The method of jointly iteratively training the speech synthesis model using a combination of supervised fine-tuning and group relative strategy optimization until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotional expression indicators, is used to determine that the speech synthesis model training is complete. This includes: The language model LLM and the dual encoder-decoder are trained under supervised supervision using the acoustic feature difference between the synthesized speech and the reference speech as the loss function. Based on the supervised fine-tuning training, the language model LLM and the dual encoder-decoder are trained using the group relative policy optimization algorithm with the reward function as the optimization objective. The above joint training process is executed iteratively until the loss function of the speech synthesis model is less than a preset threshold and the multi-dimensional rewards meet the preset emotional expression indicators.

7. A method for synthesizing emotional speech, characterized in that, include: Determine the target speech data to be processed by speech synthesis, wherein the target speech data includes at least target text information, target emotion intensity information, and target speech information; The target text information, the target emotion intensity information, and the target speech information are discretized and encoded respectively to obtain the corresponding target text token, target emotion intensity token, and target speech token; The target text token, the target emotion intensity token, and the target speech token are input into the speech synthesis model trained according to any one of claims 1 to 6 to obtain the emotional synthesized speech corresponding to the target speech data.

8. A speech synthesis model training device, characterized in that, The speech synthesis model includes a language model LLM and a dual encoder-decoder, and the method includes: The acquisition module is used to acquire sample speech data and the corresponding reference speech. The sample speech data includes at least text information, emotion intensity information and speech information. The reference speech is the expected synthesized speech corresponding to the sample speech data. The first processing module is used to perform discretization encoding on the text information, the emotion intensity information and the voice information respectively to obtain the corresponding text token, emotion intensity token and voice token; The fusion module is used to input the text token, the emotion intensity token, and the voice token into the language model LLM to perform multimodal feature fusion and obtain a semantic representation with fused multimodal information; The first synthesis module is used to input the semantic representation into the dual encoder-decoder, perform acoustic feature decoding and continuous speech signal synthesis processing on the semantic representation, and obtain sample synthesized speech; The training module is used to compare the sample synthesized speech with the benchmark speech in multiple dimensions, construct a reward function based on three dimensions: emotion classification, global emotion intensity, and local emphasis control, and perform joint iterative training on the speech synthesis model by combining supervised fine-tuning and group relative strategy optimization until the loss function of the speech synthesis model is less than a preset threshold or the multi-dimensional rewards meet the preset emotion expression index, at which point the speech synthesis model is judged to have completed training.

9. An emotional speech synthesis device, characterized in that, include: The determination module is used to determine the target speech data to be processed by speech synthesis, wherein the target speech data includes at least target text information, target emotion intensity information and target speech information; The second processing module is used to perform discretization encoding on the target text information, the target emotion intensity information and the target speech information respectively to obtain the corresponding target text token, target emotion intensity token and target speech token; The second synthesis module is used to input the target text token, the target emotion intensity token, and the target speech token into the speech synthesis model trained according to any one of claims 1 to 6, so as to obtain the emotional synthesized speech corresponding to the target speech data.

10. A computer-readable storage medium, characterized in that, Used to store computer programs that cause a computer to perform the speech synthesis model training method as described in any one of claims 1-6 or the emotional speech synthesis method as described in claim 7.

Images