Emotion controllable speech synthesis preference alignment method and system based on reinforcement learning

Abstract

The application relates to a method and system for emotion controllable speech synthesis preference alignment based on reinforcement learning, and relates to the field of speech signal processing. The method solves the problem that, in the prior art, a mechanism capable of integrating fine-grained emotion control range, local emotion expression, global speaker maintenance and multi-target relative preference into a post-training process is lacked. The method comprises the following steps: taking a fine-grained emotion control label with emotion categories, emotion intensity and action text range, target text and reference speech as training conditions; generating multiple candidate speeches by an old strategy for the same condition; positioning a target segment of the candidate speech according to the emotion control range and calculating a local emotion consistency reward, and meanwhile, calculating a global speaker consistency reward between the candidate whole-sentence speech and the reference speech; normalizing the two types of rewards in the candidate group respectively and forming a joint relative advantage; and updating a speech synthesis model through group relative strategy optimization of a current strategy and old strategy probability ratio truncation and reference strategy divergence constraint.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, specifically to a method and system for emotion-controlled speech synthesis preference alignment based on reinforcement learning. Background Technology

[0002] Existing emotion-controlled speech synthesis methods typically utilize joint training with text content, emotion tags, and reference speech. However, the implicit emotion of the text, the target emotion tag, and the emotional tendency of the reference speech in the training samples often naturally align. While this training approach can learn speech generation relationships under normal conditions, it can easily lead the model to mistakenly interpret the emotional tendency of the text itself or the emotional prosody in the reference speech as the target control condition. When users request the synthesis of a text segment with a different target emotion using a reference speech with a certain emotional tendency in actual use, target emotion shifts or timbre drift can easily occur when reinforcing the target emotion.

[0003] On the other hand, existing sentiment evaluation methods typically assign a single sentiment score based on the entire sentence. In scenarios where different sentiment control segments exist within the same sentence, the overall sentence score is easily influenced by non-target segments and cannot directly reflect whether the model expressed the target sentiment within the specified text range. Furthermore, if the sentiment score and timbre score are directly added as the absolute reward, the numerical range and fluctuation of different scores may vary, leading to a single objective dominating model updates and thus affecting the training stability under multi-condition control.

[0004] Therefore, while existing technologies can increase the number of emotion categories or expand the scale of speech synthesis models, they lack a mechanism that can integrate fine-grained emotion control, local emotion expression, global speaker preservation, and multi-objective relative preferences into the post-training process. Only by comparing multiple candidate speech under the same control conditions and updating the model based on the relative merits of the candidates can the model prioritize responding to explicit fine-grained emotion control information even in the presence of conditional interference. Summary of the Invention

[0005] This invention addresses several key challenges in existing technologies. Specifically, it addresses how to leverage emotion category, emotion intensity, and the scope of the target text to locate speech segments within generated speech that correspond to the target text range, thereby generating local emotion consistency rewards. Other challenges include: how to maintain timbre through speaker consistency rewards between candidate whole-sentence speech and reference speech while reinforcing local emotion expression; how to eliminate the influence of different reward scales among multiple candidate speech under the same input conditions to form a reliable joint relative advantage; how to utilize old policy sampling, probability ratio truncation, and reference policy divergence constraints to achieve stable group relative policy optimization; and how to ensure the training process covers combinations of inconsistent conditions between text semantics, explicit emotion control, and reference speech, reducing both text and reference speech emotion leakage. To these challenges, this invention proposes an emotion-controlled speech synthesis preference alignment method and system based on reinforcement learning. The invention achieves these technical problems through the following technical solutions: Option 1: This invention proposes an emotion-controlled speech synthesis preference alignment method based on reinforcement learning, the method comprising the following steps: Step 1: Obtain text with fine-grained emotion control labels and reference speech for maintaining speaker identity. Perform conditional reorganization on the text labels or reference speech, organize consistency conditions and auxiliary conflict training conditions. For each training condition, the old strategy generates multiple candidate speech discrete representations with text, fine-grained emotion control labels and reference speech as joint conditions, and decodes each candidate speech discrete representation into candidate speech. Step 2: Based on the scope of the fine-grained emotion control label, locate the target speech segment in the candidate speech mentioned in Step 1 and calculate the local emotion consistency reward; at the same time, calculate the global speaker consistency reward between the candidate whole sentence speech and the reference speech. Step 3: Within the candidate group corresponding to the same input condition, form a joint relative advantage based on the two types of rewards in Step 2, and use a group relative strategy optimization update of the speech synthesis strategy, including the probability ratio truncation of the current strategy and the old strategy and the divergence constraint of the reference strategy, to finally obtain an emotion-controllable speech synthesis model.

[0006] Furthermore, a preferred embodiment is provided, wherein the tags mentioned in step 1 are obtained by a preprocessing annotation process based on the training speech, the corresponding transcribed text and their time correspondence, and the fine-grained emotion control tags are obtained based on the training speech, the transcribed text and their time correspondence; and continuous text units with the same emotion category and intensity are merged into an emotion control segment.

[0007] Furthermore, a preferred embodiment is provided in which the fine-grained emotion control tag in step 2 includes at least one of emotion category, emotion intensity, and effective text range.

[0008] Furthermore, a preferred embodiment is provided, wherein the method for calculating the local emotional consistency reward in step 2 is as follows: Parse the control tags to obtain the sentiment category, sentiment intensity, and scope of each target text fragment; Based on the text boundaries of the emotion control segment and the text-speech time correspondence, the corresponding target speech segment is determined from the candidate speech; The emotional representation of the target speech segment is extracted using a pre-trained emotional representation model, and the similarity is calculated with the emotional prototype corresponding to the target emotional category to obtain the local emotional consistency reward of the candidate on the target control segment.

[0009] Furthermore, a preferred embodiment is provided in which the text-speech time correspondence is obtained through one or any combination of duration prediction, forced alignment, and duration ratio estimation.

[0010] Furthermore, a preferred embodiment is provided, wherein the candidate group in step 3 consists of multiple candidate voices sampled from an old speech synthesis strategy, which are the same text, the same fine-grained emotion control label, and the same reference speech conditions.

[0011] Furthermore, a preferred embodiment is provided, wherein the method for optimizing and updating the speech synthesis strategy in step 3 using a group relative strategy that includes truncation of the probability ratio between the current strategy and the old strategy and divergence constraints of the reference strategy is as follows: Based on the old strategy, generate a group of discrete representations of candidate speech under the same conditions, and decode each candidate into candidate speech; The target segment is located based on the tag range, and the similarity between the target segment and the target sentiment category prototype is calculated. Calculate the speaker representation similarity between candidate whole-sentence speech and reference speech, normalize the two types of rewards within the candidate group, and form a joint relative advantage according to preset weights; construct an update term using the probability ratio of the current policy to the old policy and truncate it, while adding a divergence constraint relative to the reference policy to update the speech synthesis policy.

[0012] Option 2: A reinforcement learning-based emotion-controlled speech synthesis preference alignment system, the system comprising: The fine-grained emotion control label preparation module is used to acquire text with fine-grained emotion control labels and reference speech for maintaining speaker identity. It performs conditional reorganization on the text labels or reference speech, organizes consistency conditions and auxiliary conflict training conditions. For each training condition, the old strategy generates multiple candidate speech discrete representations with text, fine-grained emotion control labels and reference speech as joint conditions, and decodes each candidate speech discrete representation into candidate speech. The reward calculation module is used to locate the target speech segment in the candidate speech described in the fine-grained emotion control label preparation module and calculate the local emotion consistency reward based on the scope of the fine-grained emotion control label; at the same time, it calculates the global speaker consistency reward between the candidate whole sentence speech and the reference speech. The group relative strategy optimization module is used to form a joint relative advantage based on the two types of rewards in step 2 within the candidate group corresponding to the same input conditions, and to update the speech synthesis strategy by using group relative strategy optimization including truncation of the probability ratio between the current strategy and the old strategy and divergence constraint of the reference strategy, so as to finally obtain an emotion-controllable speech synthesis model.

[0013] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0014] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0015] The advantages of this invention are: The reinforcement learning-based emotion-controlled speech synthesis preference alignment method described in this invention has the following beneficial technical effects: First, by locating the target segment in the candidate speech based on the fine-grained emotion control range, the emotion expression of the specified text range is directly constrained by the local emotion consistency reward, avoiding interference from non-target segments in the reward calculation; Second, by simultaneously calculating the global speaker consistency reward of the candidate whole sentence speech and the reference speech, the identity characteristics of the reference speaker are maintained while strengthening the target emotion expression, reducing timbre drift; Third, by normalizing the local emotion reward and the global speaker reward separately within the same candidate group and forming a joint relative advantage, the impact of different reward scales and distribution differences on the training process is reduced; Fourth, by optimizing the group relative policy through old policy sampling, probability ratio truncation, and reference policy divergence constraints, single sampling fluctuations and excessive policy deviations are suppressed, making the model more suitable for stable training under complex and multi-condition conditions; Fifth, by organizing auxiliary conflict conditions between the text, explicit emotion control, and reference speech, the model learns to suppress interference from the implicit emotion of the text and the emotional prosody of the reference speech during the training phase.

[0016] This invention is also applicable to real-time voice broadcasting of virtual digital humans, voice interaction in intelligent cockpits, intelligent customer service, automatic dubbing of audiobooks, voice generation of multimedia content, immersive human-computer interaction systems, and various artificial intelligence terminal devices that require personalized and controllable emotional voice output. It is especially suitable for high-end voice generation application scenarios that require fine alignment and adaptive optimization of voice emotional style, prosodic features, and expressive intensity based on human subjective auditory preferences. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the emotion-controlled speech synthesis preference alignment method based on reinforcement learning as described in Implementation Method 1.

[0018] Figure 2 This is a flowchart of the candidate group reward calculation and the relative strategy optimization of the constrained group as described in Implementation Method 1. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0020] Implementation Method 1, see [link] Figures 1 to 2 This embodiment proposes a reinforcement learning-based method for aligning emotion-controlled speech synthesis preferences. This method addresses zero-sample speech synthesis scenarios where text semantics, explicit emotion control commands, and reference speech may have different emotional tendencies. It uses pre-obtained, fine-grained emotion control labels aligned with text units and reference speech as input conditions. Under the same input conditions, multiple candidate speech samples are generated using an existing strategy. Local emotion consistency rewards are calculated for candidate speech segments corresponding to the label's scope, and global speaker consistency rewards are calculated for candidate whole sentences and reference speech. A joint relative advantage is then formed within the candidate group. The speech synthesis model is updated through group relative strategy optimization, including probability ratio truncation and reference strategy divergence constraints, resulting in a speech synthesis model that prioritizes the target emotion control while maintaining the reference speaker's timbre. During training, conflicting condition samples can be organized to ensure that the above optimization covers input conditions where text, emotion control, and reference speech interfere with each other.

[0021] Unlike traditional schemes that fit speech solely on natural consistency training samples, this scheme does not treat sentiment labels merely as supplementary descriptions of the training corpus, nor does it treat sentiment scores as only posterior results for the entire sentence. Instead, it transforms both local sentiment consistency rewards within the text segment and global speaker consistency rewards within the entire sentence into relative preference criteria during the model's post-training phase. The system simultaneously employs candidate group comparison, policy update truncation, and reference policy constraints, enabling stable, emotion-controlled speech synthesis training under complex control conditions. Furthermore, after training, it can directly output target speech based on the text, fine-grained sentiment control information, and reference speech.

[0022] The emotion-controlled speech synthesis preference alignment method based on reinforcement learning described in this embodiment specifically includes the following steps: like Figure 1 As shown, this embodiment first receives text with fine-grained emotion control labels and reference speech to maintain speaker identity; the labels are obtained by a preprocessing annotation process based on the training speech, the corresponding transcribed text, and their temporal correspondence. The system can perform conditional reorganization on the text labels or reference speech, organizing consistent conditions and auxiliary conflict training conditions. For each training condition, the old strategy generates multiple discrete representations of candidate speech using text, fine-grained emotion control labels, and reference speech as joint conditions, and decodes each discrete representation of candidate speech into candidate speech.

[0023] Secondly, based on the scope of the fine-grained emotion control labels, the system locates the target speech segment in the candidate speech and calculates the local emotion consistency reward; at the same time, the system calculates the global speaker consistency reward between the candidate whole sentence speech and the reference speech.

[0024] Finally, within the candidate group corresponding to the same input condition, the system forms a joint relative advantage based on the two types of rewards mentioned above, and adopts a group relative strategy optimization update of the speech synthesis strategy, including truncation of the probability ratio between the current strategy and the old strategy and divergence constraint of the reference strategy, and finally obtains an emotion-controllable speech synthesis model.

[0025] The fine-grained control conditions and auxiliary conflict samples are organized as follows: Fine-grained emotion control information refers to emotion conditions corresponding to one or more text units, rather than providing a single emotion label for the entire sentence. Preferably, each control label includes at least the emotion category, emotion intensity, and the range of text it applies to. The preprocessing annotation process obtains the labels based on the training speech, the transcribed text, and their temporal correspondence; consecutive text units with the same emotion category and intensity can be merged into one emotion control segment. These labels serve as input conditions for candidate speech generation along with the text and as the basis for target segment localization in subsequent local emotion reward calculations.

[0026] To ensure that the group-based relative strategy optimization covers multiple conditions of interference, the system can organize the labels and reference speech into auxiliary conflict training conditions. Specifically, the system retains the consistency condition where the text, control labels, and reference speech are naturally consistent; it can also replace the sentiment category of some control segments with different categories to form semantic conflict conditions where the explicit control label is inconsistent with the implicit sentiment tendency of the text; it can select reference speech from the same speaker whose sentiment tendency is inconsistent with the target control to form timbre conflict conditions; and it can also use two types of processing simultaneously to form dual conflict conditions. The above condition organization only provides input conditions containing interference for post-training, enabling the model to learn to prioritize explicit sentiment control and maintain speaker identity.

[0027] Text-to-speech temporal correspondences can be obtained through duration prediction, forced alignment, duration ratio estimation, or a combination thereof. This solution does not limit the scope of patent protection to a specific temporal mapping implementation, but emphasizes the consistent use of the same fine-grained emotion control scope in candidate generation, target segment localization, and policy updating.

[0028] The calculation of local emotional reward and global timbre reward in this embodiment is as follows: The local sentiment consistency reward described in this embodiment is formed for the target control region. The system first parses the control tags to obtain the sentiment category, sentiment intensity, and scope of each target text segment. Then, based on the text boundaries and text-speech temporal correspondence of the segment, the system identifies the corresponding target speech segment from the candidate speech. The system uses a pre-trained sentiment representation model to extract the sentiment representation of the target speech segment and calculates its similarity with the sentiment prototype corresponding to the target sentiment category, obtaining the local sentiment consistency reward for that candidate segment on the target control segment. Rewards for multiple target control segments can be calculated separately and then aggregated to avoid the sentiment state of non-target segments masking the control result of the target segment.

[0029] To maintain speaker identity while enhancing local emotional expression, this scheme simultaneously calculates a global speaker consistency reward. The system extracts speaker representations from both candidate sentence speech and reference speech, and determines the degree of timbre preservation in the candidate sentence speech based on their similarity. The local emotional consistency reward applies to the local area with target emotional control, while the global speaker consistency reward applies to the entire sentence. These two rewards have different granularities and cannot be replaced by a single sentence emotional score.

[0030] Specifically, the system performs intra-group normalization on the local affective consistency reward and global speaker consistency reward for each candidate in the same candidate group, and then forms a joint relative advantage according to preset weights. The joint relative advantage reflects the relative merits of each candidate under the same control conditions, rather than relying on absolute scores that cannot be directly compared across conditions; this mechanism can reduce the impact of differences in reward dimensions and distributions on policy updates.

[0031] The relative strategy optimization and stability constraints for candidate groups described in this embodiment are as follows: A candidate group refers to multiple candidate speech samples obtained from an old speech synthesis strategy for the same text, the same fine-grained emotion control label, and the same reference speech conditions. Each candidate in a candidate group has the same control objective, and therefore a preference signal can be formed based on the relative merits of local emotional rewards and global timbre rewards. This scheme does not require pre-labeling a pair of "preferred speech" and "inferior speech," but instead directly constructs the strategy update basis based on the joint relative advantages within the same candidate group.

[0032] Specifically, the policy optimization module employs Group Relative Policy Optimization (GRPO) while retaining both the old and reference policies. The system constructs a probability ratio using the generation probability relationship between candidates under the current and old policies, using the joint relative advantage as the update direction, and limits the single update magnitude corresponding to this probability ratio through a truncation mechanism. Simultaneously, the system calculates the divergence between the current policy and the reference policy, and uses this divergence as a constraint term. Thus, the Group Relative Policy Optimization module increases the generation probability of candidates with good local sentiment consistency and global speaker consistency, reduces the generation probability of candidates with sentiment misalignment or timbre shift, and suppresses situations where the model deviates from its original speech generation capabilities in pursuit of a single reward.

[0033] It is important to emphasize that the multiple candidates in this implementation are mainly used to form relative preference signals during the training phase. After training is completed, the updated speech synthesis model can directly receive text, fine-grained emotion control tags, and reference speech for inference output, without requiring the repeated generation of candidate groups or reordering of candidates each time inference is deployed.

[0034] The specific training and optimization approach in this implementation method is as follows: This implementation allows using a model with basic speech synthesis capabilities as the initial policy model, and then performing group-based policy optimization on top of it. Training records include at least text with fine-grained emotion control labels, reference speech or reference speech identifiers, and associated information for identifying the target speaker. To ensure post-training covers both regular and interference conditions, training records can be organized simultaneously into consistent conditions and auxiliary conflict conditions obtained through condition recombination, but fixed sample ratios, fixed number of emotion categories, or fixed reference speech selection rules are not necessary restrictions.

[0035] like Figure 2 As shown, for any training condition, the old strategy generates a group of discrete candidate speech representations under the same conditions and decodes each candidate into a candidate speech. The local emotion reward calculation module locates the target segment based on the label range and calculates its similarity to the target emotion category prototype. The global speaker reward calculation module calculates the speaker representation similarity between the candidate whole sentence and the reference speech. The system normalizes the two types of rewards within the candidate group and forms a joint relative advantage according to preset weights. Subsequently, the group relative policy optimization module constructs an update term using the probability ratio of the current policy to the old policy and truncates it, while adding a divergence constraint relative to the reference policy to update the speech synthesis policy. The above processing makes the training process directly revolve around local emotion control, global timbre preservation, and stable policy updates.

[0036] This implementation adopts a training approach of "first acquiring basic speech synthesis capabilities, then performing group-based relative strategy optimization." In the post-training phase, multiple candidates are generated for the same control condition. A joint relative advantage is formed using local emotion consistency rewards and global speaker consistency rewards, and the trainable adaptation parameters in the speech synthesis model are updated through group-based relative strategy optimization. Specifically, the updated objects include the speech discrete representation generation network, the emotion conditional coding module, the conditional modulation module, or a combination thereof.

[0037] It is important to emphasize that the key to this implementation method lies not in a specific number of candidates, truncation threshold, or divergence coefficient, but rather in: forming candidate groups using the old strategy with the same multi-condition input; forming a relative advantage of the candidate groups by combining the local emotional reward and the whole-sentence speaker reward corresponding to the target text range; and then optimizing the relative strategy of the groups using probability ratio truncation and reference strategy divergence constraints, feeding this relative advantage back to the emotion-controlled speech synthesis model. After training, the system uses the updated strategy for speech synthesis, ensuring that it can still prioritize explicit fine-grained emotion control even when the implicit emotion of the text or the emotional tendency of the reference speech is inconsistent with the explicit control.

[0038] Implementation Method Two: This implementation method also proposes an emotion-controlled speech synthesis preference alignment system based on reinforcement learning, the system comprising: The fine-grained emotion control label preparation module is used to acquire text with fine-grained emotion control labels and reference speech for maintaining speaker identity. It performs conditional reorganization on the text labels or reference speech, organizes consistency conditions and auxiliary conflict training conditions. For each training condition, the old strategy generates multiple candidate speech discrete representations with text, fine-grained emotion control labels and reference speech as joint conditions, and decodes each candidate speech discrete representation into candidate speech. The reward calculation module is used to locate the target speech segment in the candidate speech described in the fine-grained emotion control label preparation module and calculate the local emotion consistency reward based on the scope of the fine-grained emotion control label; at the same time, it calculates the global speaker consistency reward between the candidate whole sentence speech and the reference speech. The group relative strategy optimization module is used to form a joint relative advantage based on the two types of rewards in step 2 within the candidate group corresponding to the same input conditions, and to update the speech synthesis strategy by using group relative strategy optimization including truncation of the probability ratio between the current strategy and the old strategy and divergence constraint of the reference strategy, so as to finally obtain an emotion-controllable speech synthesis model.

[0039] This embodiment also includes an auxiliary conflict condition organization module, a candidate speech generation and decoding module, a local emotion reward calculation module, a global speaker reward calculation module, and a candidate group relative advantage construction module. The fine-grained emotion control label preparation module primarily takes training speech, transcribed text, and their temporal correspondence as input, and outputs control labels with emotion category, emotion intensity, and effective text range, providing a unified basis for candidate generation and target segment localization. The auxiliary conflict condition organization module primarily takes text with control labels, reference speech, and speaker identifiers as input, and outputs consistency conditions, semantic conflict conditions, timbre conflict conditions, or dual conflict conditions, used to cover input combinations that may lead to emotion or timbre leakage during post-training. The candidate speech generation and decoding module primarily takes text, control labels, reference speech, and the old strategy as input, and outputs multiple candidate speech under the same conditions, used to provide a unified basis for candidate generation and target segment localization. The candidate group relative advantage construction provides comparable candidate results; the local emotion reward calculation module mainly takes candidate speech, control labels, and text-speech time correspondence as input, and outputs local emotion consistency reward, which is used to form training reward based on the emotion expression of the target control region; the global speaker reward calculation module mainly takes candidate whole sentence speech and reference speech as input, and outputs global speaker consistency reward, which is used to form training reward based on the speaker identity of the whole sentence range; the candidate group relative advantage construction module mainly takes two types of rewards for each candidate under the same conditions as input, and outputs joint relative advantage, which is used to convert rewards of different granularities into relative preference signals between candidates; the group relative policy optimization module mainly takes joint relative advantage, current policy, old policy, and reference policy as input, and outputs updated speech synthesis policy, which is used to implement stable updates using probability ratio truncation and reference policy divergence constraints.

[0040] Example 1: This example is used to further explain and illustrate the above implementation methods. The system first obtains fine-grained emotion control labels aligned with the transcribed training speech text through a preprocessing annotation process. These labels include emotion category, emotion intensity, and the scope of the affected text. The system then associates each training record with reference speech from the same target speaker, preserving the training record text, complete control labels, reference speech or its identifier, and speaker association information. When necessary, the system performs conditional reorganization on the control labels or reference speech, creating consistency conditions, semantic conflict conditions, timbre conflict conditions, and dual conflict conditions.

[0041] For any training condition, the old strategy generates multiple discrete representations of candidate speech; the candidate speech decoding module converts each discrete representation of candidate speech into candidate speech. The local emotion reward calculation module obtains the corresponding target segment in the candidate speech based on the boundary of the target text segment in the control label, and calculates the similarity between the target segment and the target emotion category prototype; the global speaker reward calculation module calculates the speaker representation similarity between the candidate whole sentence speech and the reference speech.

[0042] The system then performs intra-group normalization on the two types of rewards from multiple candidates under the same training conditions, and forms a joint relative advantage according to preset weights. The group relative strategy optimization module updates the speech synthesis strategy based on the joint relative advantage, the probability ratio of the current strategy to the old strategy, and the divergence constraint of the reference strategy. If the update magnitude of the current strategy relative to the old strategy exceeds a preset range, the update magnitude corresponding to the probability ratio is truncated. After iterative updates, a speech synthesis model for fine-grained emotion control is obtained.

[0043] From a system deployment perspective, this solution includes at least the following: fine-grained emotion control label preparation service, auxiliary conflict condition organization service, candidate speech generation service, candidate speech decoding service, local emotion reward calculation service, global speaker reward calculation service, and group-based relative strategy optimization service. These modules can be deployed on the same computing node or on different computing nodes as needed for label preparation and model training. The local emotion reward calculation service uses a pre-trained emotion representation model and emotion category prototypes, while the global speaker reward calculation service uses a pre-trained speaker representation model.

[0044] During the model usage phase, the system receives the target text, fine-grained emotion control information aligned with the target text units, and reference speech. It first converts the text and control information into conditional representations, then uses the updated speech synthesis model to generate a discrete representation of the target speech and decodes it into the target speech. Since the relative advantage of candidate groups and policy constraints have already been involved in model updates during the training phase, the reward calculation and policy optimization services from the training phase can be omitted during the inference phase, thus directly outputting the target speech subject to fine-grained emotion control.

[0045] In engineering implementation, fine-grained emotion control information can be represented using structured tags with emotion category, emotion intensity, and the scope of the affected text; the text-speech temporal correspondence can be obtained by a duration prediction module, a forced alignment module, a duration ratio estimation module, or a combination thereof. Local emotion consistency rewards are obtained through the similarity between the emotion representation of the target segment and the prototype of the target emotion category; global speaker consistency rewards are obtained through the similarity between the speaker representations of candidate whole sentences and reference sentences.

[0046] Candidate speech is generated by a speech discrete representation generation model, which reconstructs waveforms through an acoustic decoding module and a vocoder. Group relative policy optimization updates the trainable adaptation parameters in the speech synthesis model. This implementation utilizes fine-grained emotion control range to locate target segments of candidate speech, combining local emotion rewards with global speaker rewards into a candidate group relative advantage. This relative advantage is then used for group relative policy optimization with probability ratio truncation and reference policy divergence constraints, rather than the name of a specific backbone network, emotion representation model, or vocoder.

[0047] In summary, this implementation method employs fine-grained emotion control labels aligned with text units, simultaneously using emotion category, emotion intensity, and scope as conditions for speech synthesis and as the basis for target segment localization. It also organizes auxiliary conflict training conditions through conditional recombination. Second, it locates target segments of candidate speech based on the label's scope and the text-speech temporal correspondence, using the similarity between the target segment and the target emotion category prototype to form a local emotion consistency reward, rather than solely using the entire sentence's emotion score. Third, it calculates global speaker consistency rewards for both candidate whole sentences and reference speech, ensuring that local emotion control and overall timbre are simultaneously constrained at different granularities. Fourth, it normalizes local emotion consistency rewards and global speaker consistency rewards within the same candidate group, forming a joint relative advantage according to preset weights. Fifth, it uses Group Relative Policy Optimization (GRPO) to feed the joint relative advantage back to the speech synthesis model, and limits the model's deviation from its basic speech generation capabilities and improves control stability under complex conditions by truncating the probability ratio between the current and old strategies and constraining the divergence relative to the reference strategy.

[0048] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0049] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for affect controllable speech synthesis preference alignment based on reinforcement learning, characterized in that, The method includes the following steps: Step 1: Obtain text with fine-grained emotion control labels and reference speech for maintaining speaker identity. Perform conditional reorganization on the text labels or reference speech, organize consistency conditions and auxiliary conflict training conditions. For each training condition, the old strategy generates multiple candidate speech discrete representations with text, fine-grained emotion control labels and reference speech as joint conditions, and decodes each candidate speech discrete representation into candidate speech. Step 2: Based on the scope of the fine-grained emotion control label, locate the target speech segment in the candidate speech mentioned in Step 1 and calculate the local emotion consistency reward; at the same time, calculate the global speaker consistency reward between the candidate whole sentence speech and the reference speech. Step 3: Within the candidate group corresponding to the same input condition, form a joint relative advantage based on the two types of rewards in Step 2, and use a group relative strategy optimization update of the speech synthesis strategy, including the probability ratio truncation of the current strategy and the old strategy and the divergence constraint of the reference strategy, to finally obtain an emotion-controllable speech synthesis model.

2. The reinforcement learning based affect controllable speech synthesis preference alignment method according to claim 1, characterized in that, The labels mentioned in step 1 are obtained by the preprocessing annotation process based on the training speech, the corresponding transcribed text and their time correspondence. The fine-grained emotion control labels are obtained based on the training speech, the transcribed text and their time correspondence. Continuous text units with the same emotion category and intensity are merged into an emotion control segment.

3. The reinforcement learning based affect controllable speech synthesis preference alignment method according to claim 1, characterized in that, The fine-grained emotion control tags mentioned in step 2 include at least one of emotion category, emotion intensity, and the range of text to which they apply.

4. The reinforcement learning based affect controllable speech synthesis preference alignment method according to claim 3, characterized in that, The method for calculating the local affective consistency reward in step 2 is as follows: Parse the control tags to obtain the sentiment category, sentiment intensity, and scope of each target text fragment; Based on the text boundaries of the emotion control segment and the text-speech time correspondence, the corresponding target speech segment is determined from the candidate speech; The emotional representation of the target speech segment is extracted using a pre-trained emotional representation model, and the similarity is calculated with the emotional prototype corresponding to the target emotional category to obtain the local emotional consistency reward of the candidate on the target control segment.

5. The emotion-controlled speech synthesis preference alignment method based on reinforcement learning according to claim 4, characterized in that, The text-speech time correspondence is obtained through one or any combination of duration prediction, forced alignment, and duration ratio estimation.

6. The emotion-controlled speech synthesis preference alignment method based on reinforcement learning according to claim 1, characterized in that, The candidate group mentioned in step 3 consists of multiple candidate speech samples obtained by the old speech synthesis strategy, using the same text, the same fine-grained emotion control label, and the same reference speech conditions.

7. The emotion-controlled speech synthesis preference alignment method based on reinforcement learning according to claim 6, characterized in that, Step 3 employs a group-based strategy optimization method to update the speech synthesis strategy, which includes truncation of the probability ratio between the current and old strategies and divergence constraints of the reference strategy. Based on the old strategy, generate a discrete representation group of candidate speech under the same conditions, and decode each candidate into candidate speech; The target segment is located based on the tag range, and the similarity between the target segment and the target sentiment category prototype is calculated. Calculate the speaker representation similarity between candidate whole-sentence speech and reference speech, normalize the two types of rewards within the candidate group, and form a joint relative advantage according to preset weights; The update term is constructed using the probability ratio of the current policy to the old policy and truncated. At the same time, a divergence constraint relative to the reference policy is added to update the speech synthesis policy.

8. A preference alignment system for emotion-controlled speech synthesis based on reinforcement learning, characterized in that, The system includes: The fine-grained emotion control label preparation module is used to acquire text with fine-grained emotion control labels and reference speech for maintaining speaker identity. It performs conditional reorganization on the text labels or reference speech, organizes consistency conditions and auxiliary conflict training conditions. For each training condition, the old strategy generates multiple candidate speech discrete representations with text, fine-grained emotion control labels and reference speech as joint conditions, and decodes each candidate speech discrete representation into candidate speech. The reward calculation module is used to locate the target speech segment in the candidate speech described in the fine-grained emotion control label preparation module and calculate the local emotion consistency reward based on the scope of the fine-grained emotion control label; at the same time, it calculates the global speaker consistency reward between the candidate whole sentence speech and the reference speech. The group relative strategy optimization module is used to form a joint relative advantage based on the two types of rewards in step 2 within the candidate group corresponding to the same input conditions, and to update the speech synthesis strategy by using group relative strategy optimization including truncation of the probability ratio between the current strategy and the old strategy and divergence constraint of the reference strategy, so as to finally obtain an emotion-controllable speech synthesis model.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images