Emotion control increment fine tuning method, system and device based on zero sample text-to-speech model, and medium

Abstract

The invention discloses an emotion control increment fine tuning method, system and device based on a zero sample text-to-speech model, and a medium, and the method comprises the following steps: selecting a data set, and carrying out the standardization preprocessing; the emotion label of the data set is expanded; weight matrix dimensions of an embedding layer and a head layer of the text are adjusted; initializing a newly added emotion label weight; constructing an adapter, and distinguishing an original emotion label weight and a newly added emotion label weight; training a newly-added emotion label weight, and fusing the newly-added emotion label weight with an original weight; grouping the texts, and performing cross pairing with different audios to expand the texts; carrying out differentiation training on the expanded text; and evaluating the performance of the training result. According to the method, the problems of lack of an effective incremental learning mechanism, improper word list expansion processing, low training data utilization efficiency, insufficient fine degree of a fine adjustment strategy and imperfect weight management mechanism in emotion control expansion of IndexTTS are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech technology, specifically to a method, system, device, and medium for incremental fine-tuning of emotion control based on a zero-sample text-to-speech model. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has developed rapidly. Text-to-speech (TTS) technology, as an important branch of AI, has spurred many practical applications. Zero-shot TTS technology has matured rapidly in numerous intelligent voice applications and is now being used in a growing number of fields, including intelligent customer service, audiobook production, voice assistants, and online education. Zero-shot TTS technology is divided into two modes: basic speech synthesis and emotion expression synthesis. Basic speech synthesis is typically used for standard text reading, while emotion expression synthesis is typically used for audio content creation and entertainment interaction. Zero-shot TTS technology has advantages such as personalized customization, rich vocal range, and flexible deployment. It can quickly generate speech in the style of the target speaker with only a small amount of reference audio. However, it still has significant shortcomings in emotion expression control.

[0003] Currently, zero-shot TTS technology is still under development. Existing text-to-speech systems such as IndexTTS and XTTS exhibit significant technical limitations and insufficient control capabilities when encountering scenarios requiring precise control of emotional expression in actual business applications. Currently, TTS systems primarily implement emotional expression in two ways, depending on the emotion control method: one is a post-processing approach based on emotion classification models, and the other is an end-to-end control approach based on text tags. In scenarios using emotion classification models for post-processing, the system typically generates basic speech first, and then performs secondary processing through an emotion conversion model. While this achieves some emotion control, the processing flow is complex and the effect is limited. End-to-end control using text tags requires deep modification of the original model, but existing solutions lack effective incremental learning mechanisms, easily damaging the original speech synthesis capabilities. When encountering application scenarios requiring precise emotion control, the second end-to-end control method can provide better results, but the first post-processing method will introduce significant quality loss and processing latency, leading to a poor user experience.

[0004] Therefore, in speech synthesis applications requiring precise emotion control, deep fine-tuning of existing TTS systems is typically necessary to ensure synthesis quality and user experience. Taking the widely used IndexTTS as an example, while it performs excellently in zero-shot synthesis quality, its lack of emotion control capabilities presents significant limitations when facing personalized expression needs. When generating speech with specific emotions, the system cannot understand and process emotion labels, only producing flat speech output. During fine-tuning, the traditional full-scale fine-tuning method is prone to catastrophic forgetting due to the lack of an effective incremental learning mechanism, damaging the original speech synthesis capabilities. When extending emotion control, the existing IndexTTS system, constrained by traditional fine-tuning methods, cannot effectively learn new emotional expression capabilities while maintaining its original capabilities, leading to a decline in model performance after fine-tuning.

[0005] When business requirements demand that TTS systems possess emotional expression capabilities, traditional fine-tuning solutions are ineffective, leading to issues such as system degradation of original capabilities and poor emotional control. When users encounter situations where the system's expressive capabilities are limited or emotional control is inaccurate, they will face problems such as decreased synthesis quality and unnatural expression, severely impacting user experience and application effectiveness.

[0006] Existing text-to-speech systems, especially IndexTTS, suffer from the following technical problems in emotion control extension: First, they lack effective incremental learning mechanisms. Traditional fine-tuning methods are prone to catastrophic forgetting, compromising the quality of original speech synthesis when learning new emotion expressions and failing to achieve smooth capability extension. Second, vocabulary expansion is inadequate. Existing systems lack reasonable vocabulary expansion and weight initialization strategies when adding new emotion tags, resulting in poor learning performance for new tags. Third, training data utilization is inefficient. Due to the limited number of emotion tag samples, existing methods cannot fully utilize the limited training data, affecting the model's learning performance and generalization ability. Fourth, fine-tuning strategies are not refined enough. The lack of differentiated fine-tuning strategies for different modules prevents efficient learning of new emotion control capabilities while maintaining existing ones. Fifth, the weight management mechanism is imperfect. The lack of effective weight management and fusion mechanisms when processing text embedding and weight updates in the output head layer affects fine-tuning results and model stability.

[0007] Extending the emotion control capabilities of the existing IndexTTS system, employing effective incremental learning mechanisms to maintain existing capabilities while learning new emotion expression abilities, and implementing refined fine-tuning strategies and data augmentation methods to ensure the accuracy and naturalness of emotion control and improve the application effect of the TTS system in personalized expression scenarios has become an important research topic. Existing methods mainly achieve emotion control through full fine-tuning or post-processing of emotion classification, which may damage existing capabilities or have limited effectiveness. To address these issues, techniques such as model distillation and multi-task learning are mainly used to balance capability preservation and new capability learning; however, these methods increase system complexity and perform poorly on small sample emotion data.

[0008] Therefore, the core issue that needs to be addressed is how to build an emotion-controlled text-to-speech system based on IndexTTS, employing an incremental learning mechanism to achieve precise emotion expression control while maintaining the original synthesis quality. This system should meet the practical needs of applications such as personalized speech expression, audio content creation, and entertainment interaction through effective vocabulary expansion, data augmentation, and phased fine-tuning strategies. Simultaneously, the technical feasibility and training efficiency of extending emotion control based on the existing IndexTTS architecture also need to be considered. Summary of the Invention

[0009] The purpose of this invention is to provide an incremental fine-tuning method, system, device, and medium for emotion control based on a zero-shot text-to-speech model, in order to solve the problems of existing IndexTTS in emotion control expansion, such as lack of an effective incremental learning mechanism, improper vocabulary expansion processing, low efficiency of training data utilization, insufficient fine-tuning strategy, and imperfect weight management mechanism.

[0010] To achieve the above objectives, the technical solution provided by the present invention is as follows: The first aspect of this application provides a method for incremental fine-tuning of emotion control based on a zero-shot text-to-speech model, comprising the following steps: S1: Select a text-audio aligned dataset containing emotion labels, perform normalization preprocessing on the audio including resampling, noise reduction and volume normalization, use the DiscreteVAE model to convert the audio into discrete mel codes, and verify the coding consistency. S2: Expand the sentiment labels of the text-audio alignment dataset using the SentencePiece model and assign a unique token ID to each new sentiment label; adjust the dimensions of the weight matrix of the text embedding layer and the output head layer to adapt to the vocabulary size of the expanded sentiment labels; and initialize the weights of the new sentiment label tokens using a multivariate Gaussian distribution based on the original weight statistical characteristics. S3: Build an adapter to wrap the original text embedding and head layers, and distinguish between the original sentiment tag token weights and the newly added sentiment tag token weights; selectively train the newly added sentiment tag token weights through weight mask and loss weighting mechanism, and merge the trained newly added sentiment tag token weights with the original sentiment tag token weights; S4: Group the text after weighted fusion of emotion tag tokens based on emotion tags and perform quality screening. Cross-pair the screened text with emotion tags with audio from different speakers to generate expanded text. S5: The AdamW optimizer is used to perform differential training on the embedding and head layers of the augmented text with different learning rates. The GPT2 self-attention and feedforward network, fine-tuned by LoRA, freezes the approximate weights through a low-rank adaptation method to obtain the final training result. The performance of the final training result, including speech quality, emotion expression, and retention of original abilities, is evaluated.

[0011] To optimize the above technical solution, the specific limitations also include: In step S1, the resampling step involves uniformly converting the audio to a mono format with a sampling rate of 22050Hz.

[0012] Furthermore, in step S2, the initialization of the weight of the newly added emotion tag is based on the mean and covariance of the weight of the original emotion tag tokens, which are sampled using a multivariate Gaussian distribution.

[0013] Furthermore, in step S3, the weight mask and loss weighting mechanism achieve weight separation through the following formula:

[0014] in, For the expanded new weight matrix, The weight of the original emotion tag token, To add weight to the new emotion tag token, This is the original vocabulary size. To determine the number of newly added emotion tags, The dimension of the weight vector.

[0015] Furthermore, in step S4, after completing the cross-pairing, it is necessary to limit the total number of enhanced samples for a single emotion category to no more than a preset threshold, and to balance the sample ratio of each emotion label.

[0016] In step S4, the sample size distribution of various emotion labels needs to be statistically analyzed in the text grouping to identify emotion categories with scarce samples; quality screening includes evaluating audio quality and the accuracy of emotion labeling.

[0017] In step S5, the low-rank adaptation method freezes the approximate weights by using the LoRA+ method. Different learning rates are set for the two low-rank matrices lora_A and lora_B, freezing the original pre-trained weight matrix of the target layer. The new weight matrix to be learned is approximated by multiplying the low-rank matrices lora_A and lora_B.

[0018] in, and These are low-rank matrices lora_A and lora_B, respectively. For the pre-trained weight matrix, This is the new weight matrix that needs to be learned.

[0019] The second aspect of this application provides an incremental fine-tuning system for emotion control based on a zero-shot text-to-speech model, comprising: The dataset preparation and preprocessing module is used to select a text-audio aligned dataset containing emotion labels, perform normalization preprocessing on the audio, including resampling, noise reduction and volume normalization, use the DiscreteVAE model to convert the audio into discrete mel codes, and verify the coding consistency. The vocabulary expansion and weight management module is used to expand the sentiment labels of the text-audio alignment dataset using the SentencePiece model and assign a unique token ID to each new sentiment label; adjust the dimensions of the weight matrix of the text embedding layer and the output head layer to adapt to the vocabulary size of the expanded sentiment labels; and initialize the weights of the new sentiment label tokens using a multivariate Gaussian distribution based on the original weight statistical characteristics. An adapter is used to wrap the original embedding and head layers of the text, distinguishing between the original sentiment tag token weights and the newly added sentiment tag token weights; the newly added sentiment tag token weights are selectively trained through a weight mask and loss weighting mechanism, and the trained newly added sentiment tag token weights are then fused with the original sentiment tag token weights. The data cross-matching module is used to group the text after the weighted fusion of emotion tag tokens according to emotion tags and perform quality screening. The screened text with emotion tags is cross-matched with audio from different speakers to generate expanded text. The fine-tuning and optimization module is used to perform differential training on the embedding and head layers of the augmented text using the AdamW optimizer at different learning rates. The GPT2 self-attention and feedforward network fine-tuned by LoRA freezes the approximate weights through a low-rank adaptation method to obtain the final training result. The performance of the final training result is evaluated, including speech quality, emotion expression, and retention of original abilities.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the emotion control incremental fine-tuning method based on a zero-sample text-to-speech model as described above.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the computer program causing a computer to perform the emotion control incremental fine-tuning method based on a zero-sample text-to-speech model as described above.

[0022] Compared with the prior art, the beneficial effects of the present invention are: This invention provides an incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model. By constructing an adapter, it provides an effective incremental learning mechanism, learning emotional expression capabilities while maintaining the original speech synthesis quality. This avoids the catastrophic forgetting problem caused by traditional full-scale fine-tuning. A weight initialization strategy based on statistical principles ensures that newly added emotion tag tokens obtain reasonable initial representations, accelerating model convergence. A robust weight management mechanism is formed through data cross-matching enhancement strategies. Limited emotion tag samples are paired with different reference audios, effectively expanding the training samples and significantly improving the model's learning effect and convergence speed. A staged fine-tuning method is used, first training the text embedding and head layers, then performing GPT2 LoRA fine-tuning. By isolating the training focus, the catastrophic forgetting problem is further suppressed, and gradient conflicts caused by simultaneously optimizing a large number of parameters are avoided, solving the problems of instability and poor performance during training. Attached Figure Description

[0023] Figure 1 : A flowchart illustrating the incremental fine-tuning method for emotion control based on a zero-sample text-to-speech model of the present invention.

[0024] Figure 2 The module architecture diagram of IndexTTS, an incremental fine-tuning method for emotion control based on a zero-sample text-to-speech model, is shown in this invention.

[0025] Figure 3 : LoRA fine-tuning pattern diagram of the emotion control incremental fine-tuning method based on zero-sample text-to-speech model of the present invention.

[0026] Figure 4 The loss function graph of the emotion control incremental fine-tuning method based on the zero-shot text-to-speech model of the present invention when fine-tuning GPT2. Detailed Implementation

[0027] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0028] The following is an explanation of some of the terms used in this invention: IndexTTS: IndexTTS is an open-source zero-shot text-to-speech model developed by the Index team on Bilibili. Based on the Tortoise TTS and XTTS architectures, it supports zero-shot speech cloning in both Chinese and English. The system uses a GPT2 autoregressive model to generate latent audio representations and DiscreteVAE to perform discrete encoding of the Mel spectrum. It features a small parameter set and supports pinyin tone control.

[0029] Zero-shot TTS: Zero-shot text-to-speech technology refers to a system that can synthesize speech in the style of a target speaker using only a small amount of reference audio, typically a few seconds to a few minutes, without requiring a large amount of training data. This technology significantly reduces the barrier and cost of personalized speech synthesis.

[0030] NewTokensTrainableAdapter: This invention proposes an incremental learning adapter for wrapping existing text embeddings and output head layers. This adapter freezes the weight parameters of the original tokens and only trains the weights related to newly added emotion tags, achieving selective parameter updates and avoiding damage to the original speech synthesis capabilities.

[0031] LoRA (Low-Rank Adaptation) is an efficient model fine-tuning technique that approximates weight updates by introducing a low-rank matrix into the linear layers of a pre-trained model. LoRA can significantly reduce the number of trainable parameters, improve fine-tuning efficiency, and maintain model performance.

[0032] Cross-matching data augmentation: The data augmentation strategy proposed in this invention generates new training samples by cross-matching text with emotion tags with different reference audio. This method can effectively expand the limited number of emotion tag samples and improve the model's learning performance.

[0033] DiscreteVAE: Discrete Variational Autoencoder, used to encode continuous Mel spectral features into discrete token sequences. In IndexTTS, DiscreteVAE is responsible for converting the Mel spectral representation of audio into discrete codes that can be processed by the GPT2 model.

[0034] BPE (Byte Pair Encoding) is a data compression and text segmentation technique that constructs a vocabulary by iteratively merging the most frequent character pairs. In this invention, BPE is used to encode text and newly added sentiment tags into a token sequence.

[0035] SentencePiece: SentencePiece is an unsupervised text segmenter developed by Google that supports BPE and Unigram language model algorithms. This invention uses SentencePiece to expand the vocabulary and add new sentiment tag tokens.

[0036] The technical solution of the present invention will be further described in detail below with reference to specific embodiments: In one embodiment, this invention provides a method for incremental fine-tuning of emotion control based on a zero-shot text-to-speech model, the flowchart of which is shown below. Figure 1 As shown, the entire method includes the following steps: S1: Select a text-audio aligned dataset containing emotion labels, perform normalization preprocessing on the audio including resampling, noise reduction and volume normalization, use the DiscreteVAE model to convert the audio into discrete mel codes, and verify the coding consistency. S2: Expand the sentiment labels of the text-audio alignment dataset using the SentencePiece model and assign a unique token ID to each new sentiment label; adjust the dimensions of the weight matrix of the text embedding layer and the output head layer to adapt to the vocabulary size of the expanded sentiment labels; and initialize the weights of the new sentiment label tokens using a multivariate Gaussian distribution based on the original weight statistical characteristics. S3: Build an adapter to wrap the original text embedding and head layers, and distinguish between the original sentiment tag token weights and the newly added sentiment tag token weights; selectively train the newly added sentiment tag token weights through weight mask and loss weighting mechanism, and merge the trained newly added sentiment tag token weights with the original sentiment tag token weights; S4: Group the text after weighted fusion of emotion tag tokens based on emotion tags and perform quality screening. Cross-pair the screened text with emotion tags with audio from different speakers to generate expanded text. S5: The AdamW optimizer is used to perform differential training on the embedding and head layers of the augmented text with different learning rates. The GPT2 self-attention and feedforward network, fine-tuned by LoRA, freezes the approximate weights through a low-rank adaptation method to obtain the final training result. The performance of the final training result, including speech quality, emotion expression, and retention of original abilities, is evaluated.

[0037] In step S1, during dataset selection, it's crucial to ensure the dataset contains sufficient emotion label samples, focusing on commonly used emotion expression labels such as LAUGHS, GIGGLES, SIGHS, and CHUCKLES. Simultaneously, audio quality must be screened to ensure clarity and accuracy of emotion labels. After dataset selection, preprocessing operations such as resampling, noise reduction, and volume normalization are required. Resampling involves uniformly converting the audio to a mono format with a sampling rate of 22050Hz. Once audio preprocessing is complete, the system uses a pre-trained DiscreteVAE model to convert the audio into discrete mel codes, establishing a mapping between audio and discrete tokens. DiscreteVAE encoding must maintain consistency with the original IndexTTS encoding method to ensure the encoded mel codes can be correctly processed by the GPT2 model. After encoding, the system possesses the basic capability to process emotional audio data.

[0038] In step S2, the SentencePiece model is expanded to support the encoding and decoding of emotion tags, thereby expanding the emotion tag vocabulary. During vocabulary expansion, the existing BPE model is loaded and new emotion tag tokens are added, including commonly used emotion expression tags such as LAUGHS, GIGGLES, SIGHS, and CHUCKLES. A unique token ID is assigned to each new tag to ensure the correctness of encoding and decoding. After vocabulary expansion, the dimensions of the weight matrix in the text-related layers of the model are adjusted accordingly, expanding the weight matrices of the text embedding layer and head layer from their original size to the new vocabulary size.

[0039] like Figure 2As shown, IndexTTS adopts the UnifiedVoice model structure of Tortoise TTS, and feeds its own inputs_embeds into the GPT2 forward propagation inference. In the GPT2LMHeadModel, wte is the text token embedding layer, and lm_head is the text token output head; their weights are shared. In the UnifiedVoice model, text_embedding is the text token embedding layer, and text_head is the text token output head layer; their weights are independent and need to be adjusted separately.

[0040] After dimensionality adjustment, the system initializes the newly added weight parameters appropriately, employing an initialization strategy based on the statistical characteristics of the original weights. The initialization of the new weights involves multivariate Gaussian distribution sampling based on the mean and covariance of the original weights to ensure that the new tokens receive a reasonable initial representation. Once initialized, the system possesses the basic capability to process sentiment tags.

[0041] In step S3, the incremental learning process of NewTokensTrainableAdapter includes adapter construction, selective training, and weight fusion management. The system creates a NewTokensTrainableAdapter incremental learning adapter, which wraps the independent text embedding layer and output head layer in the IndexTTS model, respectively. The weights of the original sentiment tag tokens are explicitly separated from the weights of the newly added sentiment tag tokens. Through parameter freezing and selective training mechanisms, the system achieves independent updating and management of the weights of the newly added tokens.

[0042] The adapter design process needs to inherit the interfaces of the original modules to ensure compatibility with the existing training process, including core interfaces such as the forward method and parameter management method; weight separation achieves selective training through a parameter masking mechanism, freezing the weights of the original sentiment tag tokens and only training the weights of the newly added sentiment tag tokens.

[0043] in, For the expanded new weight matrix, The weight of the original emotion tag token, To add weight to the new emotion tag token, This is the original vocabulary size. To determine the number of newly added emotion tags, The dimension of the weight vector.

[0044] During selective training, the parameters related to emotion labels are optimized through weight masks and loss weighting mechanisms. The loss at the position of newly added emotion label tokens is weighted to accelerate the learning and convergence of newly added emotion label tokens.

[0045] After selective training, the system merges the weights of the newly trained emotion tag tokens with the weights of the original emotion tags to unify the model parameters. Weight fusion must maintain the integrity of the original weights, updating only the weight parameters corresponding to the newly added emotion tag tokens to ensure that the original speech synthesis capabilities are not affected. After fusion, the system possesses complete emotion control capabilities.

[0046] In step S4, the sample quantity distribution of various emotion labels is statistically analyzed during the sample grouping process to identify emotion categories with scarce samples, providing a basis for subsequent cross-matching; at the same time, the quality of the original samples is evaluated to ensure that the samples used for cross-matching have good audio quality and accurate emotion labeling.

[0047] Based on the zero-shot audio cloning capability of the IndexTTS model, after sample grouping, text with emotion labels is cross-paired with different reference audio. Specifically, each emotion-labeled text is paired with reference audio from multiple different speakers to generate new training samples. The original audio is retained as the reference speech, and its corresponding text and melcodes are replaced with samples containing emotion labels. This audio-text decoupling and recombination method strengthens the model's learning of the mapping relationship between emotion labels and audio features.

[0048] After cross-pairing, the generated augmented samples undergo quality control and quantity management to avoid overfitting due to excessive augmentation. Too many training epochs and repetitive data can impair model performance, causing the model to ignore timbre features such as emotion, speech rate, and pitch from the reference audio, focusing only on the correspondence between text and mel codes. Therefore, the amount of augmented data needs to be controlled based on the distribution of the original samples, limiting the total number of augmented samples for a single emotion category to no more than a preset threshold to ensure a relatively balanced number of samples for each emotion label.

[0049] In step S5, the phased fine-tuning optimization includes text layer fine-tuning, GPT2 LoRA fine-tuning, and model performance evaluation.

[0050] In the UnifiedVoice model structure of IndexTTS, the text embedding layer and head layer are external parameter modules independent of the GPT2 Transformer backbone network. This allows the text representation layer to be learned independently without affecting the main GPT2 learning, enabling GPT2 to focus more on learning the generation of mel codes. By fine-tuning the text embedding layer and head layer in advance, the newly added emotion tag tokens obtain semantically reasonable and consistent initial vector representations, providing better initial conditions for the subsequent fine-tuning of the core module GPT2, accelerating the overall training convergence and improving the modeling accuracy of emotion control.

[0051] The parameters of NewTokensTrainableAdapter are optimized using the AdamW optimizer. Different learning rate strategies are employed to differentiate the weights and bias parameters of the text embedding and head layers. During this process, the convergence of the loss of newly added tokens is monitored to ensure the learning effect of sentiment labels.

[0052] After fine-tuning the text layer, low-rank adaptive fine-tuning is performed on the self-attention and feedforward network parts of GPT2, such as... Figure 3 As shown, the LoRA+ method is used to set different learning rates for the two low-rank matrices lora_A and lora_B, improving training efficiency and effectiveness. The original pre-trained weight matrix of the target layer is frozen, and the new weight matrix to be learned is approximated by multiplying the low-rank matrices lora_A and lora_B.

[0053] in, and These are low-rank matrices lora_A and lora_B, respectively. For the pre-trained weight matrix, This is the new weight matrix that needs to be learned. For example... Figure 4 As shown, the loss curve of GPT2 is fine-tuned using LoRA with 1200 samples after cross-matching. The new weights and configurations are saved at appropriate stages.

[0054] After LoRA fine-tuning, the system performs a comprehensive performance evaluation of the tuned model, including the retention of original capabilities and the effect of adding emotion control capabilities. The performance evaluation covers key indicators such as speech quality, accuracy of emotion expression, and the degree of retention of original capabilities. Upon completion of the evaluation, the system possesses complete emotion-controlled TTS capabilities.

[0055] In another embodiment of the present invention, an incremental fine-tuning system for emotion control based on a zero-shot text-to-speech model is proposed, comprising: The dataset preparation and preprocessing module is used to select a text-audio aligned dataset containing emotion labels, perform normalization preprocessing on the audio, including resampling, noise reduction and volume normalization, use the DiscreteVAE model to convert the audio into discrete mel codes, and verify the coding consistency. The vocabulary expansion and weight management module is used to expand the sentiment labels of the text-audio alignment dataset using the SentencePiece model and assign a unique token ID to each new sentiment label; adjust the dimensions of the weight matrix of the text embedding layer and the output head layer to adapt to the vocabulary size of the expanded sentiment labels; and initialize the weights of the new sentiment label tokens using a multivariate Gaussian distribution based on the original weight statistical characteristics. An adapter is used to wrap the original embedding and head layers of the text, distinguishing between the original sentiment tag token weights and the newly added sentiment tag token weights; the newly added sentiment tag token weights are selectively trained through a weight mask and loss weighting mechanism, and the trained newly added sentiment tag token weights are then fused with the original sentiment tag token weights. The data cross-matching module is used to group the text after the weighted fusion of emotion tag tokens according to emotion tags and perform quality screening. The screened text with emotion tags is cross-matched with audio from different speakers to generate expanded text. The fine-tuning and optimization module is used to perform differential training on the embedding and head layers of the augmented text using the AdamW optimizer at different learning rates. The GPT2 self-attention and feedforward network fine-tuned by LoRA freezes the approximate weights through a low-rank adaptation method to obtain the final training result. The performance of the final training result is evaluated, including speech quality, emotion expression, and retention of original abilities.

[0056] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described incremental fine-tuning method for emotion control based on a zero-sample text-to-speech model.

[0057] In another embodiment of the present invention, a computer-readable storage medium is proposed, storing a computer program that causes a computer to execute the above-described incremental fine-tuning method for emotion control based on a zero-sample text-to-speech model.

[0058] The technical solution of the present invention will be further described below in conjunction with specific implementation processes: The system selects open-source datasets such as MrDragonFox / Elise, which contain rich emotion annotations, ensuring that the datasets include emotion tags such as LAUGHS, GIGGLES, SIGHS, and CHUCKLES. The original audio is quality-screened, removing samples with excessive noise or inaccurate annotations. The audio is then uniformly converted to a mono format with a sampling rate of 22050Hz and volume normalized. A pre-trained DiscreteVAE model is used to convert the audio into discrete mel codes. The correspondence between the encoded mel codes and the original audio is verified to ensure encoding quality, completing the audio data preparation process. Text transcription and corresponding emotion tag information are extracted from the dataset. Finally, a triplet correspondence between text, emotion tags, and audio mel codes is established, completing the dataset preparation.

[0059] Load the existing SentencePiece BPE model and obtain the current vocabulary size and token mapping relationship; add new sentiment tag tokens, including LAUGHS, GIGGLES, SIGHS, CHUCKLES, etc.; assign a unique token ID to each new tag and update the vocabulary mapping relationship; save the expanded BPE model and verify the correctness of the encoding and decoding functions; test the encoding and decoding effect of the new sentiment tags to ensure that the function is normal and complete the vocabulary expansion; then, obtain the embedding and head layer weight matrix dimension information of the original text; adjust the weight matrix dimension according to the expanded vocabulary size to expand the corresponding parameter space; calculate the statistical characteristics of the original weights, including mean and covariance information; initialize the new weights based on a multivariate Gaussian distribution to ensure a reasonable initial representation.

[0060] Create a `NewTokensTrainableAdapter` class to wrap the original text's embedding and head layers, achieving weight separation and clearly distinguishing the weights of the original sentiment tag tokens from those of the newly added sentiment tag tokens. A weight freezing strategy ensures that the original sentiment tag token weights remain unchanged during training. A parameter mask is configured to selectively train the weights of the newly added sentiment tag tokens. The adapter's functionality is then verified to ensure compatibility with the existing training process, completing the adapter construction process. Loss weighting is configured to optimize the loss at the positions of the newly added sentiment tag tokens. The learning convergence of the newly added sentiment tag token weights is monitored, and the training strategy is adjusted accordingly. The trained weights of the newly added tokens are then fused with the original weights. Finally, the functional integrity of the fused model is verified to ensure the coordinated operation of the original and newly added capabilities.

[0061] The study involves statistically analyzing the sample size distribution of various emotion labels to identify emotion categories with scarce samples; evaluating the quality of the original samples and selecting high-quality samples for cross-pairing; establishing a sample index and classification management mechanism to provide a data foundation for cross-pairing; pairing text with emotion labels with reference audio from different speakers; generating new text-audio training samples to expand the training data scale; and controlling the amount of augmented data to avoid overfitting due to excessive augmentation and ensuring a relatively balanced sample size for each emotion label.

[0062] Configure the AdamW optimizer, setting appropriate learning rates and weight decay parameters; specifically train the parameters of the NewTokensTrainableAdapter; differentiate the training of weights and bias parameters using different learning rate strategies; monitor the loss convergence of newly added tokens to ensure the system's learning effect; evaluate the effect of text layer fine-tuning, verify the representation quality of sentiment labels, and complete the text layer fine-tuning process; configure the LoRA adapter, setting the dimension parameters of the low-rank matrix; fine-tune the self-attention and feedforward network parts of GPT2; use the LoRA+ method to set different learning rates for the lora_A and lora_B matrices; learn the correspondence between sentiment labels and audio mel codes; evaluate the speech quality and emotional expression accuracy of the fine-tuned model; finally, verify the degree of preservation of the original speech synthesis capabilities, ensuring no catastrophic forgetting, and complete the fine-tuning optimization.

[0063] This invention establishes an emotion-controlled speech synthesis generation method and system for the IndexTTS system through the deep application of incremental learning mechanisms. It achieves an emotion-controlled TTS solution with more accurate emotion expression, better preservation of original capabilities, and higher training efficiency. Furthermore, it extends the capabilities based on the existing IndexTTS architecture, improving the emotion expression ability of the IndexTTS system. It also has a significant beneficial effect on enhancing the application value of TTS systems in scenarios such as entertainment interaction and audio content creation, thus filling the gap in emotion control extension schemes based on incremental learning of IndexTTS.

[0064] This invention establishes a complete incremental learning service process using NewTokensTrainableAdapter, constructing a comprehensive technical workflow based on selective weight training, weight fusion management, and gradient control optimization. This workflow is used to learn new emotional expression functions while maintaining the original speech synthesis capabilities. It integrates four key steps: weight freezing, incremental training, weight merging, and gradient optimization, embodying the technical process of incremental learning and achieving zero-catastrophic forgetting capability expansion, demonstrating completeness and originality. Through the NewTokensTrainableAdapter adapter mechanism, it resolves the conflict between new functions and existing capabilities, and achieves smooth expansion of model capabilities through refined weight management.

[0065] Furthermore, this invention features a data augmentation cross-matching process, employing intelligent sample augmentation strategies based on the sample distribution and quantity limitations of different emotion labels. By cross-pairing emotion label text with different reference audio, the number of training samples is significantly increased, effectively addressing the problem of scarce emotion label samples. Simultaneously, a data augmentation mechanism based on sample grouping and pairing control is constructed to avoid overfitting risks caused by excessive augmentation, ensuring the model's generalization ability and learning performance.

[0066] This invention also features a phased fine-tuning and optimization process, achieving stable and efficient model training and parameter optimization. The system supports a strategy of training the text representation layer first and then fine-tuning the core GPT2 module. It significantly reduces the number of training parameters through LoRA low-rank adaptation technology and optimizes the learning rate setting using the LoRA+ method. Based on the fine-tuning results, emotional speech synthesis is performed, switching to different emotional expression strategies. When the user requires a specific emotional expression, the system accurately generates the corresponding emotional audio; when the user requires natural speech, the system maintains the original synthesis quality, greatly improving the accuracy and diversity of emotional expression in the TTS system. The fine-tuned model can accurately understand and generate various emotional expressions, including laughter, sighs, and light laughs, significantly improving the naturalness and accuracy of emotional expression. This provides a complete technical solution for the application of TTS technology in personalized expression, audio content creation, and entertainment interaction.

[0067] In the embodiments disclosed in this application, the computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the present invention and based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for incremental fine-tuning of emotion control based on a zero-shot text-to-speech model, characterized in that, Includes the following steps: S1: Select a text-audio aligned dataset containing emotion labels, perform normalization preprocessing on the audio including resampling, noise reduction and volume normalization, use the DiscreteVAE model to convert the audio into discrete mel codes, and verify the coding consistency. S2: Expand the sentiment labels of the text-audio alignment dataset using the SentencePiece model and assign a unique token ID to each new sentiment label; adjust the dimensions of the weight matrix of the text embedding layer and the output head layer to adapt to the vocabulary size of the expanded sentiment labels; and initialize the weights of the new sentiment label tokens using a multivariate Gaussian distribution based on the original weight statistical characteristics. S3: Build an adapter to wrap the original text embedding and head layers, and distinguish the weight of the original sentiment tag tokens from the weight of the newly added sentiment tag tokens; The newly added emotion tag token weights are selectively trained using a weight mask and loss weighting mechanism, and then the trained newly added emotion tag token weights are merged with the original emotion tag token weights. S4: Group the text after weighted fusion of emotion tag tokens based on emotion tags and perform quality screening. Cross-pair the screened text with emotion tags with audio from different speakers to generate expanded text. S5: The AdamW optimizer is used to perform differential training on the embedding and head layers of the augmented text with different learning rates. The GPT2 self-attention and feedforward network, fine-tuned by LoRA, freezes the approximate weights through a low-rank adaptation method to obtain the final training result. The performance of the final training result, including speech quality, emotion expression, and retention of original abilities, is evaluated.

2. The incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model according to claim 1, characterized in that: In step S1, the resampling step involves uniformly converting the audio to a mono format with a sampling rate of 22050Hz.

3. The incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model according to claim 1, characterized in that: In step S2, the initialization of the weight of the newly added emotion tag is based on the mean and covariance of the weight of the original emotion tag tokens, which are sampled by a multivariate Gaussian distribution.

4. The incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model according to claim 1, characterized in that: In step S3, the weight mask and loss weighting mechanism achieve weight separation through the following formula: in, For the expanded new weight matrix, The weight of the original emotion tag token, To add weight to the new emotion tag token, This is the original vocabulary size. To determine the number of newly added emotion tags, The dimension of the weight vector.

5. The incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model according to claim 1, characterized in that: In step S4, after the cross-pairing is completed, it is necessary to limit the total number of enhanced samples for a single emotion category to no more than a preset threshold, and to balance the sample ratio of each emotion label.

6. The incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model according to claim 1, characterized in that: In step S4, the sample size distribution of various emotion labels needs to be statistically analyzed in the text grouping to identify emotion categories with scarce samples; quality screening includes evaluating audio quality and the accuracy of emotion labeling.

7. The incremental fine-tuning method for emotion control based on a zero-shot text-to-speech model according to claim 1, characterized in that: In step S5, the low-rank adaptation method freezes the approximate weights by using the LoRA+ method. Different learning rates are set for the two low-rank matrices lora_A and lora_B, freezing the original pre-trained weight matrix of the target layer. The new weight matrix to be learned is approximated by multiplying the low-rank matrices lora_A and lora_B. in, and These are low-rank matrices lora_A and lora_B, respectively. For the pre-trained weight matrix, This is the new weight matrix that needs to be learned.

8. An incremental fine-tuning system for emotion control based on a zero-shot text-to-speech model, characterized in that, include: The dataset preparation and preprocessing module is used to select a text-audio aligned dataset containing emotion labels, perform normalization preprocessing on the audio, including resampling, noise reduction and volume normalization, use the DiscreteVAE model to convert the audio into discrete mel codes, and verify the coding consistency. The vocabulary expansion and weight management module is used to expand the sentiment labels of the text-audio alignment dataset using the SentencePiece model and assign a unique token ID to each new sentiment label; it also adjusts the dimensions of the weight matrix of the text embedding layer and the output head layer to accommodate the expanded vocabulary size of the sentiment labels. Based on the original weight statistical characteristics, a multivariate Gaussian distribution is used to initialize the weights of newly added sentiment tag tokens. The adapter is used to wrap the original embedding and head layers of the text, and to distinguish the weight of the original sentiment tag tokens from the weight of the newly added sentiment tag tokens. The newly added emotion tag token weights are selectively trained using a weight mask and loss weighting mechanism, and then the trained newly added emotion tag token weights are merged with the original emotion tag token weights. The data cross-matching module is used to group the text after the weighted fusion of emotion tag tokens according to emotion tags and perform quality screening. The screened text with emotion tags is cross-matched with audio from different speakers to generate expanded text. The fine-tuning and optimization module is used to perform differential training on the embedding and head layers of the augmented text using the AdamW optimizer at different learning rates. The GPT2 self-attention and feedforward network fine-tuned by LoRA freezes the approximate weights through a low-rank adaptation method to obtain the final training result. The performance of the final training result is evaluated, including speech quality, emotion expression, and retention of original abilities.

9. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the emotion control incremental fine-tuning method based on a zero-sample text-to-speech model as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute the emotion control incremental fine-tuning method based on a zero-sample text-to-speech model as described in any one of claims 1-7.

Images