Multi-round interaction emotion speech synthesis method and system based on context self-adaption

Abstract

The invention provides a multi-round interactive emotion speech synthesis method and system based on context self-adaption, and belongs to the technical field of speech synthesis, and the method comprises the steps: obtaining an instant multi-round dialogue with an adjacent single-round historical speech as a context window and a current to-be-synthesized text; directly deconstructing and predicting emotional and multi-dimensional side language acoustic features from historical voice signals through a two-stage trained context adaptive feature predictor; through a feature parameter mapping mechanism of a preset standardized template, the prediction features are converted into standardized control parameters available for synthesis; based on the standardized parameters and the to-be-synthesized text, driving a speech synthesis model to generate natural and coherent target speech; the system comprises a data acquisition module, a context adaptive feature prediction module, a feature parameter mapping module and a speech synthesis module. According to the method, dual optimization of emotion accuracy and voice naturalness in a multi-round interaction scene is realized, and the comprehensive performance of acoustic quality and emotion expression ability of the synthesized voice is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a multi-turn interactive emotional speech synthesis method and system based on context adaptation. Background Technology

[0002] Text-to-speech (TTS) synthesis technology, as a core support for human-computer voice interaction, has been widely used in numerous scenarios such as intelligent assistants, intelligent customer service, and in-vehicle terminals. Its technological development focuses on improving the naturalness of speech and the consistency of emotional expression. In static single-sentence synthesis scenarios, with advancements in end-to-end modeling and large language model fusion, representative models such as FastSpeech2 and Valll-E have achieved high levels of naturalness in speech. Some models have even begun to possess the ability to adjust emotion and style by introducing feature control mechanisms. However, in the typical interactive scenario of multi-turn dialogue, speech synthesis still struggles to fully utilize historical information, affecting context matching.

[0003] Current research on multi-turn interactive speech synthesis mainly attempts to utilize historical contextual information through two approaches: one is to rely on Automatic Speech Recognition (ASR) technology to transcribe historical speech into text, and then extract emotional features from the text using sentiment analysis models, integrating them into the subsequent speech synthesis process, such as Audiogpt and other dialogue-based text-to-speech (TTS) systems. However, the ASR-based approach inevitably loses fine-grained paralinguistic features from the original speech, such as tone, intonation, pauses, and stress. These features are the core carriers of emotional expression and contextual fit. The text transcription process is prone to loss or ambiguity of emotional information, thus affecting contextual understanding in multi-turn dialogues. The first issue is the accuracy and coherence of the synthesized speech. The second is to draw on the idea of ​​feature-controlled speech synthesis, which uses text prompts and reference audio style embedding vectors to regulate the emotion and prosody of the synthesized speech, such as the Mellotron and PrompTTS models. The core of these models is to explicitly control the style and emotion of a single sentence. However, most controllable speech synthesis methods are designed for isolated sentences and lack dynamic emotion modeling and paralinguistic feature adaptive mechanisms for multi-turn interaction scenarios. They cannot achieve the natural evolution of emotional state and the continuous inheritance of contextual information during the dialogue process, resulting in synthesized speech that is difficult to fit the contextual logic of multi-turn dialogues, and problems such as abrupt emotions and disjointed prosody.

[0004] In addition, existing multi-turn interactive speech synthesis systems are also insufficient in terms of the efficiency of acoustic feature utilization. Even if some models attempt to introduce historical information, the lack of effective feature prediction and standardized mapping mechanisms makes it difficult to efficiently convert the emotional and paralinguistic features in the historical context into control parameters that the synthesis system can directly utilize, further limiting the naturalness and emotional fit of the synthesized speech. Summary of the Invention

[0005] The purpose of this invention is to provide a context-adaptive multi-turn interactive emotional speech synthesis method and system. Based on historical dialogue memory, it dynamically adjusts acoustic features to achieve dynamic adaptive speech synthesis of emotion and prosody under multi-turn interaction, solving the problems of fine-grained emotional information loss, parameter disorder caused by lack of standardization of feature parameter mapping, and insufficient coherence inheritance of emotional and paralinguistic features under multi-turn context in the prior art.

[0006] To achieve the above objectives, this invention proposes a context-adaptive multi-turn interactive emotional speech synthesis method, comprising the following steps: Step S1: In an instant multi-turn dialogue scenario with adjacent single-turn historical speech as the context window, obtain the historical speech signal sequence and the current text to be synthesized in the multi-turn dialogue. The historical speech signal sequence does not need to undergo speech-to-text ASR processing. Step S2: Select a basic audio model and construct a context-adaptive feature predictor through two-stage training. This predictor adapts to the context and forecasts the emotional and paralinguistic features of the response speech, constructing and outputting an initial acoustic feature set. The two-stage training specifically involves: The first stage of training involves constructing a manually labeled dataset containing acoustic features, specifically: basic features, emotional features, and paralinguistic features. Quantitative labeling standards are established to standardize the labeling of each type of feature. All labels are presented in natural language. The audio model is then fine-tuned using the manually labeled dataset until it possesses the ability to deconstruct the feature information of a single speech segment. The mathematical expression for this is: ; in, The text is a manually annotated natural language description. For input voice signal, This is the first-stage model used for deconstructing single speech feature information; The second stage of training involves inputting the incremental dialogue dataset into the audio model trained in the first stage. The feature information is deconstructed to obtain the acoustic features of each speech segment. Then, using neighboring single-turn speech segments as context windows, a sliding window is used to construct a historical speech-response speech feature pair dataset. This dataset is then used to incrementally train the audio model until it can predict the initial acoustic features of the response speech based on historical speech. The mathematical expression is: ; in, To recover the initial acoustic features of speech, For the second-stage model used for context prediction, To reply to the voice message, To and The previous round of voice messages from the nearest neighbor, i.e., the voice messages in response. Historical voice recordings; Step S3: Establish a feature parameter mapping mechanism that includes pre-built mapping rule templates and feature field parsing. Input the initial acoustic feature set into the feature parameter mapping mechanism and output standardized acoustic control parameters. This includes the following steps: Step S31: Optimize the parameters of the feature parameter mapping mechanism. Generate multiple natural language prompt word variants with the same semantics through the frozen text large model. Manually screen and supplement them. Use Mel-Cepstral Distortion (MCD) as the quantitative evaluation standard to score and rank the similarity between the speech samples generated by different prompt word variants and the original speech. After no less than one round of testing, determine the optimal prompt word. Step S32: Construct a mapping rule template based on the optimal prompt word, and use the mapping rule template to define a standardized mask mapping Slot-Filling paradigm, including a global acoustic state template, a local sublingual feature event anchoring template, and an accent control template; Step S33: The feature field parsing module extracts key fields from the input initial acoustic features, including feature variables, occurrence positions and durations. Then, the optimal mapping rule template maps these fields to predefined mask positions in a structured manner, generating control signals to control subsequent speech synthesis and outputting standardized acoustic control parameters. Step S4: Combine the standardized acoustic control parameters with the response text input speech synthesis model to generate emotionally coherent natural speech that fits the context.

[0007] Preferably, in step S2, the basic audio model is Qwen-audio, and the LoRA fine-tuning method is used for training.

[0008] Preferably, in step S21, the acoustic features and quantitative annotation standards in the manually labeled dataset are as follows: In the basic characteristics, speaking speed is divided into words per minute (WPM), where: speaking speed ≤ 130 WPM is slow; 130 WPM < speaking speed < 160 WPM is medium; and speaking speed ≥ 161 WPM is fast. Pitch is set according to the average fundamental frequency to distinguish gender: male: low fundamental frequency ≤ 100Hz, 101Hz < mid fundamental frequency < 140Hz, high fundamental frequency ≥ 141Hz; female: low fundamental frequency ≤ 180Hz, 181Hz < mid fundamental frequency < 240Hz, high fundamental frequency ≥ 241Hz. Emotional traits are based on valence Activation level With Dominance The 5-point scale is used for quantitative definition, specifically: ; Among them, other categories must meet the consistency judgment of no less than 70% of the annotators; The time-frequency detection thresholds for paralinguistic features are set as follows: laughter duration ≥ 200ms, with significant harmonic or burst structures; breathing sound duration ≥ 150ms, with obvious inhalation or exhalation segments without a sound source; sighing sound duration ≥ 300ms, accompanied by significant energy attenuation and a fundamental frequency downward trajectory. Among them, the start and end positions of the sub-language features are marked, and the duration accuracy is retained to 0.001s.

[0009] Preferably, in step S31, the MCD reflects the difference in spectral features by calculating the Euclidean distance between the synthesized speech and the reference speech on the Mel-frequency cepstral coefficients (MFCCs). The lower the value, the closer the generated speech is to the original speech in terms of timbre and prosody. The calculation formula is as follows: ; in, In the first Frame number Mel-spectral coefficients of the original speech, For the first Frame number Mel-spectral coefficients of synthesized speech in dimensionality. The number of feature dimensions. Total number of frames Used to convert units to decibels.

[0010] Preferably, in step S32, the global acoustic state template uses a fixed sentence structure: The speed of speech is The pitch is , and the emotion is It carries parameters such as speaking speed, pitch, and emotion category; The local sublingual feature event anchoring template adopts: "The comes after the word duration: "seconds" maps the trigger location and duration of laughter, breathing sounds, etc. The accent control template uses: "Emphasize word" "" specifies the position of the accent.

[0011] Preferably, in step S33, the mathematical expression for the mapping layer is: in, For mapping rules, For the feature extraction process, Control signals for subsequent speech synthesis.

[0012] Preferably, in step S4, the speech synthesis model is CosyVoice2, and the selected pre-trained model version is CosyVoice2-0.5B. The CosyVoice2-0.5B pre-trained model itself has been fine-tuned with prompt words and supports the joint driving of speech generation by natural language prompt words and standardized acoustic control parameters.

[0013] The present invention also provides a context-adaptive multi-turn interactive emotional speech synthesis system, including a data acquisition module, a context-adaptive feature prediction module, a feature parameter mapping module, and a speech synthesis module; The data acquisition module includes a historical voice acquisition unit and a response text receiving unit; The context-adaptive feature prediction module includes a first-stage training unit and a second-stage training unit; The feature parameter mapping module includes a prompt word optimization and evaluation unit, a mapping rule template construction unit, and a feature field parsing and structured mapping unit; The speech synthesis module includes an input fusion unit, an emotion-prosodic modulation unit, and a speech generation unit.

[0014] Preferably, the data acquisition module is communicatively connected to the context-adaptive feature prediction module and the speech synthesis module, the context-adaptive feature predictor module is communicatively connected to the feature parameter mapping module, and the feature parameter mapping mechanism module is communicatively connected to the speech synthesis module.

[0015] Preferably, the speech synthesis module is connected to the chatbot and outputs the target speech through the chatbot.

[0016] Therefore, this invention proposes a context-adaptive multi-turn interactive emotional speech synthesis method and system, the beneficial effects of which are as follows: (1) The present invention uses a two-stage trained audio large model to directly process historical speech signals, bypassing the intermediate steps of traditional automatic speech recognition (ASR) to transcribe text, fully preserving emotional and paralinguistic features, and ensuring the integrity and accuracy of emotional information in historical context.

[0017] (2) After two-stage training, the context-adaptive feature predictor of this invention can dynamically and adaptively infer the emotion and acoustic features of the response speech based on the feature information of historical dialogue speech, establish the association between emotion and paralinguistic features between multiple rounds of dialogue, realize the natural evolution of emotional state and the continuous inheritance of contextual information, and avoid the problems of abrupt emotion and disjointed rhythm caused by isolated sentence adjustment in the prior art.

[0018] (3) The present invention utilizes the designed feature parameter mapping mechanism and optimizes it through Mel-Cepstral Distortion Quantization to determine the pre-constructed standardized mask mapping template, transforming the natural language features output by the predictor into standardized parameters that can be directly used by the synthesis model. This effectively avoids the problems of information redundancy and format deviation in the application of large text models, and significantly improves the availability, consistency and control accuracy of features in the speech generation process.

[0019] (4) This invention uses adjacent single-turn historical speech as the context window, adapts to real-time multi-turn dialogue scenarios, and is based on a mature base model and audio large model. The training process adopts the LoRA fine-tuning method, which takes into account both training efficiency and model performance. It is easy to implement in actual interactive scenarios such as intelligent assistants, intelligent customer service, and in-vehicle terminals. At the same time, it provides a reliable foundation for the subsequent expansion of emotional speech synthesis technology in more complex contexts. Attached Figure Description

[0020] Figure 1 Here is a diagram of a context-adaptive multi-turn interactive emotional speech synthesis framework, where: (a) is the overall framework diagram, (b) is an example diagram of feature predictor training data construction, and (c) is a diagram of the feature parameter mapping mechanism. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example 1 This invention provides a context-adaptive multi-turn interactive emotional speech synthesis method, which specifically includes the following steps: Step S1: In an instant multi-turn dialogue scenario with adjacent single-turn historical speech as the context window, obtain the historical speech signal sequence and the current text to be synthesized in the multi-turn dialogue. The historical speech signal sequence does not need to undergo speech-to-text ASR processing.

[0024] Step S2: Select the basic large audio model (Qwen-audio), and construct a context-adaptive feature predictor through two-stage training. This predictor adapts to the context and predicts the emotional and paralinguistic features of the response speech, constructs and outputs an initial acoustic feature set, and replaces the traditional speech-to-text method to enhance the understanding of the information contained in historical speech. The two-stage training specifically involves: The first stage of training involves expanding upon the speech dialogue dataset to construct a small-scale manually annotated dataset. This dataset includes acoustic features, specifically: basic features, emotional features, and paralinguistic features. To ensure data consistency and reproducibility, strict quantitative annotation standards are established to standardize the annotation of each type of feature. In the basic characteristics, speaking speed is divided into words per minute (WPM), where: speaking speed ≤ 130 WPM is slow; 130 WPM < speaking speed < 160 WPM is medium; and speaking speed ≥ 161 WPM is fast. Pitch is set according to the average fundamental frequency to distinguish gender: male: low fundamental frequency ≤ 100Hz, 101Hz < mid fundamental frequency < 140Hz, high fundamental frequency ≥ 141Hz; female: low fundamental frequency ≤ 180Hz, 181Hz < mid fundamental frequency < 240Hz, high fundamental frequency ≥ 241Hz. Emotional traits are based on valence Activation level With Dominance The 5-point scale is used for quantitative definition, specifically: ; Among them, other categories must meet the consistency judgment of no less than 70% of the annotators; The time-frequency detection thresholds for paralinguistic features are set as follows: laughter duration ≥ 200ms, with significant harmonic or burst structures; breathing duration ≥ 150ms, with obvious inhalation or exhalation segments without a sound source; sighing duration ≥ 300ms, accompanied by significant energy attenuation and a fundamental frequency downward trajectory.

[0025] Among them, the start and end positions of the sub-language features are marked, and the duration accuracy is retained to 0.001s.

[0026] All annotations are presented in natural language. Qwen-audio was fine-tuned using a manually annotated dataset until the model possessed the ability to deconstruct the feature information of a single speech segment. Its mathematical expression is: ; in, The text is a manually annotated natural language description. For input voice signal, This is the first-stage model used for deconstructing single speech feature information; The second stage of training involves inputting the incremental dialogue dataset into the audio model trained in the first stage. The feature information is deconstructed to obtain the acoustic features of each speech segment. Then, using neighboring single-turn speech segments as context windows, a sliding window is used to construct a historical speech-response speech feature pair dataset. This dataset is then used to incrementally train the audio model until it can predict the initial acoustic features of the response speech based on historical speech. The mathematical expression is: ; in, To recover the initial acoustic features of speech, For the second-stage model used for context prediction, To reply to the voice message, To and The previous round of voice messages from the nearest neighbor, i.e., the voice messages in response. Historical voice recordings.

[0027] Step S3: Establish a feature parameter mapping mechanism that includes pre-built mapping rule templates and feature field parsing. Input the initial acoustic feature set into the feature parameter mapping mechanism and output standardized acoustic control parameters. This includes the following steps: Step S31: Optimize the parameters of the feature parameter mapping mechanism. Generate multiple natural language prompt word variants with the same semantics using the frozen text model. These variants are then manually selected and supplemented. Mel-frequency cepstral distortion (MCD) is used as the quantitative evaluation standard to score and rank the similarity between the generated speech samples and the original speech. A lower MCD value indicates that the generated speech is closer to the original speech in terms of timbre and rhythm. After at least one round of testing, the optimal prompt word is determined. The calculation formula is as follows: ; in, In the first Frame number Mel-spectral coefficients of the original speech, For the first Frame number Mel-spectral coefficients of synthesized speech in dimensionality. The number of feature dimensions. Total number of frames Used to convert units to decibels.

[0028] Step S32: Construct a mapping rule template based on the optimal prompt word, and use the mapping rule template to define a standardized mask mapping Slot-Filling paradigm, including a global acoustic state template, a local paralinguistic feature event anchoring template, and an accent control template, specifically: The global acoustic state template uses a fixed sentence structure: The speed of speech is , the pitchis , and the emotion is It carries parameters such as speaking speed, pitch, and emotion category; The local sublingual feature event anchoring template adopts: "The comes after the word duration: "seconds" maps the trigger location and duration of laughter, breathing sounds, etc. The accent control template uses: "Emphasize word" "" specifies the position of the accent; The mathematical expression for the mapping layer is: in, For mapping rules, For the feature extraction process, Control signals for subsequent speech synthesis.

[0029] Step S33: The feature field parsing module extracts key fields from the input initial acoustic features, including feature variables, occurrence positions and durations. Then, the optimal mapping rule template maps these fields to predefined mask positions in a structured manner, generating control signals to control subsequent speech synthesis and outputting standardized acoustic control parameters.

[0030] Step S4: Input the standardized acoustic control parameters and the response text into the speech synthesis model to generate emotionally coherent natural speech that fits the context. The speech synthesis model is CosyVoice2, and the selected pre-trained model version is CosyVoice2-0.5B. This version and the training model itself have been fine-tuned with prompt words and support the joint speech generation driven by natural language prompt words and standardized acoustic control parameters.

[0031] Example 2 like Figure 1 As shown, the present invention also provides a context-adaptive multi-turn interactive emotional speech synthesis system, including a data acquisition module, a context-adaptive feature prediction module, a feature parameter mapping module, and a speech synthesis module; The data acquisition module provides dual input materials—raw speech signals and semantic text—for subsequent feature processing and speech synthesis, ensuring the integrity of emotional and contextual information from the source. This module includes a historical speech acquisition unit and a response text receiving unit. Specifically, the historical speech acquisition unit acquires historical speech signal sequences from multi-turn dialogues in real time (such as adjacent single-turn historical speech), directly preserving fine-grained paralinguistic features such as tone, intonation, and pause duration of the original speech without automatic speech recognition (ASR) transcription, thus avoiding information loss in intermediate stages. The response text receiving unit receives the current response text to be synthesized (such as robot response text) generated by the dialogue system, serving as the semantic basis for speech synthesis.

[0032] The context-adaptive feature prediction module replaces traditional ASR transcription and text sentiment analysis solutions. It directly models the historical context at the speech signal level, accurately predicting the emotional and multi-dimensional acoustic features required for the response speech. This is the core of achieving "context adaptation," and includes a first-stage training unit and a second-stage training unit. The first-stage training unit expands upon the IEMOCAP and DailyTalk datasets, annotating acoustic features and establishing strict quantitative standards to construct a manually annotated dataset. This annotated dataset is used to train the Qwen-audio large-scale audio model, enabling it to deconstruct fine-grained features of single-turn speech signals and output feature descriptions in natural language. The second-stage training unit inputs speech from the incremental dialogue dataset into the model trained in the first stage, deconstructing the acoustic feature information of each speech. Using adjacent single-turn speech as a context window, a sliding window and corresponding historical speech are used to construct a "historical speech-response speech feature pair" dataset. This allows the model to learn the feature correlation patterns between historical context and response speech, enabling it to "predict response speech features based on historical speech."

[0033] The feature parameter mapping module includes a prompt word optimization and evaluation unit, a mapping rule template construction unit, and a feature field parsing and structured mapping unit.

[0034] The feature parameter mapping module transforms unstructured initial acoustic features into standardized control parameters that the synthesis model can directly call, ensuring feature utilization and consistency, and solving the problem of poor feature-synthesis model compatibility. It includes a prompt word optimization and evaluation unit, a mapping rule template construction unit, and a feature field parsing and structured mapping unit. Specifically: the prompt word optimization and evaluation unit selects and determines the optimal prompt words, providing high-quality basic materials for subsequent mapping template construction; the mapping rule template construction unit builds standardized and reusable mapping rule templates based on the optimal prompt words, defines a pre-filled mask for each template type, forming a standardized Slot-Filling paradigm, and clarifies the field type corresponding to each slot, ensuring the standardization and accuracy of subsequent field mapping. The completed standardized mapping rule templates are stored to form a template library, providing a callable carrier for subsequent field filling; the feature field parsing and structured mapping unit extracts key fields from the initial acoustic features, and structurally fills the extracted key fields into the predefined mask positions of the template according to the principle of "field type matching slot type," ultimately generating and outputting standardized acoustic control parameters for direct use in subsequent speech synthesis steps.

[0035] The speech synthesis module combines standardized parameters with semantic text to achieve coordinated control of emotion, prosody, and paralinguistic features, generating target speech that is consistent with historical context and emotionally natural and coherent. It includes an input fusion unit, an emotion-prosody control unit, and a speech generation unit. Specifically: the input fusion unit receives standardized acoustic control parameters output from the feature parameter mapping mechanism and fuses them with the response text from the data acquisition module to generate joint input features for the synthesis model; the emotion-prosody control unit, based on the prompt word fine-tuning capabilities of models such as CosyVoice2-0.5B, parses the emotion, prosody, and paralinguistic event instructions in the standardized parameters to precisely control the emotional intensity, speech rate, rhythm, stress position, and timing of paralinguistic feature insertion; the speech generation unit uses CosyVoice2-0.5B as the base model, converting the joint input features into Mel spectrograms, which are then processed by a vocoder to generate the target speech, supporting streaming output to meet real-time interaction requirements. Detailed Implementation Method 1 Step 1: Configure the model and set up the experimental environment: Model configuration: The base model uses the CosyVoice2 speech synthesis model based on cue words. The pre-trained version selected is CosyVoice2-0.5B. This model itself has been fine-tuned with a large number of cue words, allowing natural language cue word text input within a certain complexity and length to control speech synthesis.

[0037] In the context-adaptive feature predictor, the large-scale audio model uses the Qwen-Audio model, with the selected pre-trained version being Qwen-Audio-Chat. This model is currently a state-of-the-art large-scale speech understanding model, capable of automatic speech recognition, acoustic scene classification, and audio question answering. After fine-tuning, it can further understand speech emotions and predict acoustic features. Specifically, the two-stage training of the Qwen-Audio model uses the LoRA fine-tuning method, with 30 training epochs and a learning rate set to... Training is performed using BFloat16 precision, with the following hyperparameter configurations: LoRA rank is set to 8, scaling factor (Alpha) is set to 32, dropout rate is 0.05, and fine-tuning range covers all linear layers within the Transformer module; The optimizer used is AdamW, with weight decay set to 0.1 and an initial learning rate set to... ; The learning rate scheduling adopts a cosine annealing strategy with a warmup ratio of 0.05. The experiment was set with a batch size of 1 and a gradient accumulation step of 16, meaning the effective batch size was 16. The model training uses the standard cross-entropy loss function, which optimizes the model parameters by minimizing the difference between the predicted probability distribution and the true label distribution; The training termination condition is set to complete 30 full training epochs. No early stopping mechanism based on validation set metrics is set during this period to ensure that the model fully converges within the preset training period.

[0038] Experimental environment: The experiment was conducted using the Communication University of China cloud platform, with PyTorch 2.0.1 and CUDA 11.8 environment, and a single NVIDIA A800 GPU.

[0039] Step 2: Conduct experiments on the IEMOCAP and DailyTalk voice dialogue datasets.

[0040] IEMOCAP is a classic emotional dialogue dataset containing scripted and impromptu dialogues performed one-on-one by five actresses and five actors, featuring natural speech and rich emotional expression. The DailyTalk dataset is used only for subjective and objective evaluation; it is a high-quality collection of binary dialogues designed specifically for text-to-speech tasks.

[0041] The IEMOCAP dataset is used for both training and evaluation: a subset of dialogue samples are randomly selected for artificial small-sample feature dimension augmentation annotation in the feature predictor and for two-stage training of the large audio model; the complete dataset is used for objective and subjective evaluation.

[0042] To ensure a fair comparison, a comparison model is set up: Fastspeech2 is a non-autoregressive end-to-end speech synthesis model that achieves efficient Mel spectrum generation by explicitly modeling acoustic features such as fundamental frequency, energy, and duration.

[0043] Vall-e: A zero-shot speech synthesis system based on a large language model, which generates speech by encoding speech into discrete acoustic tokens and combining autoregressive and non-autoregressive models.

[0044] Cosyvoice2: A prompt-based speech synthesis model that does not include the pre-extension module proposed in this paper, and is used as the base speech synthesis model for comparison.

[0045] Propose: This paper proposes a complete context-adaptive speech synthesis framework.

[0046] Based on comparisons with several advanced speech synthesis models, the performance of the proposed framework in terms of speech naturalness and emotional accuracy is experimentally verified.

[0047] Step 3: Conduct subjective and objective experiments and evaluate the results, including subjective and objective evaluations. Subjective evaluation measures the naturalness and expressiveness of the speech, while objective evaluation covers the quality of the speech and the accuracy of the emotion.

[0048] Subjective experiments were conducted: First, the Mean Opinion Score (MOS) was used to evaluate the overall speech quality of the proposed framework and the contrast model. MOS is a subjective testing method for evaluating the overall performance of speech synthesis. Participants were required to rate each synthesized speech sample using a five-point rating scale (in increments of 1 point). Then, the best-worst scaling (BWS) test was conducted to further evaluate the expressiveness of the synthesized speech. In each round of testing, participants were provided with one real speech sample and four synthesized speech samples generated by different speech synthesis methods. They were asked to select the synthesized sample that was most similar (best) and least similar (worst) to the real speech in terms of expressiveness. The proportion of samples selected as "best" / "worst" for each method was statistically analyzed.

[0049] The experiment invited 32 listeners to participate in the listening test, with a balanced male-to-female ratio, and all participants had no hearing or language impairments. During the evaluation process, 10 synthesized speech samples were randomly selected from the same dataset for each method, and 10 real speech samples were added to supplement them, for a total of 100 test samples. The selected sentences covered a variety of contexts to avoid bias in the output quality evaluation due to differences in sentence structure.

[0050] In the listening test, all speech samples were played in random order, without indicating the source of their synthesis methods. To ensure the quality of the assessment, before the formal evaluation, listeners were required to listen to a set of sample speech samples synthesized using four different methods to familiarize themselves with the range of variation in synthesized speech; at the same time, they were required to carefully read the specific definitions and scoring criteria for each dimension. To further ensure the validity of the evaluation, trap questions such as randomly inserting micro-silent samples were used to identify and eliminate invalid evaluations such as perfunctory responses. The final speech quality evaluation results are shown in Table 1, including the average score and standard deviation. The best-worst scaling (BWS) test results are shown in Table 2, where the values ​​represent the preference ratio: Table 1. Speech Quality Assessment (MOS) Results

[0051] Table 2 Best-Worst Scaling (BWS) Test Results

[0052] As shown in Table 1, the Propose framework achieved excellent performance on both the IEMOCAP and DailyTalk datasets, with MOS scores of 4.147±0.347 and 4.607±0.164, respectively, significantly outperforming all comparison models, including Fastspeech2, Vall-e, and Cosyvoice2; simultaneously, its MOS score was similar to that of ground truth speech. These results demonstrate that the synthesized speech produced by the Propose framework exhibits high levels of naturalness and overall quality, validating its effectiveness in improving the quality of synthesized speech.

[0053] As shown in Table 2, the Propose framework achieved the highest "Best" preference ratio on both datasets (IEMOCAP: 41%, DailyTalk: 45%), while having the lowest "Worst" selection ratio (IEMOCAP: 3%, DailyTalk: 10%), significantly outperforming other comparison systems. The results indicate that the Propose framework more closely resembles real speech in terms of its expressive power in synthesized speech, effectively improving the expressiveness of synthesized speech.

[0054] Objective experiments were conducted: Mel-frequency cepstral distortion (MCD) and emotion classification accuracy (EMO-ACC) were used for objective evaluation. MCD objectively measures the spectral difference between synthesized and real speech; a smaller value indicates lower distortion and better speech quality. EMO-ACC assesses the consistency of synthesized and real speech in emotion categories; this metric was calculated using a third-party emotion recognition tool. The SenseVoice model was used for emotion discrimination. This model achieved or surpassed state-of-the-art performance on multiple datasets. The objective evaluation results are shown in Table 3, including the MCD (BD) and EMO-ACC metrics. Table 3 Objective Evaluation Results

[0055] As shown in Table 3, the Propose framework achieved optimal performance on both evaluation metrics across the two datasets. Regarding speech quality, the proposed method achieved MCD values ​​of 6.75 dB and 9.72 dB on the IEMOCAP and DailyTalk datasets, respectively, significantly lower than other comparative models, indicating that its synthesized speech most closely resembles real speech in spectral features with minimal distortion. In terms of emotional expression consistency, the proposed method achieved an EMO-ACC score of 54.7% on IEMOCAP, significantly outperforming all baseline models; it also reached 47.4% on DailyTalk, still superior to other comparative methods. The objective evaluation results consistently demonstrate that the proposed method outperforms existing methods in both acoustic quality and emotional expression capabilities of synthesized speech, exhibiting better overall performance.

[0056] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0057] Therefore, this invention proposes a context-adaptive multi-turn interactive emotional speech synthesis method and system. It utilizes a large audio model trained in two stages to predict emotional and paralinguistic acoustic features from multi-turn dialogues. Simultaneously, a feature mapping mechanism is designed to map the predicted features into normalized parameters usable by the synthesis system, enabling dynamic control of speech generation. This invention achieves natural and coherent multi-turn interactive speech generation, reaching state-of-the-art performance in both emotional accuracy and speech naturalness.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A context-adaptive multi-turn interactive emotional speech synthesis method, characterized in that, Includes the following steps: Step S1: In an instant multi-turn dialogue scenario with adjacent single-turn historical speech as the context window, obtain the historical speech signal sequence and the current text to be synthesized in the multi-turn dialogue. The historical speech signal sequence does not need to undergo speech-to-text ASR processing. Step S2: Select a basic audio model and construct a context-adaptive feature predictor through two-stage training. This predictor adapts to the context and forecasts the emotional and paralinguistic features of the response speech, constructing and outputting an initial acoustic feature set. The two-stage training specifically involves: The first stage of training involves constructing a manually labeled dataset containing acoustic features, specifically: basic features, emotional features, and paralinguistic features. Quantitative labeling standards are established to standardize the labeling of each type of feature. All labels are presented in natural language. The audio model is then fine-tuned using the manually labeled dataset until it possesses the ability to deconstruct the feature information of a single speech segment. The mathematical expression for this is: ； in, The text is a manually annotated natural language description. For input voice signal, This is the first-stage model used for deconstructing single speech feature information; The second stage of training involves inputting the incremental dialogue dataset into the audio model trained in the first stage. The feature information is deconstructed to obtain the acoustic features of each speech segment. Then, using neighboring single-turn speech segments as context windows, a sliding window is used to construct a historical speech-response speech feature pair dataset. This dataset is then used to incrementally train the audio model until it can predict the initial acoustic features of the response speech based on historical speech. The mathematical expression is: ； in, To recover the initial acoustic features of speech, For the second-stage model used for context prediction, To reply to the voice message, To and The previous round of voice messages from the nearest neighbor, i.e., the voice messages in response. Historical voice recordings; Step S3: Establish a feature parameter mapping mechanism that includes pre-built mapping rule templates and feature field parsing. Input the initial acoustic feature set into the feature parameter mapping mechanism and output standardized acoustic control parameters. This includes the following steps: Step S31: Optimize the parameters of the feature parameter mapping mechanism. Generate multiple natural language prompt word variants with the same semantics through the frozen text large model. Manually screen and supplement them. Use Mel-Cepstral Distortion (MCD) as the quantitative evaluation standard to score and rank the similarity between the speech samples generated by different prompt word variants and the original speech. After no less than one round of testing, determine the optimal prompt word. Step S32: Construct a mapping rule template based on the optimal prompt word, and use the mapping rule template to define a standardized mask mapping Slot-Filling paradigm, including a global acoustic state template, a local sublingual feature event anchoring template, and an accent control template; Step S33: The feature field parsing module extracts key fields from the input initial acoustic features, including feature variables, occurrence positions and durations. Then, the optimal mapping rule template maps these fields to predefined mask positions in a structured manner, generating control signals to control subsequent speech synthesis and outputting standardized acoustic control parameters. Step S4: Combine the standardized acoustic control parameters with the response text input speech synthesis model to generate emotionally coherent natural speech that fits the context.

2. The multi-turn interactive emotional speech synthesis method based on context adaptation according to claim 1, characterized in that: In step S2, the basic audio model is Qwen-audio, and the LoRA fine-tuning method is used for training.

3. The multi-turn interactive emotional speech synthesis method based on context adaptation according to claim 1, characterized in that: In step S21, the acoustic features and quantitative annotation standards in the manually labeled dataset are as follows: In the basic characteristics, speaking speed is divided into words per minute (WPM), where: speaking speed ≤ 130 WPM is slow; 130 WPM < speaking speed < 160 WPM is medium; and speaking speed ≥ 161 WPM is fast. Pitch is set according to the average fundamental frequency to distinguish gender: male: low fundamental frequency ≤ 100Hz, 101Hz < mid fundamental frequency < 140Hz, high fundamental frequency ≥ 141Hz; female: low fundamental frequency ≤ 180Hz, 181Hz < mid fundamental frequency < 240Hz, high fundamental frequency ≥ 241Hz. Emotional traits are based on valence Activation level With dominance The 5-point scale is used for quantitative definition, specifically as follows: ； Among them, other categories must meet the consistency judgment of no less than 70% of the annotators; The time-frequency detection thresholds for paralinguistic features are set as follows: laughter duration ≥ 200ms, with significant harmonic or burst structures; breathing sound duration ≥ 150ms, with obvious inhalation or exhalation segments without a sound source; sighing sound duration ≥ 300ms, accompanied by significant energy attenuation and a fundamental frequency downward trajectory. Among them, the start and end positions of the sub-language features are marked, and the duration accuracy is retained to 0.001s.

4. The multi-turn interactive emotional speech synthesis method based on context adaptation according to claim 1, characterized in that: In step S31, MCD reflects the difference in spectral features by calculating the Euclidean distance between the synthesized speech and the reference speech on the Mel-frequency cepstral coefficients (MFCCs). The lower the value, the closer the generated speech is to the original speech in terms of timbre and prosody. The calculation formula is as follows: ； in, In the first Frame number Mel-spectral coefficients of the original speech, For the first Frame number Mel-spectral coefficients of synthesized speech in dimensionality. The number of feature dimensions. Total number of frames Used to convert units to decibels.

5. The multi-turn interactive emotional speech synthesis method based on context adaptation according to claim 1, characterized in that: In step S32, the global acoustic state template uses a fixed sentence structure: The speed of speech is The pitch is , and the emotion is It carries parameters such as speaking speed, pitch, and emotion category; The local sub-language feature event anchoring template adopts: "The comes after the word duration: "seconds" maps the trigger location and duration of laughter, breathing sounds, etc. The accent control template uses: "Emphasize word" "" specifies the position of the accent.

6. The multi-turn interactive emotional speech synthesis method based on context adaptation according to claim 1, characterized in that: In step S33, the mathematical expression for the mapping layer is: in, For mapping rules, For the feature extraction process, Control signals for subsequent speech synthesis.

7. The multi-turn interactive emotional speech synthesis method based on context adaptation according to claim 1, characterized in that: In step S4, the speech synthesis model is CosyVoice2, and the selected pre-trained model version is CosyVoice2-0.5B. The CosyVoice2-0.5B pre-trained model itself has been fine-tuned with prompt words and supports the joint driving of speech generation by natural language prompt words and standardized acoustic control parameters.

8. A context-adaptive multi-turn interactive emotional speech synthesis system, used to implement the context-adaptive multi-turn interactive emotional speech synthesis method as described in claims 1-7, characterized in that: It includes a data acquisition module, a context-adaptive feature prediction module, a feature parameter mapping module, and a speech synthesis module; The data acquisition module includes a historical voice acquisition unit and a response text receiving unit; The context-adaptive feature prediction module includes a first-stage training unit and a second-stage training unit; The feature parameter mapping module includes a prompt word optimization and evaluation unit, a mapping rule template construction unit, and a feature field parsing and structured mapping unit; The speech synthesis module includes an input fusion unit, an emotion-prosodic modulation unit, and a speech generation unit.

9. A context-adaptive multi-turn interactive emotional speech synthesis system according to claim 8, characterized in that: The data acquisition module is connected to the context-adaptive feature prediction module and the speech synthesis module. The context-adaptive feature predictor module is connected to the feature parameter mapping module. The feature parameter mapping mechanism module is connected to the speech synthesis module.

10. A context-adaptive multi-turn interactive emotional speech synthesis system according to claim 8, characterized in that: The speech synthesis module communicates with the chatbot and outputs the target speech through the chatbot.

Images