Speech synthesis method and device based on double models, equipment and medium

Abstract

The invention relates to the technical field of speech synthesis, can be applied to service system platforms of medical health, financial science and technology and the like, and discloses a speech synthesis method based on double models, which comprises the following steps: pre-training a flow matching model of non-emotion modeling according to a general field text and an emotion speech library to generate a basic model; obtaining a network structure and a model weight of the basic model, and constructing a functionally differentiated dual model based on the network structure and the model weight; constructing a preference data pair through the original text sequence, and inputting the preference data pair into the double models for training; and obtaining a target text and a reference timbre audio, inputting the target text and the reference timbre audio into the trained dual model, and outputting a target Mel spectrum. According to the method, a flow matching basic model with an emotion label is pre-trained, then a double-emotion model is constructed and optimized, fusion output is carried out during reasoning, speech synthesis with controllable emotion intensity is realized, and a target Mel spectrum is obtained.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, device, and storage medium based on a dual-model approach. Background Technology

[0002] Emotional TTS, as an important research direction in the field of speech synthesis, has seen the development of various mature technical paths and application scenarios in industry and academia. The core methods mainly revolve around three categories: First, discrete emotion generation based on style encoding, which is currently the most mainstream approach. The model learns the acoustic features of different basic emotions such as joy, anger, and sadness, and encodes them as style vectors (Style Tokens) or latent variables. During the generation stage, users only need to select preset emotion tags to trigger the corresponding speech output. Second, style transfer based on reference audio. Users provide reference audio containing the target emotion, and the model extracts prosodic rhythm and emotional style features from it, then transfers them to new text content to achieve emotion-matched speech generation. Third, fine-tuning based on a small amount of reinforcement learning. Research represented by Emo-DPO has begun to attempt to optimize the model through reinforcement learning to make the generated emotional speech more in line with user preferences.

[0003] Although the technology has been applied to a considerable scale, existing solutions still have significant shortcomings and development bottlenecks. The core problem lies in the coarse granularity of emotion control. Current technologies are mostly limited to discrete, categorized emotion regulation, where users can only choose from fixed labels such as "happy" and "sad," failing to achieve a continuous transition in emotional intensity, such as the subtle gradient expression from "somewhat happy" to "very happy" to "ecstatic." This "either / or" control mode severely limits its application in scenarios requiring refined emotional expression, such as game NPC interaction, audiobook performance, and real-time communication with virtual humans. Secondly, the design of a single model contains inherent contradictions. Both traditional supervised learning and DPO (Direct Preference Optimization) schemes rely on a single model to carry all tasks. It must learn to generate high-quality "preference-oriented" emotional speech while also mastering how to avoid "unpreference" speech features. This forces the model to balance conflicting learning objectives within a unified parameter space, easily leading to instability in the training process and ultimately failing to achieve optimal results in specific emotional expressions or neutral emotional output. Furthermore, model training also faces the risk of degradation in prosody and neutral expression: when forcing the model to learn the acoustic features of strong emotions, it may impair its ability to generate clear, rhythmically natural neutral speech; conversely, models that emphasize neutral expression often perform poorly when generating emotionally charged speech, failing to convey full emotional tension. Finally, there is a contradiction between inference efficiency and the need for emotion control. While classifier-independent guidance (CFG) provides guidance for generation direction, it can only distinguish between "conditional" (such as text information) and "unconditional" generation, without optimizing for emotion intensity control. To achieve complex emotion gradient regulation, multiple inferences and weighted calculations are usually required, resulting in low overall efficiency and difficulty in meeting the needs of real-time applications.

[0004] In fintech operations, the existing technological limitations of emotional voice synthesis can lead to multiple business risks and user experience shortcomings. This is particularly evident in scenarios such as intelligent customer service and voice notifications. For example, when informing users of positive information such as approved business transactions or profitable investments, it's difficult to adapt to different users' varying emotional levels, such as "gentle joy" or "warm congratulations," potentially resulting in overly exaggerated or bland emotional expressions that could negatively impact user trust in financial services. Similarly, in scenarios like reminding users of overdue payments or issuing risk warnings, it's challenging to precisely control the intensity of "gentle warnings" versus "serious admonitions." Too weak a tone might lead users to ignore the risk, while too strong a tone could trigger resistance. The inherent contradictions of a single model further amplify potential business risks. Financial scenarios require models to generate neutral and professional business explanations (such as interpreting product terms) while simultaneously conveying anxiety and concern when users encounter account anomalies. The compromises made by a single model under conflicting objectives can lead to unclear delivery of professional information or distorted emotional expression that causes misunderstandings. Furthermore, the risk of downgrading in rhythm and neutrality can affect the accuracy of financial information. If the model sacrifices pronunciation clarity to enhance emotional expression, it may lead to errors in the transmission of key data such as interest rates and amounts; while insufficient inference efficiency will fail to meet the timeliness requirements of scenarios such as high-frequency transaction notifications and real-time customer service responses, thus reducing business processing efficiency.

[0005] The healthcare field demands extremely high accuracy and adaptability in emotional speech synthesis. Existing technological deficiencies directly impact the quality of medical services and patient experience. Significant risks exist in patient communication and health guidance scenarios: for example, when communicating recovery progress to patients with chronic diseases, it's impossible to match the patient's psychological state with varying emotional gradients such as "gentle encouragement" and "positive affirmation." Overly strong emotions may stress anxious patients, while overly bland emotions may fail to instill sufficient confidence. When informing patients about medical precautions, balancing the intensity of "caring reminders" and "serious warnings" is difficult, potentially affecting treatment adherence due to inappropriate emotional expression. The inherent contradictions and prosodic degradation risks of single-model approaches can impair the professionalism and accuracy of medical information delivery: medical scenarios require models to generate neutral and rigorous medication guidance and examination instructions, while simultaneously conveying warm and empathetic emotions in post-operative care and psychological support. The training instability of single-model approaches under conflicting objectives can lead to unclear pronunciation of technical terms and stiff emotional delivery, affecting patients' understanding and trust in medical information. Furthermore, the conflict between inference efficiency and emotional control restricts applications in scenarios such as remote consultations and real-time health monitoring. Response delays caused by multiple inferences may prevent the timely delivery of much-needed health guidance or emotional support to patients, affecting the timeliness and effectiveness of medical services. Summary of the Invention

[0006] The main objective of this invention is to provide a speech synthesis method, apparatus, device, and storage medium based on a dual-model approach, aiming to address the shortcomings of existing technologies in terms of coarse granularity of emotion control in emotional speech synthesis, trade-offs in the performance of a single model, degradation of prosodic neutrality, and low inference efficiency.

[0007] To achieve the above objectives, the present invention provides a speech synthesis method based on a dual-model approach, comprising: A base model is generated by pre-training a stream matching model without emotion modeling based on a general domain text and emotion speech library. Obtain the network structure and model weights of the base model, and construct a functionally differentiated dual model based on the network structure and model weights; Preference data pairs are constructed using the original text sequence, and these preference data pairs are then used as input to train the dual model. Obtain the target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

[0008] Furthermore, to achieve the above objectives, the present invention provides a speech synthesis device based on a dual-model approach, comprising: The basic model training module is used to pre-train the non-emotionally modeled stream matching model based on a general domain text and emotional speech library to generate a basic model. A dual-model construction module is used to obtain the network structure and model weights of the base model, and to construct a functionally differentiated dual model based on the network structure and model weights; The dual-model training module is used to construct preference data pairs from the original text sequence and input the preference data pairs into the dual model for training. The speech synthesis module is used to acquire target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a dual-model-based speech synthesis program stored in the memory and executable on the processor, wherein when the dual-model-based speech synthesis program is executed by the processor, it implements the steps of the dual-model-based speech synthesis method as described above.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a dual-model-based speech synthesis program, wherein when the dual-model-based speech synthesis program is executed by a processor, it implements the steps of the dual-model-based speech synthesis method described above.

[0011] Beneficial Effects: This invention relates to the field of speech synthesis technology and can be applied to business system platforms such as healthcare and fintech. It discloses a speech synthesis method based on a dual-model approach, comprising: pre-training a stream matching model without emotion modeling based on general domain text and an emotion-based speech library to generate a base model; obtaining the network structure and model weights of the base model, and constructing a functionally differentiated dual model based on the network structure and model weights; constructing preference data pairs through original text sequences and inputting the preference data pairs into the dual model for training; obtaining the target text and reference timbre audio, inputting the target text and reference timbre audio into the trained dual model, and outputting the target Mel spectrum. This invention first pre-trains a stream matching base model with emotion labels, then constructs and optimizes the dual-emotion model, and fuses the outputs during inference to achieve speech synthesis with controllable emotion intensity, obtaining the target Mel spectrum. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a dual-model-based speech synthesis method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the dual-model-based speech synthesis method of the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the speech synthesis device based on a dual-model according to the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The dual-model-based speech synthesis method provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can obtain a general-domain text and emotional speech library from the user terminal, pre-train a stream matching model without emotion modeling based on the general-domain text and emotional speech library to generate a base model; obtain the network structure and model weights of the base model, and construct a functionally differentiated dual model based on the network structure and model weights; construct preference data pairs through the original text sequence, and input the preference data pairs into the dual model for training; obtain the target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum. This invention first pre-trains a stream matching base model with emotion labels, then constructs and optimizes the dual emotion model, and fuses the outputs during inference to achieve speech synthesis with controllable emotion intensity, obtaining the target Mel spectrum. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0015] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the dual-model-based speech synthesis method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0016] like Figure 2 As shown, the speech synthesis method based on a dual-model proposed in this invention includes the following steps: S100. Pre-train the stream matching model without emotion modeling based on general domain text and emotion speech library to generate a basic model; S200. Obtain the network structure and model weights of the basic model, and construct a functionally differentiated dual model based on the network structure and model weights; S300. Construct preference data pairs using the original text sequence, and train the dual model using the preference data pairs as input; S400: Obtain the target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

[0017] In this embodiment, the first step is the pre-training of the basic model. The selected stream matching model itself does not perform emotion modeling and cannot directly achieve emotion-infused speech synthesis. To overcome this limitation, during training, a general-domain text and an emotion-based speech library are combined, and explicit emotion tags (such as happy, sad, melancholic, etc.) are added to the front end of the text sequence. The integrated text sequence is then fed into the ConvNeXt-v2 encoder for encoding. The other network structures remain unchanged, and finally, a basic model that can respond to emotion tags (v...) is trained. ref This lays the foundation for the subsequent construction of the dual model. Among them, the "stream matching model" is a generative model based on stream transformation, which is used to model and generate data distribution; "ConvNeXt-v2" is a high-efficiency convolutional neural network encoder that is good at extracting deep features of text.

[0018] Next, we proceed to the dual-model construction phase. First, we obtain the base model (v ref The complete network structure and trained model weights are used as the basis for constructing two expert models with differentiating functions. The network structures of these two models are completely identical to the base model, and the weights are directly initialized using the base model's weights to ensure good initial performance. One of them is the Positive Sentiment Model (v... θ + The optimization goal is to become an expert in strong emotional expression, focusing on generating emotionally rich and rhythmically exaggerated speech; the other is the negative / neutral model (v θ - The optimization goal is to become a high-quality neutral pronunciation expert. The core task is to generate neutral speech with clear pronunciation and natural rhythm. This functional differentiation avoids the performance trade-off problem of a single model.

[0019] Next is the dual-model training phase. The key to training is constructing preference data pairs. Specifically, for the same original text, we obtain the Mel spectrum with a clear emotion (such as happiness) as preference samples (x). w ), and the Mel spectrum of neutral sentiment as a non-biased sample (x l ), forming one-to-one corresponding training data pairs (x w , x l Mel spectrum is a key feature representation in speech signal processing, effectively reflecting the acoustic characteristics of speech and serving as a core bridge connecting text and speech in emotional speech synthesis. These preference data are then used as input to a dual model, employing a patented innovative EMO loss function (L...). EMOThe training process uses a loss function inspired by CHATS, which achieves stable and decoupled optimization of the two models through multi-dimensional constraints, allowing each expert model to efficiently improve its performance towards the preset functional goals.

[0020] Finally, there's the inference and generation stage. First, the input information is defined, including the target text for speech synthesis, reference audio timbres for timbre matching, and the target emotion category (e.g., happy, sad) and an emotion intensity control scalar α (typically in the range [0, k], k≥1). These inputs are fed into the trained dual-model system. The model dynamically fuses the outputs of the two expert models through a collaborative guidance mechanism, first calculating the output of the surrogate model (v... proxy Then, combined with the CFG (Classifier-Independent Guidance) guidance coefficient, the final prediction vector field is obtained through the core collaborative guidance formula, which in turn outputs the target Mel spectrum that accurately matches the target sentiment category and sentiment intensity. This Mel spectrum can be further converted into the final speech signal. Among them, "CFG (Classifier-Independent Guidance)" is a flexible generation guidance mechanism that can effectively control the generation direction and ensure that the sentiment expression meets expectations.

[0021] For example, in the healthcare field, this technology can be used in intelligent doctor-patient interaction scenarios. First, a basic flow matching model with emotion labels is pre-trained by combining general medical text (such as consultation scripts and rehabilitation guidance corpora) with a medical emotional voice database (such as gentle reassurance, professional patience, and other emotional voices). This enables the model to understand the semantics of medical text and adapt to the emotions conveyed. Then, a dual model is constructed for positive emotions (such as care and reassurance) and negative / neutral emotions (such as professional calmness). This model is trained using preference data pairs (such as matching the spectrum of reassurance voices and calmly informing voices to the text "post-operative precautions"). In practical applications, the patient's consultation text (such as "What should I do about wound pain?") and the doctor's reference voice are input. The dual models output the Mel spectrum corresponding to the emotional intensity and synthesize the appropriate voice, such as using a strong reassurance voice for anxious patients and a neutral professional voice for rational patients, thus improving the doctor-patient communication experience.

[0022] In the fintech field, this technology can be used for intelligent customer service and financial information dissemination scenarios. First, a basic model with financial sentiment tags is pre-trained using general financial text (such as financial product introductions and bill notification corpora) and a financial sentiment voice library (such as friendly explanations and rigorous warnings). Then, based on this, a dual model is constructed for positive sentiment (such as enthusiastic explanations) and negative / neutral sentiment (such as serious voices for risk warnings). This model is trained using preference data pairs (such as the spectrum of matching enthusiastic voices to texts like "financial product returns" and serious voices to texts like "risk warnings"). In actual use, the user's query text (such as "How risky is this fund?") and a personalized customer service reference tone are input. The dual models output the corresponding sentiment's Mel spectrum and synthesize the speech, using a friendly voice to introduce returns and a serious voice to warn of risks, balancing service experience with the accuracy of risk disclosure.

[0023] In one embodiment, prior to step S100, the method further includes: S001. Obtain general domain text, process the general domain text, and generate an original text sequence; S002. Obtain the emotional speech database, process the emotional speech database, and extract discrete emotional tags; S003. Add discrete sentiment tags to the front of the original text sequence to form a text sequence; S004. Convert discrete sentiment types into discrete sentiment vectors through pre-trained word embeddings; S005. Input the text sequence and discrete sentiment vector into the stream matching model, and adjust the text encoder of the stream matching model using the text sequence and discrete sentiment vector to generate a new stream matching model.

[0024] In this embodiment, the flow matching model without emotion modeling capability is transformed into a new flow matching model that can respond to emotion commands. The core logic is to inject emotion modeling capability through "text and emotion information fusion - feature transformation - model adjustment".

[0025] The first step involves acquiring the original text sequence and discrete sentiment tags. The first step is to obtain general-domain text, covering a wide range of scenarios such as daily conversations, news, and stories. This provides a foundation for the model's general language understanding. Through text cleaning, word segmentation, and format standardization, redundant information is removed and the data format is unified, ultimately resulting in a structurally sound and semantically complete original text sequence, laying the linguistic foundation for subsequent sentiment information fusion. The second step is to simultaneously acquire a sentiment speech database, which contains speech data with clear emotional tendencies (such as happy, angry, sad, neutral, etc.). Through acoustic feature analysis and manual annotation verification, corresponding sentiment category identifiers are extracted from the speech data to obtain discrete sentiment tags. These "discrete sentiment tags" refer to non-continuous, clearly categorized sentiment identifiers (such as "happy" or "sad"), which is entirely consistent with the design of "adding sentiment tags (such as happy, sad, melancholic, etc.) to the front of the text sequence" in the basic model pre-training stage. This is the core basis for the model to identify emotional needs.

[0026] Next comes the construction of the text sequence and the conversion of sentiment vectors. To enable the model to associate text content with emotional needs, discrete sentiment labels need to be concatenated at the beginning of the original text sequence, forming a combined text sequence of "sentiment label + original text". For example, if the original text sequence is "The weather is nice today", after concatenating the discrete sentiment label "happy", the new text sequence becomes "Happy, the weather is nice today". This structure allows the model to prioritize capturing sentiment instructions when processing text. Simultaneously, to enable the model to efficiently process discrete symbols such as sentiment labels, pre-trained word embeddings are used to convert discrete sentiment labels into discrete sentiment vectors. Pre-trained word embeddings are a commonly used technique in natural language processing that maps discrete text symbols into low-dimensional, dense vectors, preserving semantic or category association information. Here, this technique is used to convert labels such as "happy" and "sad" into vectors that the model can compute, ensuring that sentiment information can synergistically interact with text features.

[0027] Finally, the new stream matching model is generated. The constructed "emotional tag + original text" combined text sequence, along with the converted discrete emotion vector, is input into the initial stream matching model (e.g., based on the F5-TTS stream matching model). The core adjustment focuses on the model's text encoder. The initial stream matching model's text encoder (such as the ConvNeXt-v2 module) can only process plain text sequences. Here, by inputting the text sequence and discrete emotion vectors together, the parameters of the text encoder are fine-tuned and optimized: allowing the encoder to simultaneously learn the emotional information carried by the discrete emotion vector while extracting the semantic features of the text, establishing a "text content - emotional need" association mapping. After multiple rounds of training and parameter tuning, the initial stream matching model has the ability to recognize emotional commands and generate corresponding speech features by combining emotional information, ultimately generating a new stream matching model that can respond to emotional control. This model is the "TTS model v that can be controlled by emotion tags" upon which the subsequent dual-model construction depends. ref This prototype provided a basic framework for the subsequent functional differentiation and optimization of the dual-model approach.

[0028] For example, in the healthcare field, this process can serve scenarios such as intelligent triage and patient care. First, general medical text (such as consultation process instructions, post-operative care guidelines, and popular science information on common diseases) is acquired and processed through cleaning and word segmentation to generate a structurally regular original text sequence. Then, discrete emotional tags such as "reassurance," "rigor," and "patience" are extracted from a medical emotional speech database (including gentle reassurance from doctors, patient explanations from nurses, and rigorous explanations of medical conditions). Corresponding tags are then concatenated at the front end of the original text sequence, such as "reassurance + post-operative wound care steps," forming a text sequence with emotional instructions. Through pre-trained word embedding, tags such as "reassurance" are converted into discrete emotional vectors, which are then input into a flow matching model along with the text sequence. The model's text encoder is adjusted to associate the semantics and emotional needs of the medical text, generating a new flow matching model. This model can then be used to synthesize medical speech adapted to the patient's emotions, such as delivering reassuring emotional content to anxious patients.

[0029] In the fintech field, this process can be used in scenarios such as intelligent customer service and financial information broadcasting. First, general financial texts (such as financial product introductions, bill notifications, risk warnings, and business handling guidelines) are collected and processed through format standardization and semantic filtering to generate a raw text sequence. Then, discrete emotional tags such as "friendly," "serious," and "objective" are extracted from a financial emotional voice library (including friendly customer service explanations, serious risk warnings, and objective profit announcements). These tags are then concatenated at the front end of the raw text sequence, such as "serious + financial product risk clauses," forming an emotionally driven text sequence. The emotional tags are then converted into emotional vectors through pre-trained word embedding and input into a flow matching model along with the text sequence. This adjusts the text encoder to understand the association between financial text and corresponding emotions, generating a new flow matching model. This model can then synthesize financial voices that meet specific scenario requirements, such as delivering risk information with a serious tone to increase user engagement.

[0030] In one embodiment, step S100 includes: S101. Parse the emotional speech database and extract the emotional speech waveform; S102. Convert the emotional speech waveform into a Mel spectrum using a vocoder; S103. Input the text sequence, discrete sentiment tags and Mel spectrum into the new flow matching model for pre-training to generate the base model.

[0031] In this embodiment, through feature transformation of the emotional speech database and joint training with multi-dimensional data, the new stream matching model with preliminary emotional association capabilities is optimized into a basic model that can stably output emotional speech features (v ref ).

[0032] The first step is the generation of emotional speech waveforms. This step transforms the raw data in the emotional speech database into basic acoustic signals that the model can process. The emotional speech database stores speech resources with clear emotional tendencies (such as happiness, sadness, anger, etc.), which may exist in the form of audio files (such as WAV, MP3 formats). These resources are processed using audio parsing techniques (such as audio decoding, format unification, noise filtering, etc.) to remove background noise, unify sampling rates and channel numbers, and ultimately extract pure emotional speech waveforms that directly reflect the acoustic characteristics of emotion. The "speech waveform" is the time-domain representation of the speech signal, intuitively showing the pattern of sound amplitude changes over time. It is the original carrier for subsequent extraction of acoustic features, laying the acoustic foundation for transforming emotional information into features that the model can learn.

[0033] Next, the emotional speech waveform is converted into Mel-spectrum data using a vocoder, which is the core step in achieving "speech-feature" conversion. The vocoder is a key tool in speech signal processing, its main function being to convert the time-domain speech waveform into frequency-domain acoustic features. Mel-spectrum data, on the other hand, is a frequency-domain feature representation that conforms to the characteristics of human hearing. Compared to ordinary spectrograms, it can more accurately capture the differences in human ear sensitivity to different frequencies and effectively reflect the rhythm, pitch, timbre, and other acoustic attributes of speech that are strongly correlated with emotional expression. This aligns with the dual-model training phase's use of Mel-spectrum data (preference sample x...) when constructing preference data pairs. w For the Mel spectrum with emotional connotations, the unbiased sample x l This design logic is consistent with that of "Mell spectrum for neutral emotion," indicating that the Mel spectrum is the core feature carrier connecting the speech waveform and model training in emotional speech synthesis. Through vocoder conversion, the emotional information of the emotional speech waveform is quantified into Mel spectrum features that the model can learn, providing crucial acoustic supervision signals for subsequent model training.

[0034] Finally, the pre-training and base model generation of the new flow matching model are performed. This step iteratively upgrades the model's capabilities through joint input of multi-dimensional data. The previously constructed "emotional tag + original text" combined text sequence, discrete emotional tags used to identify emotional categories, and the newly converted emotional Mel spectrum are used as joint training data input into the new flow matching model (i.e., a flow matching model with emotional association capabilities added to the text encoder). During training, the model uses the text sequence and discrete emotional tags as "input instructions" and the Mel spectrum as the "target output feature," continuously adjusting the model parameters through backpropagation. Specifically, the model learns the mapping relationship between "text semantics + emotional category" and "Mel spectrum acoustic features." For example, it helps the model understand that "text 'received a gift today' + emotional tag 'happy'" corresponds to the Mel spectrum features of "high pitch and brisk rhythm." This training process is similar to "modifying the F5-TTS flow matching model, adding emotional tags, and then training a TTS model that can be controlled by emotional tags." ref The goal is perfectly aligned. After multiple rounds of iterative training, the new flow matching model can stably output the corresponding sentiment Mel spectrum based on the text and sentiment instructions. Ultimately, it is upgraded to a basic model with sentiment modeling capabilities, providing core model architecture and initial parameter support for the subsequent differentiation and optimization of the dual models.

[0035] For example, in the healthcare field, this process can be used to build a basic emotional speech model adapted to doctor-patient communication scenarios. First, a medical-specific emotional speech library is parsed. This library contains speech from scenarios such as doctors comforting patients, informing them of their condition, and instructing them on medication. Background noise is removed using audio parsing technology, extracting pure emotional speech waveforms, such as gentle speech waveforms for comforting and rigorous speech waveforms for informing patients of their condition. Then, using a vocoder, these speech waveforms are converted into Mel spectrograms that meet the needs of medical scenarios, preserving the acoustic features related to emotion in the speech. Subsequently, the previously constructed "emotional tag + medical text" sequence (such as "comfort + postoperative rehabilitation precautions"), the corresponding discrete emotional tags (such as "comfort"), and the converted Mel spectrograms are input into a new flow matching model for pre-training. The trained basic model can accurately learn the mapping relationship between medical text, emotional tags, and emotional acoustic features. It can be further optimized into speech models adapted to different medical scenarios, such as a comforting speech model for anxious patients or a medication guidance speech model for patients with chronic diseases.

[0036] In the fintech field, this process can build a foundational emotional speech model for financial interaction scenarios. First, it analyzes a financial emotional speech library, covering scenarios such as customer service answering financial questions, warning of investment risks, and notifying customers of billing information. Clear emotional speech waveforms are extracted using audio processing technology, such as the friendly voice waveform when answering questions and the serious voice waveform when providing risk warnings. These waveforms are then converted into Mel spectrograms using a vocoder, preserving the acoustic details reflecting professionalism and emotional inclination in the financial speech. Next, the sequence of "emotional tag + financial text" (e.g., "serious + fund investment risk warning"), the corresponding discrete emotional tags (e.g., "serious"), and the Mel spectrogram are input into a new flow matching model for pre-training. The generated foundational model can learn the correlation between the semantics, emotional needs, and acoustic features of financial text, laying the foundation for subsequent differentiation of specialized emotional speech models such as financial announcements and risk warnings, ensuring that the synthesized financial speech is both professional and meets the emotional needs of the scenario.

[0037] In one embodiment, step S200 includes: S201. Obtain the network structure and model weights of the basic model; S202. Based on the network structure, construct a positive sentiment model and a negative / neutral sentiment model; S203. Initialize the positive sentiment model and the negative / neutral sentiment model using the model weights.

[0038] In this embodiment, based on the already trained base model, two functionally differentiated but homologous expert models are constructed to lay the structural and parameter foundation for subsequent targeted optimization of emotional expression and neutral speech quality.

[0039] The first step is to obtain the core information of the basic model, which is a prerequisite for building a dual-model system. Here, the "basic model" refers to the stream matching model (v) that is responsive to emotion tags, obtained earlier through pre-training with an emotion-based speech database and general text. ref In this step, it is necessary to extract two key pieces of information from the base model: First, the network structure, including the hierarchical design of the model (such as the number of layers and convolutional kernel size of the ConvNeXt-v2 text encoder, the flow transform network structure of the flow matching module, etc.), the connection methods of each module (such as the feature transfer paths between the encoder and the flow matching module), the selection of activation functions, and all other architectural details, to ensure that the dual model built subsequently maintains structural consistency with the base model; Second, the model weights, which are the parameter matrices learned by each network layer after multiple rounds of pre-training of the base model (such as the weight parameters of the convolutional layers, the bias terms of the fully connected layers, etc.). These weights carry the initial learning results of the base model on the relationship between text semantics, sentiment labels, and Mel-spectrum mapping, and are the key basis for the initialization of the dual model.

[0040] Next, the networks for the positive sentiment model and the negative / neutral sentiment model are built, strictly adhering to the principle of "homogeneity and isomorphism." Based on the basic model network structure obtained in the previous step, two new models with the same architecture are completely replicated, defined as the positive sentiment model (v... θ + ) and Negative / Neutral Sentiment Model (v θ - This isomorphic design is crucial: on the one hand, it ensures that the feature processing logic and data flow path of the dual models are completely consistent with the base model, avoiding feature incompatibility issues during subsequent training and inference due to structural differences; on the other hand, it provides a unified "starting point framework" for the subsequent functional differentiation of the dual models. The default function of the positive emotion model is to become a "strong emotion expression expert," focusing on enhancing the emotional fullness of speech (such as the pitch fluctuations when happy, and the changes in speech rate and volume when angry), while the default function of the negative / neutral emotion model is to become a "high-quality neutral pronunciation expert," with the core goal of consolidating the clarity and rhythmic naturalness of speech, maintaining stable neutral expression quality even when receiving emotional commands. Although their functional positioning is different, both are based on the mature architecture of the base model, ensuring the optimizability and performance ceiling of the model.

[0041] Finally, there's the initialization of the dual models, which achieves the "inheritance" of the learning outcomes from the base model. The weights of the base model obtained in the previous step are fully assigned to the corresponding network layers of the newly built positive sentiment model and negative / neutral sentiment model. For example, the convolutional weights of layer 3 of the base model's ConvNeXt-v2 encoder are directly used as the initial weights for the corresponding layers in the two new models; the flow transformation matrix of the base model's flow matching module is also simultaneously assigned to the corresponding modules in the dual models. The advantage of this initialization method is that the dual models do not need to start training with random parameters, but directly inherit the base model's preliminary capabilities in text semantic understanding and the association between sentiment and acoustic features. This significantly shortens the convergence time of subsequent training and avoids the model training instability problems (such as gradient vanishing and local optima) that may be caused by random initialization. Just as "the network structure of the positive sentiment model and the negative / neutral model and v..." ref Exactly the same, and using v ref The operation of "initializing the weights" ensures that the dual models have good basic performance in speech synthesis in the initial stage, laying a solid foundation for subsequent functional differentiation optimization through DPO (Direct Preference Optimization).

[0042] For example, in the healthcare field, this process can be used to construct specialized emotional speech models tailored to different doctor-patient interaction scenarios. First, the network structure (such as the hierarchy and connection methods of the text encoder and stream matching module) and model weights (parameters carrying medical semantics and basic emotional modeling capabilities) of the previously trained basic medical emotional speech model (which can associate medical text, emotional tags, and acoustic features) are obtained. Then, based on this network structure, two types of specialized models are built: a positive emotional model (focusing on emotions such as "soothing" and "encouragement," suitable for communication with anxious patients and post-operative psychological counseling scenarios) and a negative / neutral emotional model (focusing on emotions such as "rigorous" and "objective," suitable for scenarios involving informing patients about their condition and providing medication guidance). Finally, the weights of the basic model are used to initialize these two models, allowing them to directly inherit medical text understanding and basic speech synthesis capabilities, avoiding the low efficiency and unstable performance issues caused by training from scratch, and laying the foundation for subsequent refined optimization for medical scenarios.

[0043] In the fintech field, this process can build specialized emotional speech models adapted to different financial service scenarios. First, the complete network structure (such as the text encoder architecture for processing financial terminology and the stream matching module design) and trained model weights (including parameters for financial semantic understanding and basic emotional expression) of the previously trained financial emotional speech base model (which can associate financial text, emotional tags, and acoustic features) are extracted. Then, based on this structure, a positive emotional model (emphasizing "friendliness" and "patience," suitable for financial consultation and business processing guidance scenarios) and a negative / neutral emotional model (emphasizing "seriousness" and "professionalism," suitable for risk warnings and bill verification scenarios) are built. Subsequently, these two models are initialized with the base model weights, enabling them to directly possess basic speech synthesis capabilities in the financial field without needing to relearn the semantics of financial terminology and the mapping of basic acoustic features. This significantly shortens the subsequent optimization cycle for financial scenarios and ensures the stability of the model's initial performance.

[0044] In one embodiment, step S300 includes: S301. Based on the emotional speech library, match the positive emotional Mel spectrum and the negative / neutral emotional Mel spectrum for the original text sequence; S302. Construct preference data pairs using the positive sentiment Mel spectrum and the negative / neutral sentiment Mel spectrum; S303. The preference data is input into the positive sentiment model and the negative / neutral sentiment model respectively for training; S304. The trained positive sentiment model and negative / neutral sentiment model are decoupled and optimized using a dual-model loss function to obtain a strong sentiment expression model and a neutral pronunciation model.

[0045] In this embodiment, by constructing targeted preference data pairs and decoupling the training of sub-models and the innovation loss function, the functions of the two expert models are enhanced, and finally, a strong emotional expression model and a neutral pronunciation model are obtained, each with its own function.

[0046] The first step is constructing preference data pairs, which is the core data foundation for dual-model training. Based on the emotional speech database, precise emotional spectrum matching is performed on the original text sequence obtained from the previous processing (e.g., "going on a trip this weekend"). On one hand, it matches the Mel spectrum with clear positive emotions (e.g., happiness, excitement), defining it as the positive emotion Mel spectrum (x). w The acoustic characteristics of this type of spectrum (such as high pitch, brisk rhythm, and strong energy) can intuitively reflect the emotional fullness; on the other hand, the Mel spectrum for matching neutral emotion (no obvious emotional tendency) or weak negative emotion (such as calmness, indifference) to the same original text sequence is defined as the negative / neutral emotion Mel spectrum (x l The acoustic characteristics of this type of spectrum emphasize pronunciation clarity and natural rhythm, without exaggerated emotional fluctuations. Through this pairing method of "same text - different emotional spectrum," one-to-one corresponding preference data pairs (x...) are formed. w , x l This provides clear data supervision for subsequent models to learn the difference between "preference sentiment" and "non-preference sentiment".

[0047] Next is the training of the two models using the preference data pairs as input. This step achieves the initial functional orientation of the two models. The constructed preference data pairs are then input into the positive sentiment model (v) according to the model's functional orientation. θ + ) and negative / neutral sentiment model (v θ - For positive sentiment models, the training process focuses on guiding them to learn the "original text + positive sentiment label" and the positive sentiment Mel-ray spectrum (x). w The mapping relationship between the two is strengthened to enhance its ability to capture and generate emotional acoustic features. For example, the model is made to understand that the text "received a surprise gift" + emotion "happy" corresponds to the spectrum of "large pitch fluctuations and slightly faster speech rate". For negative / neutral emotion models, the focus is on guiding them to learn the relationship between "original text + neutral emotion label" and the negative / neutral emotion Mel spectrum (x l The mapping relationship between the two models strengthens their ability to model clear pronunciation and natural rhythm, avoiding the sacrifice of basic speech quality for the sake of emotional expression. This method of training separate models lays the initial direction for the subsequent functional differentiation of the two models, avoiding the problem of a single model learning conflicting targets at the same time.

[0048] Finally, decoupling optimization based on the dual-model loss function is crucial for achieving specialized dual-model functionality. The patent innovatively draws upon the CHATS concept to design a dual-model loss function (L... EMO This function performs parameter tuning and decoupling optimization on the trained dual models. "Decoupling optimization" means allowing the two models to iterate efficiently towards their preset functional goals without interfering with each other during training, rather than mutually restricting each other. This loss function includes four core constraint dimensions: first, it constrains the difference between the output of the positive sentiment model and its own target (positive sentiment Mel spectrum) to ensure the accuracy of its sentiment expression; second, it constrains the difference between the positive sentiment model and the base model (v... ref The system employs four key constraints: first, ensuring consistency on positive emotion tasks to prevent the model from deviating from its basic speech synthesis capabilities; second, constraining the difference between the output of the negative / neutral emotion model and its own target (negative / neutral emotion Mel spectrum) to guarantee the high quality of its neutral speech; and third, constraining the consistency between the negative / neutral emotion model and the base model on neutral tasks to solidify its robustness in basic pronunciation. Through these four constraints, the emotional expression capability of the positive emotion model is further enhanced, ultimately upgrading it to a strong emotion expression model (optimized v). θ + This can generate rhythmically exaggerated and emotionally rich speech; the neutral pronunciation ability of the negative / neutral emotion model is refined and ultimately upgraded to a neutral pronunciation model (optimized v). θ - Even when receiving emotional commands, it can stably output clear, natural, and neutral speech, completely solving the core pain point of existing single-model technology that "emotion and neutrality cannot be taken into account at the same time".

[0049] For example, in the healthcare field, this process can create specialized voice models adapted to different doctor-patient scenarios. First, from a medical emotional voice database, positive emotional Mel spectra (e.g., the spectra corresponding to "soothing" and "encouraging" voices, adapted to patient anxiety management scenarios) and negative / neutral emotional Mel spectra (e.g., the spectra corresponding to "rigorous" and "objective" voices, adapted to medication dosage notification scenarios) are matched to the original medical text sequences (such as postoperative care instructions and patient reassurance scripts) to construct preference data pairs. Then, these data pairs are input into the positive emotional model (emphasizing emotional expression) and the negative / neutral emotional model (emphasizing accurate information delivery) for training, respectively. Finally, a dual-model loss function is used for decoupling and optimization to obtain a strong emotional reassurance model (for psychological counseling) and a neutral professional model (for medical information broadcasting), ensuring that the voice in different scenarios both meets emotional needs and maintains medical rigor.

[0050] In the fintech field, this process can build specialized voice models adapted to different service scenarios. From a financial sentiment voice database, positive sentiment Mel spectrum (e.g., the spectrum corresponding to "friendly" and "patient" voices, adapted to financial consultation scenarios) and negative / neutral sentiment Mel spectrum (e.g., the spectrum corresponding to "serious" and "professional" voices, adapted to risk disclosure scenarios) are matched to the original financial text sequences (such as product introductions and risk warnings) to form preference data pairs. These pairs are then input into the positive sentiment model and the negative / neutral sentiment model for training. After decoupling and optimization through a dual-model loss function, a strong sentiment service model (for customer communication) and a neutral and rigorous model (for risk warnings and billing announcements) are obtained, enhancing the friendliness of financial services while ensuring the accuracy of key information delivery.

[0051] In one embodiment, step S400 includes: S401. Obtain the target text and reference timbre audio and emotional intensity control scalar; S402. Label the target text with discrete target sentiment categories; S403. Input the target text, reference timbre audio, and discrete target emotion category into the strong emotion expression model and the neutral pronunciation model for processing, and output the strong emotion feature vector field and the neutral feature vector field. S404. Weight the neutral feature vector field according to the emotional intensity control scalar to generate the surrogate model output; S405. The strong sentiment feature vector field and the surrogate model output are fused using the collaborative guidance formula to output the final prediction vector field. S400, Convert the final predicted vector field into the target Mel spectrum.

[0052] In this embodiment, the first step is to acquire and preprocess the inference input information, which is fundamental to ensuring that the synthesized speech meets the user's needs. Three key inputs are required: first, the target text (the text content the user wants to convert into speech, such as "Today's performance was fantastic"), which is the semantic core of speech synthesis; second, reference timbre audio, by extracting the timbre features of this audio (such as the speaker's voice and timbre texture), allowing the synthesized speech to match the target timbre and meet personalized needs; and third, the emotion intensity control scalar α, whose value range is usually [0, k] (k≥1), which is the core parameter for achieving continuous emotion control. The closer the α value is to 0, the more neutral the synthesized speech; the larger the α value, the stronger the emotional expression. Simultaneously, the target text needs to be labeled with discrete target emotion categories (such as "happy," "excited," "sad," etc.) to clarify the emotional direction of the synthesized speech. This is consistent with the emotion label design in the pre-training stage of the basic model, ensuring that the model can accurately locate the core direction of emotional expression.

[0053] Next comes the generation of strong sentiment and neutral feature vector fields, a crucial step for the dual-model architecture to perform its specialized functions. The target text, reference audio timbre, and discrete target sentiment categories are input into the optimized strong sentiment expression model (v...). θ + ) and neutral pronunciation model (v θ - The strong emotion expression model, based on its "emotion expert" positioning, combines the target emotion category with text semantics to generate a strong emotion feature vector field (v) containing strong emotional acoustic features (such as pitch fluctuations, rhythm changes, and energy intensity). θ + (x,t,emo)), this vector field directly reflects the feature distribution of highly emotional speech; the neutral pronunciation model, relying on its "neutral expert" capability, ignores excessive emotional fluctuations and generates a neutral feature vector field (v) that focuses on pronunciation clarity and rhythmic naturalness. θ - (x,t,c neutral This ensures the stability of the basic voice quality.

[0054] Next is the calculation of the surrogate model output and the fusion of features from both models, which is the core innovation for achieving continuous control of sentiment intensity. First, based on the sentiment intensity control scalar α, the neutral feature vector field is weighted and calculated to obtain the surrogate model output (v... proxy The specific formula is: v proxy (x,t,α)=(1 α)*v θ - (x,t,c neutral )+α*v θ - (x,t,c unconditional ) calculate, where c neutral Representing a neutral emotional condition, c unconditional This represents an unconditional (no explicit sentiment bias) condition. This weighting process essentially dynamically adjusts the ratio of "pure neutral features" to "unconstrained features" in the neutral model output through the α value, providing flexible foundational features for subsequent sentiment intensity adjustment. Then, the strong sentiment feature vector field is fused with the surrogate model output through a collaborative guidance formula: the final predicted vector field v final (x,t,α)=v θ + (x,t,emo)+s*(v θ + (x,t,emo) v proxy(x,t,α)), where s is the standard CFG (classifier-independent guidance) guidance coefficient. CFG technology enhances the model's guidance in generating the target direction. Here, this formula is used to make strong sentiment features complementary to the dynamically adjusted surrogate model output. Changes in the value of α directly affect v. final The characteristic distribution changes, thereby achieving a continuous transition from "complete neutrality" to "extreme emotion".

[0055] Finally, the target Mel spectrum is output, which is the final result of the inference stage. The final predicted vector field obtained by fusion is converted into the target Mel spectrum through the generation logic of the stream matching model. This spectrum not only contains the semantic information of the target text and the timbre features of the reference audio, but also accurately matches the discrete emotion category and continuous emotion intensity specified by the user. It is the core feature carrier for subsequent conversion into an audible speech signal. After this step is completed, the target Mel spectrum can be further converted into the final speech waveform through a vocoder, realizing speech synthesis with controllable emotion intensity and completely solving the pain point of coarse emotion control granularity in existing technologies.

[0056] For example, in the healthcare field, this process can meet the speech synthesis needs of diverse doctor-patient communication scenarios. For instance, when generating rehabilitation guidance speech for patients, the process first obtains the target text (e.g., "daily rehabilitation training steps"), reference doctor's voice audio (ensuring patient familiarity and trust), and an emotional intensity control scalar (e.g., setting a higher reassurance intensity for anxious patients), and labels it with a discrete target emotional category of "reassurance." These three elements are then input into a strong emotional expression model (outputting a feature vector field with reassurance characteristics) and a neutral pronunciation model (outputting a clear and professional feature vector field), respectively. The neutral feature vector field is weighted according to the emotional intensity scalar to obtain the surrogate model output, which is then fused with the strong emotional feature vector field to generate the final prediction vector field, ultimately converting it into the target Mel spectrum. The synthesized speech in this way can not only convey accurate rehabilitation information but also alleviate patient anxiety through controllable reassurance, improving the healthcare service experience.

[0057] In the fintech field, this process can be adapted to the voice requirements of different financial service scenarios. For example, when generating voice announcements for financial products for users, the process first obtains the target text (such as "Explanation of the returns and risks of a certain fund"), references the audio of a dedicated customer service representative (to enhance user familiarity), and an emotional intensity control scalar (setting a high seriousness intensity for risk-related sections), labeling it with discrete target emotional categories such as "serious" and "objective." These three elements are then input into a strong emotional expression model (outputting a feature vector field with serious characteristics) and a neutral pronunciation model (outputting a clear and easy-to-understand feature vector field). The neutral feature vector field is weighted according to the emotional intensity scalar to obtain the output of the proxy model, which is then fused to generate the final predicted vector field and converted into the target Mel spectrum. The synthesized voice maintains objectivity when introducing returns and strengthens seriousness when highlighting risks, ensuring accurate information delivery while guiding users to pay attention to risks and enhancing the professionalism of financial services.

[0058] In one embodiment, step S300 specifically includes: S3041. Obtain preferred and non-preferred samples based on the positive sentiment Mel spectrum and the negative / neutral sentiment Mel spectrum; S3042. Calculate the prediction bias of the positive sentiment model and the deviation between the positive sentiment model and the base model based on the preference samples; S3043. Calculate the prediction bias of the negative / neutral sentiment model and the bias between the negative / neutral sentiment model and the base model based on the unbiased samples; S3044. Adjust the weights of the positive sentiment model prediction bias, the positive sentiment model vs. the base model bias, the negative / neutral sentiment model prediction bias, and the negative / neutral sentiment model vs. the base model bias respectively using the temperature coefficient. S3045. The prediction bias of the positive sentiment model after weight adjustment, the bias between the positive sentiment model and the base model, the prediction bias of the negative / neutral sentiment model, and the bias between the negative / neutral sentiment model and the base model are integrated to form a dual-model loss function. S3046. The trained positive sentiment model and negative / neutral sentiment model are decoupled and optimized using a dual-model loss function to obtain a strong sentiment expression model and a neutral pronunciation model.

[0059] In this embodiment, by accurately calculating model bias and dynamically adjusting bias weights, a loss function that takes into account the functional positioning of both models is constructed, ultimately achieving decoupling optimization and professional upgrading of the two models.

[0060] First, the preferred and non-preferred samples must be clearly defined, as this forms the basis for subsequent bias calculations. According to the technical definition, from the constructed sentiment spectrum data, the positive sentiment Mel spectrum is directly used as the preferred sample (x). w These samples carry the emotional acoustic features that the model needs to reinforce (such as high pitch and fast tempo when happy), and are the target learning objects for positive emotion models; at the same time, the Mel spectrum of negative / neutral emotions is used as non-preference samples (x). l These samples focus on clear, natural, and neutral speech features, serving as the core optimization basis for the negative / neutral sentiment model. This sample definition, which clarifies the "target benchmark" and "comparison benchmark" for calculating the bias in the dual-model approach, establishes a clear target benchmark and a comparison benchmark.

[0061] Next is the calculation of the four types of model biases, which is crucial for quantifying the performance differences between models. For the preferred sample (x... w Two types of bias need to be calculated: one is the positive sentiment model prediction bias, i.e., the positive sentiment model (v θ +The difference between the predicted results of the preference sample and the true characteristics of the preference sample (positive sentiment Mel spectrum) The first bias reflects the accuracy of the positive sentiment model in fitting the target sentiment features; the second bias is the difference between the positive sentiment model and the base model, that is, the difference between the positive sentiment model's prediction results for preferred samples and the base model's prediction results (v...). ref The difference between prediction results for the same preference sample ( This is used to prevent the positive emotion model from deviating from its basic speech synthesis capabilities when enhancing emotions. For non-preferred samples (x... l Similarly, two types of bias are calculated: one is the negative / neutral sentiment model prediction bias, i.e., the negative / neutral sentiment model (v θ - The difference between the prediction results for unbiased samples and the true characteristics of unbiased samples (negative / neutral sentiment Mel spectrum) The first is to measure the fitting effect of the neutral speech features; the second is the deviation between the negative / neutral sentiment model and the base model, that is, the difference between the prediction results of the negative / neutral sentiment model for unbiased samples and the prediction results of the base model for the same unbiased sample. This ensures that the basic capabilities are not lost when optimizing neutral performance.

[0062] Next, the temperature coefficient is used to adjust the bias weights, enabling dynamic control over the degree of bias influence. A temperature coefficient (T) is introduced. This coefficient is a commonly used parameter in machine learning, its core function being to adjust the "steepness" of the probability distribution or the "weight proportion" of biases. In this embodiment, it is used to balance the contributions of the four types of biases to the loss function. Specifically, the four types of biases are weighted using the temperature coefficient T: when it is necessary to enhance the influence of a certain type of bias (e.g., to make the positive sentiment model fit the preferred samples more accurately), the weight of the corresponding bias can be increased by adjusting T; when it is necessary to weaken the interference of a certain type of bias (e.g., to reduce the constraint of the base model on the negative / neutral model), the weight of the corresponding bias can be decreased by decreasing T. This dynamic adjustment mechanism allows the loss function to flexibly adjust its optimization focus according to the training progress and functional requirements of the two models, avoiding optimization imbalance caused by a single weight.

[0063] Finally, the construction of the dual-model loss function and the optimization of model decoupling are the core aspects of achieving dual-model specialization. The four types of biases, after weight adjustment, are integrated according to innovative technological design to form the dual-model loss function (L...). EMO The function transforms the weighted result of the four types of biases into a backpropagable loss value through the logarithmic function and the sigmoid function (σ). The overall formula is as follows:

[0064] To ensure the loss value effectively reflects the overall performance gap between the two models, this loss function was then used to decouple and optimize the trained positive sentiment model and negative / neutral sentiment model. "Decoupling" means that the two models do not interfere with each other during optimization. The positive sentiment model adjusts its parameters only through its own biases (bias in predicting preferred samples and biases from the base model) in the loss function, further enhancing its emotional expression capabilities. The negative / neutral sentiment model optimizes its parameters only through its own biases (bias in predicting unpreferred samples and biases from the base model), consolidating neutral pronunciation quality. After multiple rounds of iterative optimization, the positive sentiment model is upgraded to a strong sentiment expression model, capable of stably generating emotionally rich and rhythmically exaggerated speech; the negative / neutral sentiment model is upgraded to a neutral pronunciation model, capable of consistently outputting clear and natural neutral speech, completely solving the core pain point of existing technologies where a single model cannot simultaneously achieve both emotional expression and neutrality.

[0065] For example, in the healthcare field, this process can optimize the performance of voice models in doctor-patient communication scenarios. First, from a medical emotional voice database, positive emotional Mel spectra of "soothing" and "encouraging" are used as preferred samples, while negative / neutral emotional Mel spectra of "rigorous" and "objective" are used as non-preferred samples. Based on the preferred samples, the prediction bias of the positive emotional model (used for psychological counseling) and its deviation from the base model are calculated. Based on the non-preferred samples, two types of biases of the negative / neutral model (used for medication guidance) are calculated. The bias weights are adjusted using a temperature coefficient, such as increasing the prediction bias weight of the positive model to enhance the soothing effect. The biases are integrated to form a dual-model loss function. After decoupling and optimization, a strong emotional soothing model and a neutral professional model are obtained, ensuring that the voice both matches the emotional needs of the scenario and maintains the accuracy of medical information.

[0066] In the fintech field, this process enhances the scenario adaptability of financial service voice models. From a financial sentiment voice database, positive sentiment Mel spectrum ("friendly") is set as preferred samples, while negative / neutral sentiment Mel spectrum ("serious") is set as unpreferred samples. The prediction bias between the positive model (used for financial advice) and the negative / neutral model (used for risk warnings), as well as the bias against the base model, are calculated. A temperature coefficient is used to adjust the weights, such as increasing the weight of the negative / neutral model's prediction bias to ensure the rigor of risk warnings. This is then integrated to form a decoupled optimization loss function, resulting in a strong sentiment service model and a neutral, rigorous model. This allows the voice to enhance service friendliness while ensuring the accurate delivery of key financial information.

[0067] In one embodiment, a dual-model-based speech synthesis device is provided, which corresponds one-to-one with the dual-model-based speech synthesis method described in the above embodiments. (Refer to...) Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the speech synthesis device based on a dual-model architecture of the present invention. The modules include a basic model training module 10, a dual-model construction module 20, a dual-model training module 30, and a speech synthesis module 40. Detailed descriptions of each functional module are as follows: The basic model training module 10 is used to pre-train the non-emotionally modeled stream matching model based on a general domain text and emotional speech library to generate a basic model. The dual-model construction module 20 is used to obtain the network structure and model weights of the base model, and to construct a functionally differentiated dual model based on the network structure and model weights. The dual-model training module 30 is used to construct preference data pairs from the original text sequence and input the preference data pairs into the dual model for training. The speech synthesis module 40 is used to acquire target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

[0068] In one embodiment, the stream matching model module includes: Obtain general domain text, analyze the general domain text, and obtain the original text sequence; Obtain an emotional speech database, analyze the emotional speech database, and obtain discrete emotional tags; Discrete sentiment tags are appended to the beginning of the original text sequence to form a new text sequence; Discrete sentiment types are converted into discrete sentiment vectors through pre-trained word embeddings; The text sequence and discrete sentiment vector are input into the stream matching model. The text encoder of the stream matching model is adjusted using the text sequence and discrete sentiment vector to generate a new stream matching model.

[0069] In one embodiment, the base model training module 10 includes: The emotional speech database is parsed to extract emotional speech waveforms; The emotional speech waveform is converted into a Mel spectrum using a vocoder; The text sequence, discrete sentiment tags, and Mel spectrum are input into the new flow matching model for pre-training to generate the base model.

[0070] In one embodiment, the dual-model construction module 20 includes: Obtain the network structure and model weights of the basic model; Based on the network structure described, construct a positive sentiment model and a negative / neutral sentiment model; The positive sentiment model and the negative / neutral sentiment model are initialized using the model weights.

[0071] In one embodiment, the dual-model training module 30 includes: Based on the emotional speech library, positive emotional Mel spectra and negative / neutral emotional Mel spectra are matched for the original text sequence; Preference data pairs are constructed using the positive sentiment Mel spectrum and the negative / neutral sentiment Mel spectrum; The preference data is input into the positive sentiment model and the negative / neutral sentiment model respectively for training; A dual-model loss function is used to decouple and optimize the trained positive sentiment model and negative / neutral sentiment model, resulting in a strong sentiment expression model and a neutral pronunciation model.

[0072] In one embodiment, the speech synthesis module 40 includes: Acquire the target text and reference timbre audio and emotional intensity control scalar; The target text is labeled with discrete target sentiment categories; The target text, reference timbre audio, and discrete target emotion category are respectively input into the strong emotion expression model and the neutral pronunciation model for processing, and the strong emotion feature vector field and the neutral feature vector field are output. The neutral feature vector field is weighted according to the emotional intensity control scalar to generate the surrogate model output; The strong sentiment feature vector field and the surrogate model output are fused through a collaborative guided formula to output the final prediction vector field. The final predicted vector field is converted into the target Mel spectrum.

[0073] In one embodiment, the decoupling and optimization of the trained positive sentiment model and negative / neutral sentiment model using a dual-model loss function to obtain a strong sentiment expression model and a neutral pronunciation model specifically includes: Based on the positive sentiment Mel spectrum and the negative / neutral sentiment Mel spectrum, preferred and non-preferred samples are obtained; The prediction bias of the positive sentiment model and the bias between the positive sentiment model and the base model are calculated based on the preference samples. The negative / neutral sentiment model prediction bias and the negative / neutral sentiment model deviation from the base model are calculated based on the unbiased samples. The weights of the positive sentiment model prediction bias, the positive sentiment model vs. the base model bias, the negative / neutral sentiment model prediction bias, and the negative / neutral sentiment model vs. the base model bias are adjusted by temperature coefficients. The prediction biases of the weight-adjusted positive sentiment model, the positive sentiment model and the base model, the negative / neutral sentiment model, and the negative / neutral sentiment model and the base model are integrated to form a dual-model loss function; A dual-model loss function is used to decouple and optimize the trained positive sentiment model and negative / neutral sentiment model, resulting in a strong sentiment expression model and a neutral pronunciation model.

[0074] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a dual-model-based speech synthesis method on the server side.

[0075] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a user-side speech synthesis method based on a dual-model approach. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: A base model is generated by pre-training a stream matching model without emotion modeling based on a general domain text and emotion speech library. Obtain the network structure and model weights of the base model, and construct a functionally differentiated dual model based on the network structure and model weights; Preference data pairs are constructed using the original text sequence, and these preference data pairs are then used as input to train the dual model. Obtain the target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

[0076] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: A base model is generated by pre-training a stream matching model without emotion modeling based on a general domain text and emotion speech library. Obtain the network structure and model weights of the base model, and construct a functionally differentiated dual model based on the network structure and model weights; Preference data pairs are constructed using the original text sequence, and these preference data pairs are then used as input to train the dual model. Obtain the target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

[0077] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0080] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on a dual-model approach, characterized in that, Includes the following steps: A base model is generated by pre-training a stream matching model without emotion modeling based on a general domain text and emotion speech library. Obtain the network structure and model weights of the base model, and construct a functionally differentiated dual model based on the network structure and model weights; Preference data pairs are constructed using the original text sequence, and these preference data pairs are then used as input to train the dual model. Obtain the target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

2. The speech synthesis method based on a dual-model as described in claim 1, characterized in that, Before generating the base model by pre-training the non-emotionally modeled stream matching model based on a general domain text and emotional speech library, the process also includes: Obtain general domain text, process the general domain text, and generate an original text sequence; Obtain an emotional speech database, process the emotional speech database, and extract discrete emotional tags; Discrete sentiment tags are appended to the beginning of the original text sequence to form a new text sequence; Discrete sentiment types are converted into discrete sentiment vectors through pre-trained word embeddings; The text sequence and discrete sentiment vector are input into the stream matching model. The text encoder of the stream matching model is adjusted using the text sequence and discrete sentiment vector to generate a new stream matching model.

3. The speech synthesis method based on a dual-model as described in claim 2, characterized in that, The process of pre-training a stream matching model without emotion modeling based on a general domain text and emotion speech library to generate a base model includes: The emotional speech database is parsed to extract emotional speech waveforms; The emotional speech waveform is converted into a Mel spectrum using a vocoder; The text sequence, discrete sentiment tags, and Mel spectrum are input into the new flow matching model for pre-training to generate the base model.

4. The speech synthesis method based on a dual-model as described in claim 1, characterized in that, The step of obtaining the network structure and model weights of the base model, and constructing a functionally differentiated dual model based on the network structure and model weights, includes: Obtain the network structure and model weights of the basic model; Based on the network structure described, construct a positive sentiment model and a negative / neutral sentiment model; The positive sentiment model and the negative / neutral sentiment model are initialized using the model weights.

5. The speech synthesis method based on a dual-model as described in claim 1, characterized in that, The step of constructing preference data pairs from the original text sequence and then using these preference data pairs as input to train the dual model includes: Based on the emotional speech library, positive emotional Mel spectra and negative / neutral emotional Mel spectra are matched for the original text sequence; Preference data pairs are constructed using the positive sentiment Mel spectrum and the negative / neutral sentiment Mel spectrum; The preference data is input into the positive sentiment model and the negative / neutral sentiment model respectively for training; A dual-model loss function is used to decouple and optimize the trained positive sentiment model and negative / neutral sentiment model, resulting in a strong sentiment expression model and a neutral pronunciation model.

6. The speech synthesis method based on a dual-model as described in claim 1, characterized in that, The process of acquiring the target text and reference timbre audio, inputting the target text and reference timbre audio into the trained dual model, and outputting the target Mel spectrum includes: Acquire the target text and reference timbre audio and emotional intensity control scalar; The target text is labeled with discrete target sentiment categories; The target text, reference timbre audio, and discrete target emotion category are respectively input into the strong emotion expression model and the neutral pronunciation model for processing, and the strong emotion feature vector field and the neutral feature vector field are output. The neutral feature vector field is weighted according to the emotional intensity control scalar to generate the surrogate model output; The strong sentiment feature vector field and the surrogate model output are fused through a collaborative guided formula to output the final prediction vector field. The final predicted vector field is converted into the target Mel spectrum.

7. The speech synthesis method based on a dual-model as described in claim 5, characterized in that, The method employs a dual-model loss function to decouple and optimize the trained positive sentiment model and negative / neutral sentiment model, resulting in a strong sentiment expression model and a neutral pronunciation model. Specifically, this includes: Based on the positive sentiment Mel spectrum and the negative / neutral sentiment Mel spectrum, preferred and non-preferred samples are obtained; The prediction bias of the positive sentiment model and the bias between the positive sentiment model and the base model are calculated based on the preference samples. The negative / neutral sentiment model prediction bias and the negative / neutral sentiment model deviation from the base model are calculated based on the unbiased samples. The weights of the positive sentiment model prediction bias, the positive sentiment model vs. the base model bias, the negative / neutral sentiment model prediction bias, and the negative / neutral sentiment model vs. the base model bias are adjusted by temperature coefficients. The prediction biases of the weight-adjusted positive sentiment model, the positive sentiment model and the base model, the negative / neutral sentiment model, and the negative / neutral sentiment model and the base model are integrated to form a dual-model loss function; A dual-model loss function is used to decouple and optimize the trained positive sentiment model and negative / neutral sentiment model, resulting in a strong sentiment expression model and a neutral pronunciation model.

8. A speech synthesis device based on a dual-model approach, characterized in that, The dual-model-based speech synthesis device includes: The basic model training module is used to pre-train the non-emotionally modeled stream matching model based on a general domain text and emotional speech library to generate a basic model. A dual-model construction module is used to obtain the network structure and model weights of the base model, and to construct a functionally differentiated dual model based on the network structure and model weights; The dual-model training module is used to construct preference data pairs from the original text sequence and input the preference data pairs into the dual model for training. The speech synthesis module is used to acquire target text and reference timbre audio, input the target text and reference timbre audio into the trained dual model, and output the target Mel spectrum.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a dual-model-based speech synthesis program stored in the memory and executable on the processor, wherein the dual-model-based speech synthesis program, when executed by the processor, implements the steps of the dual-model-based speech synthesis method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a dual-model-based speech synthesis program, which, when executed by a processor, implements the steps of the dual-model-based speech synthesis method as described in any one of claims 1-7.

Images