A controllable speech synthesis method and system based on chain-of-thought style reasoning

Abstract

The application discloses a controllable speech synthesis method and system based on thought chain style reasoning, and belongs to the field of speech synthesis. A multimodal large model generates target text, context description and thought chain sequence based on a speech sample, and extracts a target semantic sequence from the speech sample to form a four-tuple training data; the target text and the context description are taken as input, the thought chain sequence is taken as an intermediate supervision signal, and the target semantic sequence is taken as a final supervision signal, so that a thought chain enhanced semantic language model is trained; a target semantic sequence is generated according to a to-be-synthesized target text and a context description thereof; target acoustic features are generated based on the target semantic sequence and semantic and acoustic features of a reference speech, and are converted into synthesized speech. The application realizes complete automation of style control, does not need manual annotation, significantly improves the naturalness of expression and logical coherence of synthesized speech under a complex context, and achieves a high-fidelity and highly controllable speech synthesis effect.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis, and more particularly to a controllable speech synthesis method and system based on thought chain style reasoning. Background Technology

[0002] Currently, speech synthesis technology has attracted widespread attention and become one of the core research directions in the field of artificial intelligence. Speech synthesis aims to generate natural and fluent speech signals from given text information. With the development of neural network generative modeling and large-scale training data, current speech synthesis systems have achieved significant breakthroughs in sound quality and naturalness, and are widely used in various application scenarios such as human-computer interaction, content creation, and digital human-driven systems.

[0003] Current mainstream controllable speech synthesis methods and technologies are mainly based on instruction-driven techniques, that is, controlling the style of generated speech through natural language descriptions. Although existing methods provide some style control capabilities, they still rely on users to manually input detailed style description information (such as emotion, speech rate, pitch, etc.), which requires extremely high manual costs in multi-scenario, long text generation tasks and cannot achieve true automated generation. Moreover, existing methods lack the ability to understand deep context, often failing to transform abstract contextual information (such as dialogue background, scene description) into fine-grained prosodic control parameters, resulting in synthesized speech exhibiting logical discontinuities or emotional incoherence in complex contexts. Summary of the Invention

[0004] To overcome the above problems, this invention proposes a controllable speech synthesis method and system based on thought chain style reasoning. It makes full use of deep semantic information in the context and automatically obtains refined style descriptions through thought chain reasoning, thereby achieving high-fidelity speech generation with logical consistency.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] In a first aspect, this invention proposes a controllable speech synthesis method based on thought chain style reasoning, comprising the following steps:

[0007] S1. Obtain speech samples. The multimodal large model generates target text, context description, and thought chain sequence based on the speech samples. The pre-trained semantic encoder extracts the target semantic sequence from the speech samples to form a quadruple training data. The thought chain sequence includes intermediate reasoning process text and style description text used to describe speech style attributes.

[0008] S2. Using the target text and context description in the training data of the quadruples as input, the thought chain sequence as the intermediate supervision signal and the target semantic sequence as the final supervision signal, the large language model is trained by maximizing the conditional probability to obtain the thought chain enhanced semantic language model.

[0009] S3. Input the target text to be synthesized and its corresponding context description into the mind chain enhanced semantic language model to generate the mind chain sequence and the target semantic sequence to be synthesized in sequence.

[0010] S4. Based on the target semantic sequence to be synthesized, the reference semantic features of the reference speech, and the reference acoustic features, generate the target acoustic features and convert the target acoustic features into synthesized speech.

[0011] Furthermore, the context description includes the dialogue scenario, the speaker's identity, and the speaker's emotional state.

[0012] Furthermore, the speech style attributes include speech rate, volume, pitch, and emotion.

[0013] Furthermore, the multimodal large model, guided by preset prompts, performs the following operations in sequence:

[0014] Transcribe speech samples into target text;

[0015] Context description based on speech samples and target text;

[0016] The reasoning process is based on the target text and context description. First, intermediate reasoning process text is generated, and then structured style description text is generated based on the intermediate reasoning process text.

[0017] Furthermore, the loss function for training the large language model includes thought chain loss and semantic sequence loss; the thought chain loss is the cross-entropy loss between the predicted thought chain sequence and the actual thought chain sequence; the semantic sequence loss is the cross-entropy loss between the predicted target semantic sequence and the actual target semantic sequence.

[0018] Furthermore, in S4, the process of generating the target acoustic features includes:

[0019] The target semantic sequence to be synthesized is concatenated with the reference semantic features to form a complete semantic input feature;

[0020] After referencing the acoustic features, a Gaussian noise sequence corresponding to the length of the target semantic sequence to be synthesized is concatenated to form the acoustic input features to be processed;

[0021] The semantic input features and the acoustic input features to be processed are input into the prefix conditional renderer. The Gaussian noise sequence is gradually denoised using the reference acoustic features and semantic input features as conditions to obtain the target acoustic features.

[0022] Furthermore, the prefix conditional renderer is a conditional diffusion model, and the training process includes:

[0023] The acoustic input features of the speech samples are randomly split into a prefix part and a target part along the time axis;

[0024] During the forward diffusion process, the target portion is gradually noise-added to Gaussian noise;

[0025] In the reverse denoising process, the target part is gradually restored from Gaussian noise by using the prefix part and semantic input features as conditions.

[0026] Secondly, this invention proposes a controllable speech synthesis system based on thought chain style reasoning, for implementing the aforementioned controllable speech synthesis method based on thought chain style reasoning.

[0027] Thirdly, the present invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned controllable speech synthesis method based on thought chain style reasoning.

[0028] Fourthly, the present invention provides a computer electronic device, including a memory and a processor;

[0029] The memory is used to store computer programs;

[0030] The processor is used to implement the above-described controllable speech synthesis method based on thought chain style reasoning when executing the computer program.

[0031] Compared with the prior art, the beneficial effects of this invention are:

[0032] (1) This invention achieves fully automated speech style control by constructing a quadruple training data consisting of target text, context description, thought chain sequence, and target semantic sequence. Among them, the thought chain sequence is automatically generated by a multimodal large model based on speech samples, without requiring the user to manually input specific style parameters such as emotion, speech rate, and pitch.

[0033] (2) During the training of the large language model, the thought chain sequence is used as the intermediate supervision signal and the target semantic sequence is used as the final supervision signal, which gives the large language model the ability to understand and reason about deep context. It can transform the target text and context description into structured prosodic and emotional features through thought chain reasoning, effectively overcoming the shortcomings of existing methods in synthesizing speech in complex contexts with logical breaks and emotional incoherence, and ensuring that the style of the subsequent synthesized speech is highly matched with the input context.

[0034] (3) In the final speech synthesis stage, the target semantic sequence to be synthesized is combined with the reference semantic features and reference acoustic features of the reference speech, and the semantic features and acoustic features are decoupled. While preserving the timbre of the reference speech, the style attributes derived from the thought chain are accurately applied, achieving a highly controllable style and high fidelity speech synthesis effect. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a controllable speech synthesis method based on thought chain style reasoning.

[0036] Figure 2 This is a schematic diagram of a thought chain-enhanced semantic language model;

[0037] Figure 3 This is a schematic diagram of the training phase of the acoustic rendering module;

[0038] Figure 4 This is a schematic diagram of the inference stage of the acoustic rendering module;

[0039] Figure 5 This is a schematic diagram of a controllable speech synthesis system based on thought chain style reasoning;

[0040] Figure 6 This is a schematic diagram of a computer electronic device. Detailed Implementation

[0041] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0042] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] This invention proposes a controllable speech synthesis method based on thought chain style reasoning. It utilizes the trained thought chain to enhance the reasoning ability of the semantic language model and automatically parses fine speech style attributes from text and context description, replacing the tedious manual annotation process in traditional methods.

[0045] like Figure 1 As shown, the method mainly includes the following steps:

[0046] S1. Construct a quadruple dataset.

[0047] Specifically, the first step is to acquire a massive amount of voice samples, which should cover a wide range of scenarios, emotions, and speaking styles, such as recordings from audiobooks, podcasts, film and television dubbing, and everyday conversations.

[0048] For each speech sample, perform the following operations:

[0049] (1.1) Semantic Feature Extraction: A pre-trained self-supervised speech model is used as a semantic encoder to extract discrete or continuous semantic representations from the speech waveform, defined as the target semantic sequence. This sequence contains the content information of the speech, stripping away acoustic details such as timbre. For example, for the sentence "The weather is really nice today," its semantic sequence might be a series of vectors representing phonemes or semantic units. In this embodiment, the pre-trained self-supervised speech model uses HuBERT or wav2vec2.0.

[0050] (1.2) Generate target text, context description and construct thought chain sequence: Input the speech sample into a powerful multimodal large model (e.g., a model with speech understanding and text generation capabilities). The multimodal large model has speech understanding and text generation capabilities and has a huge number of parameters. It can be implemented using existing multimodal large models.

[0051] By designing prompts, the multimodal large model is guided to complete two tasks:

[0052] Speech-to-text: accurately transcribing speech samples into target text T.

[0053] Contextual reasoning: Based on the speech sample and the transcribed text T, the contextual description S of its occurrence is inferred. The contextual description includes the dialogue scene, the speaker's emotional description, etc., which is used to guide the subsequent thought chain reasoning process and analyze the rhythmic and emotional features hidden behind the target text.

[0054] For example, the context description could be: "Scene: A cozy living room, where a kind grandmother is answering her grandson's questions about his childhood. The grandmother's tone is full of nostalgia and patience."

[0055] Based on the above context description S and target text T, the multimodal large model further performs chain-of-thought reasoning. This process simulates the thinking steps of human experts when specifying a speech style, and generates a chain-of-thought sequence. The chain-of-thought sequence includes the intermediate reasoning process text based on the target text and the context description, and the style description text for describing speech style attributes, where the speech style attributes include speech rate, volume, pitch, and emotion.

[0056] For example, the multimodal large model will first generate the intermediate reasoning process text: "The speaker is a grandmother, facing a curious grandson, and her answer should be full of care and patience. The topic is about childhood, so the tone should have a sense of nostalgia. This is a daily conversation, the speech rate should not be too fast, and the pitch is moderate."; Then, based on this reasoning, the multimodal large model generates the final structured style description text, which can be a JSON format string, for example: {Emotion: warm and nostalgic; Speech rate: medium-slow; Pitch: medium; Volume: soft}. The above intermediate reasoning process text and style description text together constitute the chain-of-thought sequence C.

[0057] Through the above steps, a quadruple training data (T, S, C, O) is constructed for each speech sample, where T represents the target text, S represents the context description, C represents the chain-of-thought sequence, and O represents the target semantic sequence.

[0058] In a specific implementation of the present invention, an optional prompt content of the multimodal large model is as follows:

[0059] "You are an assistant that helps users generate detailed instructions for a text-to-speech system. The user will provide a speech clip, please complete the following tasks based on the speech:

[0060] 1) Transcribe the speech sample into the target text, which must be strictly contained within the <Transcribed Text>< / Transcribed Text> tags;

[0061] 2) Predict the preceding context, that is, the text or dialogue content that may appear before the speech clip, which must be strictly contained in <context>< / context> tags;

[0062] 3) Perform reasoning, and then provide the user with answers about the speech attributes, including: emotion, speech rate, pitch, and volume. The intermediate reasoning process and the style description text answer must be strictly contained in <think>< / think> and <answer>< / answer> tags respectively.

[0063] Audio: Speech sample

[0064] Output example: "A"; "

[0065] The above prompt text is only an optional example. The output example "A" is for illustrative reference format and does not represent the actual output content. Those skilled in the art can adjust, supplement, or modify the prompt text according to actual application needs, and can also redesign a prompt structure more suitable for specific scenarios based on the above template.

[0066] S2. Use the quadruple dataset to train the mind chain-enhanced semantic language model.

[0067] Train a large language model using the quadruple data constructed in step S1, such as... Figure 2 As shown, latent space reasoning is performed using the target text and context description in the quadruplet training data as input, and the labeled thought chain sequence and target semantic sequence as supervision and guidance signals.

[0068] During the reasoning process, the large language model calculates the correlation between various lexical units in the context through an attention mechanism, generating a series of intermediate feature vectors that represent the emotional tone and prosodic intent behind the text. This multi-step reasoning process generates a sequence of thought chains. The generation process of the thought chain sequence is as follows:

[0069]

[0070] Where C represents the thought chain sequence and T represents the target text. Indicates the target text length, and S represents the context description. This represents the result of the thought chain generated up to step t. This represents the token corresponding to the thought chain result generated in step t. Indicates the t-th step according to generate The probability, Indicates according to generate The probability is used to generate the thought chain sequence C by maximizing the probability.

[0071] Furthermore, the target semantic sequence is predicted based on the thought chain sequence to generate a predicted semantic sequence. This process combines the target text, context description, and thought chain sequence to calculate the semantic tag probability distribution at each time step, thus realizing the transformation from text input to discrete semantic sequence.

[0072] Specifically, large language models, based on attention mechanisms, predict discrete target semantic sequences by maximizing conditional probabilities. The conditional probability distribution can be represented as:

[0073]

[0074] in, Indicating in the thought chain sequence Context description Target text and before One generated semantic token Based on this, the first A semantic token The probability distribution; K represents the length of the target semantic sequence. This represents the parameters of a large language model.

[0075] The loss function consists of two parts:

[0076] Thought chain loss: Cross-entropy loss between the predicted thought chain sequence and the actual thought chain sequence. This loss term forces the model to learn how to make correct style inferences.

[0077] Semantic sequence loss: Cross-entropy loss between the predicted target semantic sequence and the true target semantic sequence. This loss term ensures that the model can generate accurate speech content representations based on the inferred style.

[0078] By introducing thought chain sequences as constraints, large language models can explicitly modulate the sentiment and prosodic rhythm during the generation of target semantic sequences. This modeling approach based on autoregressive probabilistic decomposition not only clarifies the logical dependencies between the input conditions (including the target text, contextual description, and thought chain sequence) and the target semantic sequence, but also significantly improves the convergence speed and generation stability of large language models in long text semantic modeling tasks through a structured parameter update mechanism. The thought chain-enhanced semantic language model trained in this way possesses powerful capabilities: given a text and its contextual description, it can sequentially generate thought chain sequences and target semantic sequences.

[0079] By constructing quadruplet training data and training a large language model, this invention transforms the style control process, which originally relied on manual fine-grained annotation, into an automatically learnable structured task. Specifically, the thought chain sequence serves as an explicit supervisory signal for the intermediate inference process, guiding the large language model to progressively derive fine-grained speech style attributes from the target text and contextual descriptions. This eliminates the need for users to manually input specific parameters such as speech rate, emotion, and pitch, achieving complete automation of style control and significantly reducing manual costs in multi-scenario, long text generation tasks.

[0080] S3, Training Acoustic Rendering Module.

[0081] The acoustic rendering module consists of a speech encoder, a prefix conditional renderer, and a speech decoder, and the training process is as follows: Figure 3 As shown, it includes the following steps:

[0082] (3.1) For a speech sample, its target semantic sequence has been obtained as semantic input features through step S1. Simultaneously, a speech encoder is used to extract continuous acoustic input features from the waveform of the speech sample. The sampling rates of the acoustic input features and the semantic input features may differ, requiring upsampling to align them to the same temporal resolution. In this embodiment, the speech encoder can employ a CNN, a Transformer, or a combination of both.

[0083] (3.2) Randomly split the acoustic input features into prefix parts along the time axis. and target part The proportion of split points is randomly sampled from a uniform distribution U(0.1, 0.9) to ensure that the model can adapt to prefixes of different lengths. For example, the first 30% can be used as the prefix and the last 70% as the target.

[0084] (3.3) The acoustic input features and semantic input features are used to form an initial mixed conditional feature sequence. The initial mixed conditional feature sequence is used as the input of the prefix conditional renderer. The prefix conditional renderer is essentially a conditional diffusion model. During the forward diffusion process, the conditional diffusion model first converts the target part of the acoustic input features into the conditional feature sequence. The noise is gradually increased to Gaussian noise until it becomes pure noise. In the reverse denoising process, the prefix portion of the acoustic input features is used... Using semantic input features as conditions, the system gradually predicts and removes noise, starting from pure noise, and ultimately reconstructs the target part of the input features; in this process, the prefix part of the acoustic input features... It provides cues on timbre and initial pronunciation style, while semantic input features provide global semantic and prosodic cues. The prefix conditional renderer needs to learn how to generate realistic speech features under the common constraints of these two conditions.

[0085] (3.4) The predicted target part is input into the speech decoder to generate the final synthesized speech. In this embodiment, the decoder uses a stack of transposed convolution and residual modules to convert high-dimensional speech features into the final speech waveform.

[0086] (3.5) The total loss function includes noise prediction loss, adversarial loss, and spectral loss. Noise prediction loss measures the accuracy of the model in predicting noise during the diffusion process; adversarial loss uses a discriminator to distinguish between real and synthesized speech, prompting the generator to produce more natural speech; spectral loss calculates the mean square error between the Mel spectra of synthesized and real speech. Through joint training, the trained acoustic rendering module is obtained.

[0087] S4, Reasoning and Speech Synthesis.

[0088] First, obtain the target text and provide a contextual description. Then, input the trained thought chain-enhanced semantic language model from step S2. The model will first perform thought chain reasoning and then output the target semantic sequence.

[0089] Secondly, inference is performed using the trained acoustic rendering module, such as... Figure 4 As shown. A reference audio segment is acquired. The pre-trained semantic encoder from step S1 is used to generate reference semantic input features. Using the speech encoder trained in step S3, reference acoustic input features are extracted from the reference audio segment. The reference semantic input features are aligned with the reference acoustic input features. The target semantic sequence is concatenated after the reference semantic input features as semantic input features. A Gaussian noise sequence aligned with the target semantic sequence is added after the reference acoustic input features as acoustic input features. The semantic and acoustic input features form an initial mixed conditional feature sequence, which is input to a prefix conditional renderer. In the prefix conditional renderer, the reference acoustic input features and semantic input features in the acoustic input features are used as conditions to iteratively denoise the Gaussian noise sequence in the acoustic input features, ultimately generating the target acoustic features corresponding to the target semantic sequence. The target speech features are then input to a speech decoder to generate synthesized speech.

[0090] To verify the effectiveness of the present invention, the above method is applied to the following embodiments to demonstrate the technical effects of the present invention. The specific steps in the embodiments will not be repeated.

[0091] This invention was tested on the CoTSpeech benchmark dataset for enhanced speech synthesis using thought chains. 400 random samples from the dataset were selected as the test sample set for comparative experiments. To objectively evaluate the performance of this invention, it was trained on 2,500 hours of speech data containing 1.1 million thought chain style tags, and comprehensive tests were conducted on the quality of thought chains during the inference phase and the style consistency of speech generation.

[0092] This comparative test uses speech attribute accuracy as the core metric for evaluating the matching degree between the generated audio style and the target context. This evaluation metric aims to measure whether the audio generated by the model maintains a high degree of consistency with the target audio style in the real context in terms of emotion, pitch, volume, and speech rate. Specifically, in the evaluation process, the model infers speech with the corresponding style based on the input context; subsequently, by calculating the emotion category matching degree, average pitch difference, speech rate offset, and average volume difference, a quantitative evaluation of the accuracy of the generated speech style is achieved.

[0093] The benchmark models used for comparison included CosyVoice 2, TextrolSpeech, and Parler-TTS. During the experimental inference phase, the open-source implementations and pre-trained model weights officially released by these benchmarks were used to ensure that all models were compared under the same experimental conditions, thereby verifying the performance advantages of this invention in style control driven by thought chains. Since there is currently no speech synthesis model that infers audio style for specific scene inputs, we used a pre-trained Qwen2.5-7B-Instruct large text model for audio style inference during testing, and then input the corresponding style inference results into the benchmark models for controllable speech generation. For benchmark models requiring specific preprocessing (such as audio resampling), the pre-processed raw speech data was uniformly regarded as the ground truth for evaluating the generated speech.

[0094] The experimental results obtained by following the steps described in the specific implementation method are shown in Table 1.

[0095] Table 1: Comparative test results of controllable speech synthesis obtained from the CoTSpeech dataset in this invention.

[0096]

[0097] As shown in Table 1, overall, on the CoTSpeech test set, this invention significantly outperforms existing benchmark models in accuracy across various speech attributes, including speech rate, volume, pitch, and emotional expression. The average accuracy of this invention reaches 85.88%, far exceeding benchmark models such as TextrolSpeech (69.69%), Parler-TTS (72.63%), and CosyVoice2 (73.06%). Specifically, thanks to the thought chain style inference mechanism employed in this invention, the model can automatically parse and deduce fine-grained style descriptions from the input context, thus demonstrating superior control over speech rate and emotional expression. For example, in terms of emotional accuracy, this invention achieves 82.25%, compared to the best-performing existing model of only 60.75%. Furthermore, the present invention also achieves high accuracy of 82.00% and 88.25% in controlling pitch and volume, respectively. This indicates that by enhancing the decoupling architecture of semantic and acoustic information through the thought chain, the present invention has achieved a leapfrog improvement in the style consistency and realism of synthesized speech in specific contexts, demonstrating significant technological leadership.

[0098] It should also be noted that the controllable speech synthesis method based on thought chain style reasoning in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, another preferred embodiment of the present invention also provides a controllable speech synthesis system based on thought chain style reasoning, corresponding to the controllable speech synthesis method based on thought chain style reasoning provided in the above embodiments, such as... Figure 5 As shown, it includes:

[0099] The quadruple data construction module is used to acquire speech samples. The multimodal large model generates target text, context description and thought chain sequence based on the speech samples. The pre-trained semantic encoder extracts the target semantic sequence from the speech samples to form quadruple training data. The thought chain sequence includes intermediate reasoning process text and style description text used to describe speech style attributes.

[0100] The training module for the enhanced semantic language model of the thinking chain is used to take the target text and context description in the training data of the four-tuple as input, the thinking chain sequence as the intermediate supervision signal and the target semantic sequence as the final supervision signal, and train the large language model by maximizing the conditional probability to obtain the enhanced semantic language model of the thinking chain.

[0101] The semantic reasoning module is used to input the target text to be synthesized and its corresponding context description into the mind chain enhanced semantic language model, and generate the mind chain sequence and the target semantic sequence to be synthesized in sequence.

[0102] The acoustic rendering module is used to generate target acoustic features based on the target semantic sequence to be synthesized, the reference semantic features of the reference speech, and the reference acoustic features;

[0103] The speech decoding module is used to convert the target acoustic features into synthesized speech.

[0104] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0105] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0106] It is understood that the controllable speech synthesis method based on thought chain style reasoning in the above embodiments can essentially be implemented by a computer program. Therefore, another preferred embodiment of the present invention also provides a computer program product corresponding to the controllable speech synthesis method based on thought chain style reasoning provided in the above embodiments, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, it can implement the controllable speech synthesis method based on thought chain style reasoning as described in the above embodiments.

[0107] Similarly, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the controllable speech synthesis method based on thought chain style reasoning provided in the above embodiments, such as... Figure 6 As shown, it includes a memory and a processor;

[0108] The memory is used to store computer programs;

[0109] The processor is configured to implement the controllable speech synthesis method based on thought chain style reasoning in the above embodiments when executing the computer program.

[0110] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0111] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the controllable speech synthesis method based on thought chain style reasoning provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the controllable speech synthesis method based on thought chain style reasoning in the above embodiments.

[0112] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0113] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A controllable speech synthesis method based on thought chain style reasoning, characterized in that, Includes the following steps: S1. Obtain speech samples. The multimodal large model generates target text, context description, and thought chain sequence based on the speech samples. The pre-trained semantic encoder extracts the target semantic sequence from the speech samples to form a quadruple training data. The thought chain sequence includes intermediate reasoning process text and style description text used to describe speech style attributes. S2. Using the target text and context description in the training data of the quadruples as input, the thought chain sequence as the intermediate supervision signal and the target semantic sequence as the final supervision signal, the large language model is trained by maximizing the conditional probability to obtain the thought chain enhanced semantic language model. S3. Input the target text to be synthesized and its corresponding context description into the mind chain enhanced semantic language model to generate the mind chain sequence and the target semantic sequence to be synthesized in sequence. S4. Based on the target semantic sequence to be synthesized, the reference semantic features of the reference speech, and the reference acoustic features, generate target acoustic features and convert the target acoustic features into synthesized speech; In S4, the process of generating the target acoustic features includes: The target semantic sequence to be synthesized is concatenated with the reference semantic features to form a complete semantic input feature; After referencing the acoustic features, a Gaussian noise sequence corresponding to the length of the target semantic sequence to be synthesized is concatenated to form the acoustic input features to be processed; The semantic input features and the acoustic input features to be processed are input into the prefix conditional renderer. The Gaussian noise sequence is gradually denoised using the reference acoustic features and semantic input features as conditions to obtain the target acoustic features. The prefix conditional renderer is a conditional diffusion model, and its training process includes: The acoustic input features of the speech samples are randomly split into a prefix part and a target part along the time axis; During the forward diffusion process, the target portion is gradually noise-added to Gaussian noise; In the reverse denoising process, the target part is gradually restored from Gaussian noise by using the prefix part and semantic input features as conditions.

2. The controllable speech synthesis method based on thought chain style reasoning according to claim 1, characterized in that, The context description includes the dialogue scenario, the speaker's identity, and the speaker's emotional state.

3. The controllable speech synthesis method based on thought chain style reasoning according to claim 1, characterized in that, The speech style attributes include speech rate, volume, pitch, and emotion.

4. The controllable speech synthesis method based on thought chain style reasoning according to claim 1, characterized in that, The multimodal large model, guided by preset prompts, performs the following operations in sequence: Transcribe speech samples into target text; Context description based on speech samples and target text; The reasoning process is based on the target text and context description. First, intermediate reasoning process text is generated, and then structured style description text is generated based on the intermediate reasoning process text.

5. The controllable speech synthesis method based on thought chain style reasoning according to claim 1, characterized in that, The loss function for training the large language model includes thought chain loss and semantic sequence loss; the thought chain loss is the cross-entropy loss between the predicted thought chain sequence and the actual thought chain sequence; the semantic sequence loss is the cross-entropy loss between the predicted target semantic sequence and the actual target semantic sequence.

6. A controllable speech synthesis system based on thought chain style reasoning, used to implement the controllable speech synthesis method based on thought chain style reasoning as described in claim 1, characterized in that the system... include: The quadruple data construction module is used to acquire speech samples. The multimodal large model generates target text, context description and thought chain sequence based on the speech samples. The pre-trained semantic encoder extracts the target semantic sequence from the speech samples to form quadruple training data. The thought chain sequence includes intermediate reasoning process text and style description text used to describe speech style attributes. The training module for the enhanced semantic language model of the thinking chain is used to take the target text and context description in the training data of the four-tuple as input, the thinking chain sequence as the intermediate supervision signal and the target semantic sequence as the final supervision signal, and train the large language model by maximizing the conditional probability to obtain the enhanced semantic language model of the thinking chain. The semantic reasoning module is used to input the target text to be synthesized and its corresponding context description into the mind chain enhanced semantic language model, and generate the mind chain sequence and the target semantic sequence to be synthesized in sequence. The acoustic rendering module is used to generate target acoustic features based on the target semantic sequence to be synthesized, the reference semantic features of the reference speech, and the reference acoustic features; The speech decoding module is used to convert the target acoustic features into synthesized speech.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the controllable speech synthesis method based on thought chain style reasoning as described in any one of claims 1 to 5.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement, when executing the computer program, the controllable speech synthesis method based on thought chain style reasoning as described in any one of claims 1 to 5.

Images