A method for generating training data for a contextualized speech synthesis

Abstract

This invention relates to a method for generating scenario-based speech synthesis training data, comprising: parsing speech data to obtain its acoustic feature representation; generating a scenario description text describing the background of speech data generation based on the acoustic feature representation and the corresponding transcribed text, using the reasoning ability of a language model; generating a causal reasoning text explaining the logical relationship between the acoustic feature representation, the scenario description text, and the transcribed text, using the reasoning ability of the language model, based on the scenario description text, the transcribed text, and the acoustic feature representation; and associating the scenario description text, the transcribed text, the causal reasoning text, and the speech data to form training data for training a speech synthesis model. Through the above design, this invention achieves a paradigm shift from "attribute-driven" to "scenario-driven," ultimately significantly improving the interactive intuitiveness, ease of use, and intelligence level of the speech synthesis system.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a method for generating scenario-based speech synthesis training data. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, text-to-speech (TTS) technology is no longer satisfied with generating clear and natural speech, but is moving towards rich expressiveness and controllability. Against this backdrop, mainstream audio data preprocessing pipelines primarily focus on automated data acquisition and cleaning, aiming to efficiently construct large-scale text-to-speech pairs to provide training data for prompt-based generative speech synthesis models.

[0003] However, with the rise of instruction-based speech synthesis technology, the focus of data processing has shifted from simple text transcription to deep acoustic feature mining. These approaches aim to extract audio attributes from multiple dimensions, including speaker emotion, pitch, volume, and speech rate, and transform these unstructured features into explicit text instructions or semantic embeddings. In this way, the model can accurately receive and execute complex acoustic instructions, achieving controllable and expressive speech synthesis. The open-source Speechcraft is an example. [1] The framework defines a standardized paradigm for transforming raw acoustic properties into natural language descriptions, becoming a representative framework for imperative speech synthesis data preparation. Speechcraft employs a three-stage pipeline architecture, addressing the key pain point of lacking natural language descriptions in imperative training. The system first invokes multiple specialized expert models to extract discrete labels from the raw audio, including dimensions such as gender, age, emotion, pitch, volume, and speech rate. Furthermore, Speechcraft can capture word-level stress. The core of this framework is description generation based on a large language model. It utilizes a fine-tuned large language model to reorganize and expand discrete labels (such as male, fast speech rate, angry, high pitch, high volume) into natural and diverse descriptive text.

[0004] While existing technologies like Speechcraft can generate descriptive text for command-driven communication, they still have significant limitations. First, command-based speech synthesis models trained using data from the current pipeline require specific audio attributes as input (such as pitch and speech rate), an interaction method that doesn't align with typical user habits. In real-world applications, users tend to provide abstract descriptions of the speech context (e.g., "speaking in an anxious situation") rather than specifying explicit, technical audio attribute parameters. Second, while the discrete label data generated by existing technologies facilitates model learning for stable control, a semantic gap exists between this data and the abstract descriptions users are accustomed to using. Existing data cannot bridge this gap, preventing the model from directly understanding and executing the user's abstract intentions, thus limiting the usability and intelligence of speech synthesis systems in real-world scenarios.

[0005] It is evident that while existing technologies have made progress in acoustic feature description, there is a fundamental mismatch between the data paradigm they generate and natural human interaction methods. This constitutes the main technical bottleneck for achieving intelligent, contextualized speech synthesis. New data construction methods are urgently needed in this field to overcome this bottleneck.

[0006] It should be noted that the information disclosed in the background section above is only for understanding the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The technical problem this invention aims to solve is that "existing instruction-driven speech synthesis technology suffers from limitations in training data paradigms, resulting in counterintuitive model interaction methods that fail to understand and respond to high-level, human-friendly contextual intentions, thereby limiting the ease of use and intelligence level of speech synthesis systems in real-world applications."

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows.

[0009] This invention provides a method for generating scenario-based speech synthesis training data, comprising the following steps: The speech data is parsed to obtain its acoustic feature representation, which includes prosodic features extracted from the speech data. Based on acoustic feature representation and transcribed text corresponding to speech data, the reasoning ability of language models is used to generate scene description text describing the background of speech data generation. Based on scene description text, transcribed text, and acoustic feature representation, the reasoning ability of the language model is used to generate causal reasoning text that explains the logical relationship between acoustic feature representation and scene description text and transcribed text. Scene description text, transcribed text, causal reasoning text, and speech data are associated to form training data for training speech synthesis models.

[0010] In some embodiments, prosodic features include: macro-prosodic features for describing the global auditory properties of the entire speech segment; and word-level stress features for identifying one or more words in the speech that need to be stressed.

[0011] In some embodiments, macroprosodic features include at least one of pitch, volume, and speech rate.

[0012] In some embodiments, the extraction of macro-prosodic features includes quantizing at least one feature among pitch, volume, and speech rate into discrete labels.

[0013] In some embodiments, acoustic feature representation also includes emotional attribute features extracted from speech data.

[0014] In some embodiments, the acoustic feature representation further includes paralinguistic information extracted from the speech data, which includes at least one of laughter, sighs, coughs, and cries.

[0015] In some embodiments, the step of extracting paralinguistic information includes: detecting non-verbal sound events in speech data and obtaining their timestamp information; performing temporal semantic alignment between the tags corresponding to the non-verbal sound events and the transcribed text of the speech data to generate enhanced transcribed text; and applying the enhanced transcribed text to the generation of scene description text and causal reasoning text.

[0016] In some embodiments, the language model used to generate scene description text and / or causal reasoning text is a Large Language Model (LLM).

[0017] In some embodiments, a robustness enhancement step is further included, which includes at least one of the following: cross-validating the extracted acoustic feature representations; and applying confidence threshold filtering to the extracted acoustic feature representations.

[0018] In some embodiments, when the acoustic features are represented as emotional attribute features, the robustness enhancement step includes: extracting emotional attribute features using a first emotion recognition model and obtaining a first confidence level; if the first confidence level is lower than a first threshold, marking the emotional attribute as uncertain; if the first confidence level is higher than the first threshold, performing cross-validation using a second emotion recognition model, and determining the final emotional attribute based on the output results of the first emotion recognition model and the second emotion recognition model.

[0019] In some embodiments, when the acoustic features are characterized as word-level stress features, the robustness enhancement step includes: obtaining a first stress word list using a first stress detection tool; obtaining a second stress word list using a second stress detection tool; comparing the first stress word list with the second stress word list, and taking the overlapping words that exist in both lists as the finally confirmed stress words.

[0020] In some embodiments, the step of generating causal reasoning text employs a templated cue word generation method, which constrains the generated content of the language model through predefined structured cue word templates.

[0021] In some embodiments, the step of generating scene description text is constrained by a scene-acoustic mapping knowledge base, which defines the mapping relationship between scene types and the acceptable range of acoustic features.

[0022] In some embodiments, a system for generating contextualized speech synthesis training data is also provided, including: The parsing module is configured to parse speech data to obtain its acoustic feature representation. The scene generation module is configured to generate scene description text that describes the background of speech data generation based on acoustic feature representation and transcribed text corresponding to speech data, and utilizes the reasoning ability of the language model. The reasoning generation module is configured to generate causal reasoning text that explains the logical relationship between the acoustic feature representation and the scene description text and the transcribed text, based on the scene description text, transcribed text and acoustic feature representation, using the reasoning ability of the language model. The data association module is configured to associate scene description text, transcribed text, causal reasoning text, and speech data to form training data for training speech synthesis models.

[0023] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the generation method of the present invention.

[0024] In some embodiments, an electronic device is also provided, including: a processor; and a memory for storing a computer program executable by the processor; wherein, when the computer program is executed by the processor, it implements the generation method of the present invention.

[0025] In some embodiments, a contextualized speech synthesis training dataset generated by the generation method of the present invention is also provided. The dataset consists of multiple training samples, each training sample containing the following associated elements: speech data segments; transcribed text corresponding to the speech data segments; contextual description text describing the background of the speech data segments; and causal inference text explaining the logical relationship between the acoustic feature representation of the speech data segments and the contextual description text and transcribed text.

[0026] In some embodiments, a contextualized speech synthesis training data generation system is also provided, configured to generate the contextualized speech synthesis training dataset of the present invention, the contextualized speech synthesis training data generation system comprising: The acoustic feature analysis module is used to analyze the input speech data to obtain its acoustic feature representation. The scene text generation module, connected to the acoustic feature parsing module, is used to generate scene description text based on acoustic feature representation and corresponding transcribed text; The reasoning text generation module, connected to the scene text generation module and the acoustic feature parsing module, is used to generate causal reasoning text based on scene description text, transcribed text and acoustic feature representation; The data synthesis and output module is connected to the scene text generation module and the reasoning text generation module. It is used to associate and output scene description text, transcribed text, causal reasoning text and speech data.

[0027] In some embodiments, a method for training a speech synthesis model is also provided, comprising: obtaining a scenario-based speech synthesis training dataset of the present invention; and training a speech synthesis model using the scenario-based speech synthesis training dataset, so that the trained speech synthesis model learns to generate synthesized speech that conforms to the context described by the scenario description text based on the input scenario description text and the text to be synthesized.

[0028] In some embodiments, a speech synthesis method is also provided, comprising: receiving input scene description text and text to be synthesized; inputting the input scene description text and text to be synthesized into a speech synthesis model; and obtaining synthesized speech output by the speech synthesis model; wherein the speech synthesis model is trained using the training method of the present invention.

[0029] In some embodiments, a speech synthesis apparatus is also provided, comprising: a processor; and a memory coupled to the processor and used to store a computer program; wherein, when executed by the processor, the computer program causes the apparatus to implement the speech synthesis method of the present invention.

[0030] The present invention has the following beneficial effects: This invention transforms unstructured raw speech signals into structured, machine-readable feature data by "analyzing speech data to obtain its acoustic feature representation," providing a reliable data foundation for subsequent intelligent reasoning and ensuring that the entire method is built on objective acoustic analysis. By "generating scene description text based on acoustic feature representation and transcribed text, and utilizing the reasoning ability of language models," it achieves reverse reasoning and construction from low-level acoustic cues to high-level scene semantics. This allows the generated training data to contain rich contextual information, transforming acoustic features from "targets" to be controlled into "clues" for constructing scenes. Third, by "generating causal reasoning text based on scene description text, transcribed text, and acoustic feature representations, utilizing the reasoning ability of the language model," an interpretable logical chain is provided for the mapping relationship between "scene" and "speech," increasing the depth of the data. More importantly, it provides the speech synthesis model with the inherent logic of "why it needs to be expressed in this way in this scene," enhancing the model's understanding and generalization ability. Finally, by "associating scene description text, transcribed text, causal reasoning text, and speech data to form quadruple training data," a novel training data product rich in semantic logic is produced. This data paradigm enables the finally trained speech synthesis model to directly understand and respond to high-level contextual intentions.

[0031] In summary, this invention generates a novel quadruple training data by organically combining the aforementioned technical features. This data can guide the speech synthesis model to learn to autonomously infer appropriate acoustic representations from abstract scene descriptions, thereby realizing a paradigm shift from "attribute-driven" to "scene-driven" approaches. Ultimately, this significantly improves the interactive intuitiveness, ease of use, and intelligence level of the speech synthesis system.

[0032] Preferably, this invention further defines "prosodic features" specifically as "macro-prosodic features" and "word-level stress features," making the extraction of acoustic features more hierarchical and targeted. Macro-prosodic features grasp the overall style of speech, while word-level stress features capture the key emphasis points in expression. The combination of the two provides richer acoustic clues for generating more accurate and detailed scene and reasoning texts.

[0033] Preferably, the present invention further defines "macro prosodic features" as "at least one of pitch, volume, and speech rate". These features are the most intuitive dimensions for describing the overall expressiveness of speech, making the implementation of the technical solution more specific and clear, and helping to ensure the stability and consistency of the generated data.

[0034] Preferably, the present invention further reduces the complexity of the data by "quantifying" continuous features such as pitch, volume, and speech rate into discrete labels, making the data easier for language models to understand and process. This also facilitates the learning and control of subsequent speech synthesis models, thereby improving the efficiency and reliability of the entire technical process.

[0035] Preferably, the present invention further expands the semantic connotation of acoustic feature representation by adding "emotional attribute features". Emotion is an important element of scene composition. The introduction of this feature makes the generated scene description text and causal reasoning text more accurate and richer in terms of emotion, thereby improving the quality and expressiveness of the final training data.

[0036] Preferably, this invention further enriches the dimensions of acoustic feature representation by introducing the extraction and processing of "paralinguistic information." First, through a temporal semantic alignment step, this invention precisely maps non-linguistic events to text content on a timeline, generating enhanced transcribed text. Then, this enhanced text is applied to the generation of scene and reasoning text. This series of processes enables the large language model to directly perceive "where" and "what" non-linguistic events occurred, thereby generating high-level scenes and causal inferences that are logically consistent with and accurately describe such events (e.g., explaining why laughter occurs after specific words or phrases). This not only significantly improves the realism, expressiveness, and contextual richness of the training data but also greatly enhances the correlation and interpretability between the text generated by the large language model and the original speech data.

[0037] Preferably, the present invention further utilizes the powerful semantic understanding and logical reasoning capabilities of the Large Language Model (LLM) to generate text, thereby ensuring that the generated scene description text and causal reasoning text have high rationality, fluency and diversity.

[0038] Preferably, this invention further verifies the reliability of acoustic feature extraction by introducing a "robustness enhancement step," including "cross-validation" and "confidence threshold filtering." This effectively filters out low-quality or unreliable feature data, improving the input quality of subsequent text generation stages from the source, thereby ensuring the overall accuracy and reliability of the final quadruple training data. Specifically, this invention designs specific hybrid validation strategies for sentiment attributes and word-level stress features. For sentiment attributes, a combination of "confidence threshold filtering" and "cross-validation" effectively identifies and handles uncertainties with low confidence and high conflict, avoiding the transmission of incorrect sentiment labels to subsequent stages and ensuring the logical rationality of the generated data from the source. For word-level stress, cross-validation is performed using two tools with different technical principles, and the intersection is taken, significantly improving the accuracy of stressed word location, avoiding the generation of absurd causal reasoning due to stress misjudgment, and ensuring the high reliability of the finally confirmed stressed words. These specific measures work together to systematically improve the robustness of the entire data generation process and the quality of the output data.

[0039] Preferably, the present invention also uses a "template-based prompt word generation method" to constrain the generation of causal reasoning text, which can effectively guide the output direction of the language model, reduce the randomness and irrationality of the generated content, and ensure that the generated causal reasoning text maintains a high degree of consistency in format and basic logic, which is conducive to the stable learning of the speech synthesis model.

[0040] Preferably, this invention further constrains scene generation by introducing a "scene-acoustic mapping knowledge base," injecting prior human knowledge and domain rules into the language model generation process. This effectively avoids generating illogical or overly imaginative scenes, ensuring a reasonable and expected correlation between the generated scene description and acoustic features, thus enhancing the controllability of the technical solution and the practicality of the generated results.

[0041] Other beneficial effects of the present invention will be further described below. Attached Figure Description

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a method for generating training data for contextualized speech synthesis; Figure 2 This is an architecture diagram of a speech synthesis model that is fine-tuned based on quadruples data; Figure 3 This is a schematic diagram of sequence modification for a speech synthesis model. Detailed Implementation

[0043] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0044] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0045] This invention recognizes that the technical deficiency of existing "command-driven speech synthesis models with unintuitive interaction" stems from the limitations of the inherent logical paradigm of the training data: the data only establishes a shallow mapping from low-level acoustic features to their text labels. This paradigm sets acoustic features as "control targets" that require precise user specification, leading to a semantic gap between user intent (high-level scene description) and model input (low-level acoustic attributes). To address this, this invention constructs a novel training data and method that transforms acoustic features from "control targets" into "clues for constructing contextualized speech expressions." By introducing high-level scene description text and causal inference text, a deep "scene-speech" mapping is established. The resulting speech synthesis model learns to autonomously infer appropriate acoustic expressions from high-level scenes, thus eliminating the need for users to painstakingly devise specific acoustic commands. Users can obtain the expected speech output simply by providing a natural scene description, fundamentally solving the problem of unintuitive interaction.

[0046] In some embodiments, this invention proposes a method for generating scenario-based speech synthesis training data. Through multi-level acoustic feature extraction and contextual reasoning of a language model, traditional [text, speech] pairs are converted into [scene, text, reasoning, speech] quadruples. This automatically generates speech synthesis training data containing causal reasoning with high-level scene descriptions, enabling the speech synthesis model (TTS) to autonomously infer appropriate speech representations based on scene context, without relying on low-level acoustic commands.

[0047] In some embodiments, the method for generating contextualized speech synthesis training data includes the following steps: Multi-level acoustic feature extraction: First, multi-dimensional acoustic attribute analysis is performed on the speech data. Specifically, Speechcraft can be used to extract macroscopic prosodic features of the speech, which are used to describe the global auditory attributes of the entire speech segment. Global auditory attributes refer to the overall, general auditory impression of a speech segment, which differs from the local perception of a specific segment or phoneme. These attributes typically include, but are not limited to: pitch (high or low); volume (loud or loud); speaking rate (fast or slow speech per unit time); and the overall timbre or style perception formed by the combination of these basic features. Extracting these features aims to grasp the overall expressive style and emotional tone of the speech; the Kimi-Audio model is used to identify the emotional attributes of the speech; for stress information, the Wavelet Prosody Toolkit is used for automatic annotation; and pYin is also used... [3] The pitch trajectory estimation is compared with the phoneme-level alignment results of MFA (Montreal Forced Aligner, a tool for speech-to-text alignment) to extract key points of pitch change. Through the above method, a multidimensional audio attribute representation of speech is obtained.

[0048] In some embodiments, to enhance the richness of multi-level acoustic feature representation, multi-level acoustic feature extraction also includes the extraction of paralinguistic information. Paralinguistic information refers to the sound features and phenomena in human speech other than core vocabulary and grammar, such as laughter, sighs, coughs, and cries. It does not focus on "what to say" but on "how to say it," serving as a sound carrier of emotion, intention, and context. Its role is to provide extremely subtle yet powerful contextual cues for the generation of scene descriptions and causal inferences, and to serve as the learning target of the speech synthesis model, enabling it to ultimately generate highly natural and expressive speech containing these rich sound behaviors, thereby achieving a leap from "mechanical reading" to "vivid storytelling."

[0049] In some embodiments, the process of extracting paralinguistic information and aligning temporal semantics is as follows: A frame-level detection model pre-trained based on frameworks such as Wav2vec-Bert can be used to detect paralinguistic information (such as laughter, coughing, and sighing) in the input speech, outputting start and end timestamps for specific non-linguistic sound categories. To address potential alignment errors between the detected timestamps and the speech-transcribed text, a temporal semantic alignment method can be employed. 1. Audio Segmentation and Segment Recognition: Based on the detected paralinguistic information boundaries, the original audio is segmented into multiple segments (e.g., segments containing paralinguistic information and segments not containing it). Each segmented audio segment undergoes independent speech recognition (ASR) to obtain the corresponding short text. The paralinguistic information tag (e.g., [laughter]) is then concatenated with the recognized text of the corresponding short segment.

[0050] 2. Global Semantic Correction: Since segmentation recognition may lead to lexical errors or loss at segmentation points, the concatenated complete text is compared with the global ASR transcribed text of the original audio. Algorithms such as edit distance are used to calculate the differences between the two, thereby accurately correcting the paralinguistic information tags and inserting them into the correct positions in the global transcribed text.

[0051] Through the above steps, a text that retains the semantic coherence of the globally transcribed text while containing precise paralinguistic information annotations is finally obtained. This text can be used, along with other acoustic features, as input for subsequent large language models to generate scene descriptions and causal inference texts.

[0052] Scene reconstruction and inference chain generation based on Large Language Model (LLM): Leveraging the powerful semantic reasoning capabilities of LLM, and based on extracted multi-dimensional acoustic features and transcribed text enhanced with paralinguistic information, a high-level scene description text is generated that makes the spoken expression seem reasonable. After obtaining the scene, the [scene, text, audio attributes] are input into the LLM, allowing the model to further construct a causal chain inference from [scene, text] to [audio attributes], outputting interpretable text, and finally forming a four-tuple structured training data of [scene, text, inference, speech].

[0053] An end-to-end data generation pipeline enables fully automated conversion from raw speech to semantically rich inference data, eliminating the need for manual annotation. This ensures both scalability of the data and effectively improves data consistency and semantic richness. In some embodiments, the data processing pipeline of this invention is applied as follows: Figure 1 As shown, by simply inputting the speech and its corresponding transcribed text, the corresponding [scene, reasoning] text can be generated. For the text "This is truly incredible, I simply cannot understand how they managed to do such a massive project without any errors, not even the slightest problem.", the pipeline first extracts their audio attributes "speech rate: medium, volume: high, pitch: high, emotion: excited" and the stressed words "incredible" and "even the slightest problem". If there is paralinguistic information in the speech, such as laughter, a [laughter] tag is inserted at the corresponding position in the transcribed text (e.g., after "incredible"). Subsequently, LLM is used to synthesize the scene, and after obtaining the scene, LLM is used to generate the reasoning text.

[0054] In some embodiments, multi-level acoustic feature extraction uses a macro-prosodic extractor and signal processing methods to extract the overall pitch and energy values ​​from the speech data. These values ​​are then categorized into high, medium, and low levels based on their distribution range to obtain corresponding discrete labels. For speech rate labels, a scalar representing speech rate is calculated using words per second (SPS) and categorized into fast, medium, and slow levels based on their distribution range. For speaker emotion, Kimi-Audio is used. [2] The model outputs emotional content through a question-and-answer format.

[0055] In some embodiments, pitch variation extraction utilizes pYin [3] The fundamental frequency (F0) of speech is estimated over time, and peaks and troughs in the curve are identified, which typically correspond to points of emotional emphasis. Then, MFA is used for word-level timestamp alignment to accurately locate words that produce pitch fluctuations.

[0056] In some embodiments, the accent detection process uses the open-source tool wavelet prosody toolkit. [4] The input is speech and timestamped text. Wavelet transform is used to calculate the salience score of each word, and words with higher salience scores are designated as stressed words.

[0057] In some embodiments, after obtaining acoustic features and transcribed text enhanced with paralinguistic information, LLM is used to construct a high-level contextual description of the speech and a matching causal logical explanation.

[0058] In some embodiments, the scene description text generation process is as follows: Acoustic features are transformed into a scene description using an LLM. (This is followed by a reference to Qwen2.5-32B-Instruct.) [5] or Mistral-Small-3.2-24B-Instruct [6] The input prompts (e.g., "Please conceive a reasonable scene based on the following text and sound features, text: {enhanced transcribed text}, sound features: {audio features}") and the enhanced transcribed text and acoustic features of the speech data are used. To ensure the richness of the generated content and reduce mechanical repetition, the temperature parameter is adjusted to 0.7 during the generation process. LLM integrates these discrete acoustic features and transcribed text into a logically consistent narrative, thereby generating a background story and event environment description that makes the acoustic features seem reasonable. This step supplements the original speech data with scene information.

[0059] In some embodiments, the causal inference text generation process is as follows: The scene, all acoustic features (such as pitch: high, volume: loud, speech rate: fast, emotion: happy, stressed words: hurry), and transcribed text are concatenated into a natural language string according to a preset template format and used as input for the LLM. The LLM then explains why specific words need to be stressed, or why the speech rate suddenly increases in this context. The output of the LLM is a description of the speech attributes containing causal logic. Finally, a complete dataset of the [scene, text, inference, speech] quadruple is constructed.

[0060] In some embodiments, such as Figure 2 , 3 As shown, when using the quadruplet dataset in practice, a pre-trained audio-based large speech model (similar to LLM, but supporting speech input and output, capable of dialogue) is used as a base. The generated quadruplet dataset is then used to fine-tune this model, achieving a speech synthesis system that takes a scene as input and outputs speech consistent with that scene. Specifically, the Kimi-Audio model can be used, and its overall architecture is as follows... Figure 2 This model is capable of generating text and audio, and this feature can be used to adapt it to tasks that output thought chains.

[0061] By transforming the model processing sequence as follows Figure 3 As shown, the model can be fine-tuned using quadruples of data. The input text contains [scene, text], and the target text contains [inference] and [enhanced text] (i.e., text containing paralinguistic labels). Through training, the model learns this mapping relationship, thereby generating context-appropriate synthesized audio [speech] based on [scene, text].

[0062] It should be noted that when implementing the above scheme, considering the reliance on the reverse reasoning path of "inferring scene and causal chain from acoustic features," and the fact that the correspondence between acoustic features and scenes is not stable and unique, directly using free generation based on Large Language Model (LLM) may result in the generated scene description text being disconnected from the real causal relationship, which may lead to problems related to the accuracy and robustness of subsequent model training. To further improve the reliability and consistency of the generated results, this invention also provides the following optimization strategies.

[0063] In some embodiments, acoustic features are defined as "contextual cues" for constructing reasonable speech expressions. Specifically, when generating scene description text, the goal is not to achieve an exact reproduction of the actual recording scene, but to guide the LLM to construct a logically consistent contextual environment, so that the acoustic features and paralinguistic information of the input speech have reasonable interpretability in that context. For example, for speech that is fast-paced, high-pitched, and accompanied by laughter, the LLM may generate various reasonable scenarios such as "the commentator cheering for a great goal in an exciting match" or "a cheerful discussion among friends watching a comedy movie." For speech that is fast-paced, high-pitched, but without laughter, the LLM may generate scenarios such as "an emergency announcement" or "a heated argument on the field." Although these scenarios are different, they all provide logical support for specific acoustic features, thereby establishing an effective "scene-speech" mapping relationship.

[0064] When generating causal reasoning text, the goal is to construct interpretable connections between the generated high-level scene and the observed acoustic features and paralinguistic information. The focus is not on reproducing true physical causality, but on forming logically sound deductive chains to enhance the interpretability and guidance of the data for model training. For example, LLM can infer, based on a generated scene, why certain acoustic behaviors such as pitch increase, speech rate increase, emphasis on specific words, or laughter occur in a particular context. This type of reasoning text provides the speech synthesis model with learnable expressive logic, improving the model's generalization and inference capabilities in unseen scenarios.

[0065] Through the above design, this invention indirectly solves the technical problem of the complex and variable correspondence between acoustic features and single real-world scenes, which is difficult to maintain stably. It constructs a "scene-speech" mapping relationship that possesses logical rationality, structural stability, and scalability. This not only transforms acoustic features from uncontrollable "target parameters" into "reasoning cues" for semantic generation, but also transforms paralinguistic information from uncontrollable noise signals into interpretable and learnable semantic labels. Furthermore, it lays a high-quality data foundation for training speech synthesis models with scene understanding and autonomous reasoning capabilities.

[0066] In some embodiments, the present invention also introduces a "scene-acoustic mapping knowledge base" as a constraint to guide and standardize the LLM generation process. Specifically, the scene-acoustic mapping knowledge base can be a structured database whose content defines the mapping relationship between common high-level scene types and the acceptable range of acoustic features for each scene. It should be noted that the scene-acoustic mapping knowledge base can be constructed in one or more of the following ways: 1) manually defined based on the experience of experts in fields such as phonetics and broadcasting; 2) summarized after statistical analysis of a large amount of labeled multimedia data such as films, TV series, and audiobooks; 3) dynamically expanded from high-quality human feedback through an online learning mechanism, without specific limitations here.

[0067] Scene types: such as "emergency news broadcast", "heartwarming storytelling", "exciting sports commentary", "calm weather forecast", etc.

[0068] Acoustic feature range: For each scene type, the knowledge base defines its legal acoustic feature values ​​or combinations, for example: For "emergency news broadcast" scenarios, the legitimate range of acoustic characteristics may include: speech rate: fast or medium, pitch: high or medium, volume: loud or medium, and emotion: tense or serious. For the "heartwarming storytelling" scenario, the legal range may be: speaking speed: slow or medium, pitch: low or medium, volume: low or medium, and emotion: gentle or pleasant.

[0069] When using LLM to generate scene description text and causal reasoning text, this method integrates the constraints of the knowledge base through the following steps: 1. Acoustic Feature Matching: After extracting the acoustic feature set of the input speech, the set is first matched with the acoustic feature range in the scene-acoustic mapping knowledge base to filter out all candidate scene types whose acoustic features fall within its legal range.

[0070] 2. Cue Word Integration Constraints: Subsequently, the selected candidate scene list is used as context information and input into the LLM through a specific cue word template. For example, the cue word can be designed as: "Based on the provided text and sound features, please select the most suitable scene from the following candidate scene list {candidate scene list} and generate a detailed scene description. Text: {transcribed text}, sound features: {audio features}." 3. LLM generation within constraints: LLM generation based on prompt words will be restricted to a reasonable range defined by the scene-acoustic mapping knowledge base, thereby effectively reducing the possibility of generating unreasonable or completely fictitious scenes and ensuring that there is a preset and reasonable association between the output scene and the acoustic features.

[0071] Preferably, the present invention designs the scene-acoustic mapping knowledge base as a structure containing a general layer and a domain adaptation layer. The general layer contains widely applicable basic scene-acoustic mappings. The domain adaptation layer allows for the injection of domain-specific scene and acoustic rules for specific application domains (such as in-vehicle voice systems and audiobooks), and can even dynamically expand its coverage by expanding the knowledge base from high-quality human feedback through online learning mechanisms.

[0072] Specifically, the basic scene-acoustic mapping in the "general layer" aims to cover the most common and universal expression patterns in human speech communication. Its definition is not dependent on a specific task or domain, but is based on general consensus in phonetics and sociolinguistics. It may include the following: 1. Scene Dimension (Example): Announcements and announcements: such as news broadcasts, announcements, weather forecasts, etc. Their core contextual characteristics are the objectivity and clarity of information delivery.

[0073] Narrative storytelling: such as recounting events, introducing products, etc. The core contextual feature is the linear unfolding of time or logic.

[0074] Interactive communication types: such as dialogues, Q&A, customer service, etc. The core contextual features are turn-taking and immediate feedback.

[0075] Emotional expression: such as excited cheers, sorrowful confessions, gentle comfort, etc. The core contextual feature is the direct expression of emotional state.

[0076] 2. Acoustic Feature Mapping Rules (Example): For each of the above basic scenario types, define its typical and widely accepted acoustic characteristic range: (1) For "announcement / command" scenarios: Speech rate: moderate, steady, avoiding speaking too fast or too slow to ensure clarity.

[0077] Pitch: Medium, with minimal fluctuation, conveying solemnity and stability.

[0078] Volume: Medium to high, to ensure full and penetrating sound.

[0079] Emotion: Neutral, serious, solemn.

[0080] Paralinguistic information: Usually contains very little or no strong emotional paralinguistic information such as laughter or crying; there may be occasional, slight throat clearing and other sounds used to adjust one's state.

[0081] (2) For "emotional expression (excitement)" scenarios: Speech rate: fast or drastically changing, which may be accompanied by pauses for emphasis.

[0082] Pitch: Overall high, with a wide range of fluctuations.

[0083] Volume: Loud and impactful.

[0084] Emotions: Excitement, elation, joy.

[0085] Paralinguistic information: It is very likely to include positive, high-arousal paralinguistic information such as laughter, cheering, and amazement.

[0086] (3) For "emotional expression (gentle)" scenarios: Speech rate: slow and gentle.

[0087] Pitch: Overall low, with gentle fluctuations.

[0088] Volume: low or medium, soft tone.

[0089] Emotions: Gentle, calm, and kind.

[0090] Paralinguistic information may include soft laughter, soothing sighs, and soft responses of agreement (such as "uh-huh").

[0091] The role of the general layer is to establish a reasonable initial set of constraints to ensure that the base scenario generated by LLM does not deviate too far from human consensus (for example, it will not map "a passionate speech" to an extremely uncommon combination of "slow speech rate and low pitch").

[0092] Specifically, the domain adaptation layer, built upon the general layer, is a refined rule base customized for specific vertical domains. It makes the generated data more closely aligned with real-world application needs. The following examples illustrate some specific domains and their unique scenarios: 1. In-vehicle voice control: Specific scenarios include: "Navigation guidance", "Hazardous road condition warning", "In-vehicle equipment control confirmation", and "Entertainment content playback".

[0093] Examples of specific acoustic rules: For "Hazardous Road Condition Warnings": the acoustic signature must include high volume, rapid speech, and high pitch to maximize driver attention. A serious, urgent emotional tone may be mandatory. Paralinguistic information: should not include any distracting sounds such as laughter or yawning that conflict with the emergency atmosphere.

[0094] For "entertainment content playback": the acoustic characteristics tend towards a relaxed and pleasant mood, with more expressive variations in speech rate and pitch. Paralinguistic information: may include lighthearted laughter or sighs of enjoyment to enhance the entertainment atmosphere.

[0095] 2. Audiobooks: Unique scenes: "narration" and "dialogue between different characters" (which can be further subdivided into "dialogue between the elderly", "dialogue between children", "dialogue between villains", etc.).

[0096] Examples of specific acoustic rules: For dialogue involving elderly characters: acoustic features may default to a slower speech rate, lower pitch, and a slightly breathy quality. Paralinguistic information may include mild coughing, wheezing, or a kind chuckle.

[0097] Regarding "narration": the rules may require consistency with the emotional tone of the current storyline (e.g., a mysterious, somber tone corresponds to a suspenseful plot). Paralinguistic information: in horror or suspenseful plots, the narrator may include tense gasps or meaningful pauses (which can be considered a form of paralinguistic behavior); in lighthearted plots, the narrator may include cheerful laughter.

[0098] Specifically, the online learning mechanism can be implemented through the following technical means: 1. Human Feedback Loop: A feedback interface can be provided, allowing domain experts to rate or correct the scenario-voice pairings generated by the LLM. For example, an expert can mark a scenario generated by a "navigation guide" voice as "not clear enough," or directly correct it to "navigating on a complex overpass." This high-quality correction data will be automatically added to the knowledge base of the domain adaptation layer.

[0099] 2. Data-driven updates: When a TTS model trained using data generated by this method performs well in a specific domain application, the scene-acoustic mapping in that batch of data can be considered effective. The system can periodically collect these "verified effective" data patterns and, through statistical analysis, extract new mapping rules to enrich the knowledge base.

[0100] Preferably, this invention does not force the LLM to select from the candidate scene list, but rather employs a "soft constraint." The prompts encourage the LLM to prioritize scenes in the candidate scene list, while simultaneously allowing it to propose new scenes, provided the LLM can provide a high-confidence explanation for the match between the new scene and the acoustic features. A confidence threshold can be set to retain only high-confidence new scene mappings, and these can be selectively added to the scene-acoustic mapping knowledge base, enabling the knowledge base to be self-updating.

[0101] Specifically, the "confidence threshold" can be determined based on one or more of the following criteria: Model intrinsic confidence: Based on the probability scores output by the LLM itself for the generated new scene and its explanatory text, the threshold is set to an empirical value (such as 0.7 or greater than 0.7, the specific value can be flexibly set according to the implementer's needs).

[0102] External consistency verification: Evaluation is performed through automated verification mechanisms. For example, the new scenario is used to drive the speech synthesis model to generate speech, and the acoustic feature similarity between the speech and the original speech is calculated. The similarity is used as part of the confidence score.

[0103] Through the above design, this invention improves the controllability and reliability of generated data, suppresses the "illusion" of LLM, ensures that the generated data is within the preset reasonable boundaries, and enhances the technical robustness of the solution; it injects human prior knowledge into the generation process, so that data generation is no longer a completely black box process, and improves the predictability of the results; through hierarchical design and soft constraint mechanism, it overcomes the limitations of static knowledge bases to a certain extent, making them scalable and adaptable.

[0104] It should be noted that, in one implementation, the generation of causal reasoning text relies on LLM-based free generation of reasoning chains. While this ensures the diversity and richness of the content, it may also introduce "noise knowledge" that does not conform to true causality. Therefore, this invention preferably employs a templated prompt word generation method, as follows: 1. Template definition and variable filling mechanism: This is achieved by predefining one or more highly structured prompt word templates. Each template consists of a fixed phrase and variable slots to be filled. For example, a basic template could be defined as: "In the context of the scene description, in order to express the emotion of the emotional attribute, the speaker adopted the speed of speech and the tone of pitch, added paralinguistic information at the end of the sentence, and emphasized the meaning or function of the key words by stressing the stressed words." The content within “<>” represents variable slots. During execution, variables such as <scene description>, <emotional attributes>, <speech rate>, <pitch>, <paralinguistic information>, and <stress words> generated in the upstream steps are automatically filled into the corresponding slots. Slots requiring semantic generation, such as <keyword meaning or function>, are filled to a limited extent by the LLM based on the current context, thus constraining the LLM's creative freedom within a specific range.

[0105] 2. Multi-template strategy and constraint optimization: To avoid the text becoming stiff due to a single template, a pre-set library of templates with the same semantics but diverse sentence structures can be used. The system randomly selects a template each time it generates content to introduce text diversity. Furthermore, additional constraints can be imposed on the slots filled by the LLM, such as instructions in the prompts like, "Please describe the purpose of the rereading in no more than 5 words, such as 'expressing surprise,' 'making a comparison,' or 'highlighting the core,'" thereby further guiding the LLM to generate accurate and concise content.

[0106] This invention transforms open reasoning generation tasks into structured fill-in-the-blank tasks by using templated prompts, effectively reducing the probability of LLM generating unreasonable content and ensuring the consistency of generated data in terms of format and basic logic.

[0107] It should be noted that any unidirectional data processing pipeline may be subject to the risk of errors amplifying at each stage, and the Pipeline of this invention is no exception. To ensure the high quality and reliability of the final generated quadruple data, this invention introduces a reliability verification mechanism for intermediate results (acoustic features) to prevent low-quality data from flowing downstream.

[0108] In some embodiments, the present invention implements a hierarchical hybrid verification strategy at the acoustic feature extraction node. Specifically: First layer: Confidence threshold filtering. For tools that can provide intrinsic confidence scores (such as WPT, pYin), set a threshold for initial screening to quickly filter out obviously unreliable results.

[0109] The second layer: cross-validation. For features that are prone to errors or have a significant impact on subsequent processes (such as emotion and accent), at least one independent technical approach is introduced for result comparison, and the result is adopted only when the results are consistent or highly similar.

[0110] Decision logic: Only features that pass both confidence filtering (if applicable) and cross-validation will be passed to the LLM for subsequent scene and inference generation.

[0111] Specifically, the following section elaborates on the implementation details for two important and error-prone aspects: emotional attribute features and word-level stress features.

[0112] 1. Mixed verification of emotional attribute features: Ensure that the emotion labels passed to the LLM are reliable to avoid the LLM generating completely illogical scenarios due to emotion recognition errors (e.g., describing a "sad" speech as a "celebration" scenario). The specific process is as follows: (1) Extraction and preliminary confidence assessment: Using the Kimi-Audio model as the main model, emotion recognition is performed on the input speech to obtain its predicted main emotion label (denoted as P1, such as "happy") and the confidence score of the prediction (denoted as S1).

[0113] Preset an emotional confidence threshold T1 (e.g., T1 = 0.7). Compare S1 with T1.

[0114] If the confidence level S1 is lower than or equal to the threshold T1, the system considers the emotion recognition result unreliable. The system will no longer perform cross-validation, but will instead conservatively label the emotion attribute of the speech as "Emotion: Uncertain". This label will be directly passed to the subsequent LLM, and the prompt words given to the LLM will explicitly state that this feature has high uncertainty, requiring the LLM to handle it appropriately when generating scenarios and inference.

[0115] (2) Cross-validation process (triggered when S1>T1): When the confidence level S1 is higher than the threshold T1, the cross-validation process is initiated.

[0116] A different sentiment recognition model than the Kimi-Audio model, but which can also output confidence scores, is used as an auxiliary validation model, such as the WavLM-Large model. This auxiliary model analyzes the same speech independently and outputs its predicted sentiment label (denoted as P2) and confidence score (denoted as S2).

[0117] (3) Consistent decision-making and final label determination: Scenario 1: Consistent Results. If the result P1 of the main model is exactly the same as the result P2 of the auxiliary model, or is highly semantically similar (e.g., "happy" and "excited"), the system considers the cross-validation to be successful. The sentiment label P1 from the main model is then adopted and passed to the LLM.

[0118] Scenario 2: Conflicting Outcomes. If P1 and P2 are completely different (e.g., "happy" and "sad"), the system enters a conservative decision-making phase: Compare S1 and S2. Adopt the sentiment label output by the model with higher confidence.

[0119] If both S1 and S2 are high (e.g., both greater than 0.8) but conflicting results are given, this indicates that the emotion of the current speech may be ambiguous. In this case, no specific emotion will be adopted, and instead, the safest and least informative label, "emotion: neutral," will be used as the final result; or the emotional attribute of the speech will be conservatively labeled as "emotion: uncertain."

[0120] 2. Mixed verification of word-level stress features: Ensure accurate location of accented words, as they are crucial for LLM to generate causal inferences. Incorrect location can lead to absurd interpretations. The specific process is as follows: (1) Extraction and significance threshold filtering: Using a first stress detection tool, such as the open-source Wavelet Prosody Toolkit (WPT) based on signal processing and prosodic saliency calculation, the speech is analyzed to calculate a stress saliency score for each word in the transcribed text.

[0121] A dynamic thresholding strategy is used for filtering: for example, in a speech segment, only words whose salience scores rank within a certain range globally (e.g., the top 15%) are retained as candidate stressed words (denoted as list W1). The remaining words are discarded directly due to insufficient salience and do not participate in subsequent analysis.

[0122] (2) Cross-validation process: For the candidate accented word list W1 that has passed the threshold filtering, cross-validation is initiated.

[0123] Cross-validation was performed using a second stress detection tool. This second stress detection tool refers to a model employing a semantic-prosodic joint analysis technique. Specifically, it can be a multimodal deep learning model based on a pre-trained architecture, such as using pre-trained speech models like Wav2Vec2.0 or HuBERT to extract prosodic features, and then fusing and jointly deciding with the semantic features output by text encoders like BERT. The technical principle of the second stress detection tool differs from that of the first stress detection tool. It is not based on pure signal processing, but rather on a comprehensive judgment combining the semantic context of words and the prosodic features of speech, outputting another list of stressed words W2. By utilizing natural language semantic understanding to compensate for the shortcomings of pure audio signal analysis, it can effectively identify words stressed in specific contexts to emphasize logical points or express emotions, thereby significantly improving the semantic relevance and accuracy of stress annotation, and providing a more reliable input basis for generating logically sound causal reasoning text.

[0124] (3) Consistency decision-making and final stress word determination: Calculate the intersection of list W1 and list W2 (i.e., words that appear in both lists).

[0125] Scenario 1: High Consistency. If the number of words in the intersection exceeds a preset consistency threshold T3 (e.g., T3 = 50%), the cross-validation is considered successful. The words from this intersection are then adopted as the confirmed accent words.

[0126] Scenario 2: Significant Disagreement. If the overlap is too small (less than T3), it indicates a serious disagreement between the two technical approaches at this point, and the result is unreliable.

[0127] At this point, a conservative strategy can be adopted: avoid using any stressed words to prevent conveying incorrect information.

[0128] Alternatively, as an alternative, check if there is a single word in the entire speech that is significantly more salient than the others. If such a word exists and is also within the intersection, then only this single word is adopted as the stressed word; otherwise, the strategy of "not marking stressed words" is still adopted.

[0129] In some embodiments, the robustness enhancement step also applies to paralinguistic information extraction. Specifically, the detection result is adopted and subsequent temporal semantic alignment processing is triggered only when the confidence score of the paralinguistic information detection model for a certain non-linguistic sound event (such as laughter, sighs, coughs, crying, etc.) is higher than a preset threshold. If the confidence score is lower than the threshold, the detection result is discarded, and the corresponding audio segment will not be labeled with paralinguistic information. This strategy effectively filters out uncertain outputs from the detection model, improving the accuracy of paralinguistic information labeling from the source.

[0130] Through the above design, this invention, while ensuring effectiveness, avoids the excessive computational overhead caused by indiscriminate full cross-validation, reduces the probability of passing erroneous or uncertain features to LLM, improves the logical rationality and quality of the final generated data from the source, and enhances the ability to process diverse and complex speech data.

[0131] It should be noted that when processing massive amounts of raw speech data to construct ultra-large-scale training sets, directly using large language models with a huge number of parameters for inference will result in extremely high GPU computing costs. To achieve the objectives of this invention in a more economical and efficient manner, the following knowledge distillation scheme provides a feasible solution.

[0132] In some embodiments, a "two-stage knowledge distillation" process is introduced: Phase 1 (Seed Data Generation): Using a high-performance, high-cost teacher model, a small subset of high-quality scenario and reasoning data is generated as the "standard" seed dataset.

[0133] Phase Two (Student Model Training and Large-Scale Generation): Using the seed dataset generated in Phase One, a student model with a much smaller number of parameters, specifically designed for this task, is trained. Thereafter, large-scale data generation tasks are all handled by the efficient student model.

[0134] This process transforms a one-time large-scale investment into a one-time small-scale investment (generating seed data) and reusable model training, thereby greatly reducing the overall cost while ensuring the baseline of data quality.

[0135] The specific implementation details are as follows: 1. First stage: Use the teacher model to generate high-quality seed data.

[0136] (1) Teacher model selection and input preparation: Model selection: High-performance, large-scale models were chosen as instructors, such as Qwen2.5-32B-Instruct or Mistral-Small-3.2-24B-Instruct. The goal was to ensure the highest quality seed dataset.

[0137] Data selection: From the original speech database, carefully select representative {speech, text} pairs. The number of selections (e.g., 1000 to 5000) can be determined according to specific needs.

[0138] Selection criteria include: High voice quality: high signal-to-noise ratio and clear pronunciation.

[0139] Diverse acoustic features: covering different speech rates, pitches, volumes, and emotions.

[0140] Diverse text content: covering texts from different fields and styles.

[0141] Feature extraction and rigorous filtering: For these 1000-5000 data points, the acoustic feature extraction of the present invention is performed, and the robustness enhancement strategy of "confidence threshold + cross-validation" in the aforementioned embodiments is enabled to ensure that the acoustic features input to the teacher model are reliable.

[0142] (2) Generation and Post-processing: The processed high-quality {text, acoustic features} pairs are input into the teacher model to generate corresponding [scene description] and [causal reasoning] text.

[0143] The generated results undergo necessary manual sampling review and correction to ensure logical coherence and fluent language. This results in a high-quality "seed dataset" containing approximately 1000-5000 well-structured quadruple data sets representing [scene, text, reasoning, and speech].

[0144] 2. Second stage: Training student models and using them for large-scale generation.

[0145] (1) Student model selection and training data construction: Model selection: Choose an open-source model with a small number of parameters (e.g., 1B, 7B, or 14B), fast inference speed, and suitable for fine-tuning as the student model base, such as Qwen2.5-7B-Instruct or Llama-3-8B-Instruct.

[0146] Training data format construction: The seed dataset is converted into the dialogue format required for fine-tuning the student model. Each data point is used to construct a "dialogue pair": Input: "Text: [Enhanced transcribed text]. Acoustic features: [Macroprosodic], [emotion], [stressed words], [paralinguistic information], etc." It should be noted that [paralinguistic information] here refers to the extracted list of tags, such as laughter, cough, etc.

[0147] Output: "Scene: [Generated scene description]. Reasoning: [Generated causal reasoning text]". Using the above method, the task is defined as a text generation task that generates "scene + reasoning" based on "text + acoustic features".

[0148] (2) Student model training (distillation process): The Supervised Fine-Tuning (SFT) method is used to fine-tune the student model base using pre-constructed training data (i.e., seed dataset).

[0149] Training objective: To enable the student model to learn and mimic the generative style and logical capabilities of the teacher model, that is, to generate scene and reasoning text of similar quality to the teacher model based on the input acoustic features. Furthermore, efficient parameter fine-tuning techniques such as QLoRA can be used to further reduce the GPU memory resources required for training.

[0150] (3) Deployment and large-scale generation of student models: After training, the student model is deployed. For the large amount of speech data to be processed, the acoustic features obtained through the previous steps are combined with the enhanced transcribed text and then input into the student model in batches. The student model efficiently generates the corresponding scene and inference text, thus completing the construction of a large-scale dataset. Because the student model has small parameters and extremely fast inference speed, this stage achieves cost-effective generation of large-scale data.

[0151] It is understandable that this invention addresses the bottleneck of large-scale data generation from an economic perspective through a knowledge distillation scheme, shifting the main cost from expensive large-model inference to one-time student model training, with extremely low marginal costs for subsequent generation. Simultaneously, by preparing a seed dataset as a standard, it ensures a minimum quality level for the data generated by the student model, achieving a balance between cost and quality. Furthermore, the framework itself is scalable, allowing for easy integration of better teacher models or new scenario requirements in the future, enabling continuous evolution of capabilities.

[0152] It should be noted that this application employs knowledge distillation, aiming to achieve a significant improvement in generation efficiency at the cost of acceptable performance loss. In practical applications, the generation results of the teacher model and the student model can be compared by sampling to ensure that the generation quality of the student model meets a preset threshold before it is used for large-scale data generation.

[0153] It should be noted that, in the specific implementation of this invention, the acoustic feature extraction tools and language models relied upon can be based on existing mature technologies. For clarity, the key references and corresponding technical terms cited in the specification are listed below: Speechcraft [1] ; Kimi-Audio [2 ; pYin [3] ; wavelet prosody toolkit [4] ; Qwen2.5-32B-Instruct [5] ; Mistral-Small-3.2-24B-Instruct [6] .

[0154] The corresponding reference list is as follows: [1] Jin Z, Jia J, Wang Q, et al. Speechcraft: A fine-grained expressive speech dataset with natural language description[C] / / Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 1255-1264. [2] Ding D, Ju Z, Leng Y, et al. Kimi-audio technical report[J].arXiv preprint arXiv:2504.18425, 2025. [3] Mauch M, Dixon S. pYIN: A fundamental frequency estimator using probabilistic threshold distributions[C] / / 2014 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 2014: 659-663. [4] Suni A, Šimko J, Aalto D, et al. Hierarchical representation and estimation of prosody using continuous wavelet transform[J]. Computer Speech&Language, 2017, 45: 123-136. [5] Team Q. Qwen2 technical report[J]. arXiv preprint arXiv:2407.10671, 2024, 2(3). [6] Jiang D, Liu Y, Liu S, et al. From clip to dino: Visual encodersshout in multi-modal large language models[J]. arXiv preprint arXiv:2310.08825, 2023. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0159] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A method for generating scenario-based speech synthesis training data, characterized in that, Includes the following steps: The speech data is parsed to obtain its acoustic feature representation, which includes prosodic features extracted from the speech data; Based on the acoustic feature representation and the transcribed text corresponding to the speech data, the reasoning ability of the language model is used to generate scene description text describing the background of the speech data generation. Based on the scene description text, the transcribed text, and the acoustic feature representation, the reasoning ability of the language model is used to generate causal reasoning text that explains the logical relationship between the acoustic feature representation, the scene description text, and the transcribed text; The scene description text, the transcribed text, the causal reasoning text, and the speech data are associated to form training data for training the speech synthesis model.

2. The generation method as described in claim 1, characterized in that, The prosodic features include: macro-prosodic features, used to describe the global auditory properties of the entire speech segment; and word-level stress features, used to identify one or more words in the speech that need to be stressed.

3. The generation method as described in claim 2, characterized in that, The macro-prosodic features include at least one of pitch, volume, and speech rate.

4. The generation method as described in claim 3, characterized in that, The extraction of the macro-prosodic features includes quantizing at least one of the features of pitch, volume, and speech rate into discrete labels.

5. The generation method as described in claim 1, characterized in that, The acoustic feature representation also includes emotional attribute features extracted from the speech data.

6. The generation method as described in claim 1, characterized in that, The acoustic feature representation also includes paralinguistic information extracted from the speech data, which includes at least one of laughter, sighs, coughs, and cries.

7. The generation method as described in claim 6, characterized in that, The steps for extracting the paralinguistic information include: detecting non-linguistic sound events in the speech data and obtaining their timestamp information; aligning the tags corresponding to the non-linguistic sound events with the transcribed text of the speech data in a temporal semantic manner to generate enhanced transcribed text; and applying the enhanced transcribed text to the generation of the scene description text and the causal inference text.

8. The generation method as described in claim 1, characterized in that, The language model used to generate the scene description text and / or the causal reasoning text is a Large Language Model (LLM).

9. The generation method as described in claim 1, characterized in that, It also includes a robustness enhancement step, which includes at least one of the following: cross-validating the extracted acoustic feature representation; applying confidence threshold filtering to the extracted acoustic feature representation.

10. The generation method as described in claim 9, characterized in that, When the acoustic features are characterized as emotional attribute features, the robustness enhancement step includes: using a first emotion recognition model to extract emotional attribute features and obtain a first confidence level; if the first confidence level is lower than a first threshold, then marking the emotional attribute as uncertain; if the first confidence level is higher than the first threshold, then using a second emotion recognition model for cross-validation, and determining the final emotional attribute based on the output results of the first emotion recognition model and the second emotion recognition model.

11. The generation method as described in claim 9, characterized in that, When the acoustic feature is characterized as a word-level stress feature, the robustness enhancement step includes: obtaining a first stress word list using a first stress detection tool; obtaining a second stress word list using a second stress detection tool; comparing the first stress word list with the second stress word list, and taking the overlapping words that exist in both lists as the finally confirmed stress words.

12. The generation method as described in claim 1, characterized in that, The step of generating the causal reasoning text employs a templated cue word generation method, which constrains the generated content of the language model through predefined structured cue word templates.

13. The generation method as described in claim 1, characterized in that, The step of generating the scene description text is constrained by a scene-acoustic mapping knowledge base, which defines the mapping relationship between scene types and the acceptable range of acoustic features.

14. A system for generating scenario-based speech synthesis training data, characterized in that, include: The parsing module is configured to parse speech data to obtain its acoustic feature representation. The scene generation module is configured to generate scene description text describing the background of the speech data generation based on the acoustic feature representation and the transcribed text corresponding to the speech data, using the reasoning ability of the language model. The reasoning generation module is configured to generate causal reasoning text that explains the logical relationship between the acoustic feature representation and the scene description text and the transcribed text, based on the scene description text, the transcribed text and the acoustic feature representation, using the reasoning ability of the language model; The data association module is configured to associate the scene description text, the transcribed text, the causal inference text, and the speech data to form training data for training the speech synthesis model.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the generation method as described in any one of claims 1 to 13.

16. An electronic device, characterized in that, include: processor; And a memory for storing a computer program executable by the processor; wherein, when executed by the processor, the computer program implements the generation method as described in any one of claims 1 to 13.

17. A scene-based speech synthesis training dataset generated by the generation method according to any one of claims 1 to 13, characterized in that, The dataset consists of multiple training samples, each containing the following associated elements: a speech data segment; the transcribed text corresponding to the speech data segment; a scene description text describing the background in which the speech data segment was generated; and a causal inference text explaining the logical relationship between the acoustic feature representation of the speech data segment and the scene description text and the transcribed text.

18. A scenario-based speech synthesis training data generation system, characterized in that, The contextualized speech synthesis training data generation system is configured to generate the contextualized speech synthesis training dataset as described in claim 17, wherein the contextualized speech synthesis training data generation system comprises: The acoustic feature analysis module is used to analyze the input speech data to obtain its acoustic feature representation. The scene text generation module is connected to the acoustic feature parsing module and is used to generate scene description text based on the acoustic feature representation and the corresponding transcribed text; The reasoning text generation module, connected to the scene text generation module and the acoustic feature parsing module, is used to generate causal reasoning text based on the scene description text, the transcribed text, and the acoustic feature representation; The data synthesis and output module is connected to the scene text generation module and the reasoning text generation module, and is used to associate and output the scene description text, the transcribed text, the causal reasoning text and the speech data.

19. A training method for a speech synthesis model, characterized in that, include: Obtain the contextualized speech synthesis training dataset as described in claim 17; The speech synthesis model is trained using the contextualized speech synthesis training dataset, so that the trained speech synthesis model learns to generate synthesized speech that conforms to the context described in the contextual description text based on the input contextual description text and the text to be synthesized.

20. A speech synthesis method, characterized in that, include: The system receives input scene description text and text to be synthesized; inputs the input scene description text and text to be synthesized into a speech synthesis model; and obtains the synthesized speech output by the speech synthesis model; wherein the speech synthesis model is trained using the training method described in claim 19.

21. A speech synthesis device, characterized in that, include: processor; and a memory coupled to the processor and used to store a computer program; wherein, when executed by the processor, the computer program causes the device to implement the speech synthesis method as described in claim 20.

Images