Multi-modal voice interaction large model training method and system based on voice acoustic feature regulation and control, terminal equipment and medium

Abstract

The invention discloses a multi-modal voice interaction large model training method and system based on voice acoustic feature regulation and control, terminal equipment and a medium, and relates to the technical field of multi-modal voice interaction.The method comprises the steps that a text token of a text training sample is obtained, a corresponding voice token is constructed, and pre-training data used for converting the text token into the voice token is obtained; in combination with the multi-modal input sample and the pre-training data, constructing fine-tuning training data for speech understanding and dialogue generation; constructing and pre-training a basic model by using the pre-training data; a multi-modal voice interaction large model is constructed based on a pre-training basic model, and fine tuning data is used for training, so that voice acoustic features can be regulated and controlled based on multi-modal input, and voice is output. According to the method, through alignment and staged training of the text token and the voice token, fine regulation and control of voice acoustic characteristics are realized, long voice continuity and interaction naturalness are improved, and a model is efficiently endowed with voice interaction capability of controllable timbre and emotion.

Description

Technical Field

[0001] This invention relates to the field of multimodal voice interaction technology, and in particular to a method, system, terminal device and medium for training a large multimodal voice interaction model based on the modulation of voice acoustic features. Background Technology

[0002] In the field of multimodal speech interaction technology, the multimodal large model GPT-4o enables real-time multimodal speech interaction, driving the development of end-to-end speech-to-speech systems. Such systems eliminate the need for intermediate text generation, directly processing speech modalities and improving the naturalness and responsiveness of the interaction. Meanwhile, the emergence of open-source large language model frameworks such as Moshi, Mini-Omni, and LLaMA-Omni allows multimodal large language models to incorporate speech capabilities, reducing latency in traditional speech-text-speech systems and solving the alignment problem between audio and text modalities.

[0003] However, existing technologies still have shortcomings. First, the response speech of large-scale voice interaction models lacks diversity. Most speech codecs over-compress audio, resulting in generated speech with a monotonous timbre and a lack of emotional or other parasemantic expression capabilities, making it difficult to meet personalized interaction needs. These issues affect the practicality and improvement of the user experience of multimodal voice interaction systems. Second, existing large-scale voice interaction models have weak processing capabilities for long speech. Data sets for long speech understanding and generation are scarce, and after encoding long speech, the number of processing units far exceeds that of text of equal length, placing higher demands on training resources and model contextual capabilities. The mismatch between text and speech lengths also limits generation.

[0004] Therefore, there is an urgent need for a training method for large-scale multimodal speech interaction models that can efficiently process long speech, generate diverse speech, and adapt to commonly used multimodal models, in order to fill the gaps in existing technologies. Summary of the Invention

[0005] The technical problem this invention aims to solve in the field of multimodal speech interaction technology is that existing models suffer from insufficient speech diversity due to over-compression of speech codecs, difficulty in generating long speech, scarcity of relevant datasets, large number of coding units, mismatch between text and speech lengths, high resource consumption for transferring multimodal speech interaction capabilities, difficulty in accurately capturing modal semantic relationships, low training efficiency, and poor scalability. Therefore, an effective solution is urgently needed to address these technical problems.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation, the method comprising: Obtain the text token of the text training sample, and construct a speech token containing the semantic features and speech acoustic features expressed by the text token based on the text token, so as to obtain pre-training data for converting the text token into a speech token; Based on the multimodal input samples and the pre-training data, fine-tuning training data for speech understanding and dialogue generation is constructed, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations; Based on the pre-trained data, a basic model is constructed and pre-trained, wherein the basic model is used to convert the input text token into a speech token; Based on the pre-trained base model, a large multimodal speech interaction model is constructed, and the fine-tuned training data is used to train the large multimodal speech interaction model. The large multimodal speech interaction model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs and output the speech responses.

[0007] In one implementation, the step of obtaining text tokens from text training samples and constructing speech tokens containing semantic features and acoustic features expressed by the text tokens to obtain pre-training data for converting text tokens into speech tokens includes: Acquire initial training data for single voice, which includes speech format dialogue plain text data and long text dataset. The speech format dialogue plain text data includes a filtered general natural dialogue text dataset and a reconstructed dialogue text dataset. The reconstructed dialogue text dataset is obtained by reconstructing the plain text instruction fine-tuning data based on preset prompt words using a general large model. Based on preset rules, special characters in the response round text content of the voice format dialogue plain text data and special characters in the long text dataset are filtered out, and symbol partitioning is used for segmentation to obtain preprocessed single-voice training initial data. The initial training data for the single voice is converted into a token to obtain the text token, and the semantic features expressed by the text token are extracted. Speech synthesis is performed on the initial training data of the single voice to generate matching single voice speech samples, and speech acoustic features are extracted from the single voice speech samples; The speech token is obtained by associating the text token, the semantic features expressed by the text token, and the speech acoustic features.

[0008] In one implementation, the step of obtaining text tokens from text training samples and constructing speech tokens containing semantic features and speech acoustic features expressed by the text tokens to obtain pre-training data for converting text tokens into speech tokens further includes: Acquire initial training data for multi-person voices, which includes dialogue text, corresponding multi-person voice data, and corresponding speech acoustic feature annotations. The initial training data for the multi-person voice is cleaned and tokenized to obtain the text token, and the semantic features expressed by the text token are extracted. The speech token is obtained by associating the text token, the semantic features expressed by the text token, and the speech acoustic feature annotations.

[0009] In one implementation, the fine-tuning training data includes speech understanding capability alignment data, fine-tuning data from multi-turn speech interaction dialogues, text-to-speech generation polishing fine-tuning data, and a pre-trained capability-preserving dataset. The fine-tuning training data for speech understanding and dialogue generation is constructed based on multimodal input samples and the pre-training data. The fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations, including: Acquire speech-to-text sample data, perform speech and semantic annotation on the speech-to-text sample data, and construct the speech understanding ability alignment data based on the speech-to-text sample data and the corresponding speech and semantic annotations, wherein the speech understanding ability alignment data is used for learning the speech understanding ability of the model. Based on the multi-turn dialogue data in the pre-training data, the context association logic of the multi-turn dialogue data is obtained, and based on the multi-turn dialogue data and the corresponding context association logic, the fine-tuning data of the voice interaction multi-turn dialogue is constructed. The voice interaction multi-turn dialogue includes dialogue generation logic annotation, and the fine-tuning data of the voice interaction multi-turn dialogue is used for learning the context association logic of the model. Obtain spoken language output sample data and construct the text speech generation polishing and fine-tuning data, wherein the text speech generation polishing and fine-tuning data is used for the model's spoken language learning; Obtain multimodal input samples and construct the pre-trained capability-preserving dataset, wherein the multimodal input samples include at least visual input samples, text input samples, and speech input samples, and the pre-trained capability-preserving dataset is used to learn the multimodal recognition capability of the model.

[0010] In one implementation, constructing a base model and performing pre-training based on the pre-trained data includes: The text token, the preceding voice token, and the preset prompt words based on the pre-trained data are input into the pre-constructed speech autoregressive module of the base model to obtain the predicted voice token. The speech autoregressive module includes 24 Transformer blocks. The predicted speech token is output to a 4096-dimensional prediction head to obtain the predicted synthesized speech; The loss function is calculated iteratively, and the parameters of the base model are optimized through backpropagation.

[0011] In one implementation, when the base model generates a speech response, it generates synthesized speech of arbitrary length through block synthesis, including: The input text corresponding to the speech to be generated is divided into several semantically complete text blocks, with a period as the unique truncation position. According to the order of the text blocks, the text token corresponding to the first text block is input into the pre-trained base model, and the speech token corresponding to the first text block and the synthesized speech segment are generated through the speech autoregression module. The voice token corresponding to the text block is used as a voice information prompt, and together with the text token of the next text block, it is input into the basic model to generate the corresponding voice token and synthesized voice segment; According to the order of the text blocks, generate the synthesized speech segments corresponding to all text blocks, and concatenate all the synthesized speech segments to obtain synthesized speech of arbitrary length.

[0012] In one implementation, the step of constructing a large-scale multimodal voice interaction model based on the pre-trained base model, and training the large-scale multimodal voice interaction model using the fine-tuned training data, includes: The Whisper decoder is modified using the Qformer architecture to construct a large-scale multimodal voice interaction model. The structural expression of the large-scale multimodal voice interaction model is as follows:

[0013]

[0014]

[0015]

[0016]

[0017] in, For fixed-length query vectors, For vectors, For hidden layer output, For audio, For query, It constitutes a fixed-length query vector The Hidden layer output components It is the top-level output of the pre-trained audio encoder, Whisper encoder. For the multi-head self-attention layer, For multi-head cross-attention layer, For layer normalization, It is a multilayer perceptron. This is the corresponding output of the Whisper encoder. for The corresponding output content, for The corresponding output content, For fixed-length query vectors The multi-head self-attention computation performed, This represents the total number of initial vectors. This represents the output of the cross-attention module, used to extract the main content of the input audio. The output of the cross-attention module The hidden layer output after processing by a multilayer perceptron is then subjected to four identical operations, and a learnable linear layer is applied to project the final output onto the representation space of a large language model. The fine-tuned training data is used to train the large multimodal voice interaction model.

[0018] Secondly, embodiments of the present invention also provide a multimodal speech interaction large model training system based on speech acoustic feature modulation, the system comprising: The pre-training data construction module is used to obtain text tokens of text training samples, and construct speech tokens containing semantic features and speech acoustic features expressed by the text tokens based on the text tokens, so as to obtain pre-training data for converting text tokens into speech tokens. The fine-tuning training data construction module is used to construct fine-tuning training data for speech understanding and dialogue generation based on multimodal input samples and the pre-training data, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations. The basic model construction and pre-training module is used to construct a basic model and perform pre-training based on the pre-training data, wherein the basic model is used to convert the input text token into a speech token. The multimodal speech interaction large model construction and training module is used to construct a multimodal speech interaction large model based on the pre-trained base model, and to train the multimodal speech interaction large model using the fine-tuned training data. The multimodal speech interaction large model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs, and to output the speech responses.

[0019] Thirdly, embodiments of the present invention also provide a terminal device, the terminal device including a memory, a processor, and a multimodal speech interaction large model training program based on speech acoustic feature modulation stored in the memory and executable on the processor. When the processor executes the multimodal speech interaction large model training program based on speech acoustic feature modulation, it implements the steps of the multimodal speech interaction large model training method based on speech acoustic feature modulation as described in any of the above schemes.

[0020] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a training program for a large multimodal speech interaction model based on speech acoustic features. When the training program for a large multimodal speech interaction model based on speech acoustic features is executed by a processor, it implements the steps of the training method for a large multimodal speech interaction model based on speech acoustic features as described in any of the above schemes.

[0021] Beneficial Effects: A method, system, terminal device, and medium for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation, relating to the field of multimodal speech interaction technology, the method includes: firstly, acquiring text tokens of text training samples, and based on the text tokens, constructing speech tokens containing semantic features and speech acoustic features expressed by the text tokens, obtaining pre-training data for converting text tokens into speech tokens. Subsequently, based on multimodal input samples and the pre-training data, constructing fine-tuning training data for speech understanding and dialogue generation, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations. Next, based on the pre-training data, constructing a basic model and performing pre-training, wherein the basic model is used to convert input text tokens into speech tokens. Finally, based on the pre-trained basic model, constructing a large-scale multimodal speech interaction model, and using the fine-tuning training data to train the large-scale multimodal speech interaction model, wherein the large-scale multimodal speech interaction model is used to modulate speech acoustic features for generating speech responses based on multimodal inputs, and output the speech responses. This invention addresses the problems of insufficient speech diversity, difficulty in generating long speech, and high resource consumption in multimodal adaptation of existing models. It realizes acoustic features such as timbre and emotion, supports controllable and coherent generation of long speech, quickly activates the model's speech interaction capabilities, and retains the core multimodal capabilities, thereby improving the naturalness and generalization of speech interaction in large multimodal models. Attached Figure Description

[0022] Figure 1 The flowchart illustrates a specific implementation method for training a large multimodal speech interaction model based on speech acoustic feature modulation, as provided in this embodiment of the invention.

[0023] Figure 2 This is a schematic diagram of the pre-training data construction process of the multimodal speech interaction large model training method based on speech acoustic feature modulation provided in the embodiments of the present invention.

[0024] Figure 3 This is a schematic diagram of the process of constructing fine-tuning training data in the multimodal speech interaction large model training method based on speech acoustic feature modulation provided in the embodiments of the present invention.

[0025] Figure 4 This is a schematic diagram of each training stage of the multimodal speech interaction large model in the multimodal speech interaction large model training method based on speech acoustic feature modulation provided in the embodiments of the present invention.

[0026] Figure 5 This is a schematic diagram of the principle of a multimodal speech interaction large model training device based on speech acoustic feature modulation provided in this embodiment of the invention.

[0027] Figure 6 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0029] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content, operations, or steps, nor does it require execution in the described order. For example, some operations or steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0030] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0031] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. For example, "first control information" and "second control information" are only used to distinguish different control information and do not limit their order.

[0032] Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or the order of execution, and that the words "first" and "second" do not necessarily imply that they are different.

[0033] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] In the field of multimodal speech interaction technology, the multimodal large model GPT-4o enables real-time multimodal speech interaction, driving the development of end-to-end speech-to-speech systems. Such systems eliminate the need for intermediate text generation, directly processing speech modalities and improving the naturalness and responsiveness of the interaction. Meanwhile, the emergence of open-source large language model frameworks such as Moshi, Mini-Omni, and LLaMA-Omni allows multimodal large language models to incorporate speech capabilities, reducing latency in traditional speech-text-speech systems and solving the alignment problem between audio and text modalities.

[0035] However, some problems still exist in existing technologies. First, current speech dialogue models struggle to effectively generate long speech. From a dataset perspective, existing datasets for long speech understanding and generation are severely lacking, failing to fully activate the model's comprehensive understanding and generation capabilities for long speech. From a model generation perspective, most models using speech codecs encode a large number of long speech units, resulting in the model needing to process ten to one hundred times more speech coding units than text of equal length. This places extremely high demands on training resources and the model's context processing capabilities. Furthermore, the mismatch between text and speech lengths also limits the generation capabilities of many models. Second, many existing speech models use speech codecs that excessively compress audio content, ultimately leading to a lack of diversity and uniformity in generated speech, failing to achieve expressive capabilities beyond semantics, such as variations in timbre and emotion.

[0036] Therefore, addressing the shortcomings of existing speech dialogue models in long text processing, diverse speech synthesis, and multimodal adaptation, this paper proposes an end-to-end aligned speech dialogue structure to enhance the long speech interaction capabilities of large language models. Specifically, at the speech understanding level, a fine-grained pre-trained speech and audio processing model, Whisper, is used as the input module. Its encoder is innovatively used as the speech encoding structure, and a decoder combining a query token of length 200 is constructed to build a cross-modal feature alignment architecture, QFormer projection architecture, based on query vectors and attention mechanisms. A variable-length encoding strategy of 200 speech tokens (speech tokens, cross-modal feature units) every 30 seconds is adopted to balance information acquisition and encoding efficiency, achieving modal mapping from speech to text. Simultaneously, a speech generation module is designed, consisting of 24 Transformer layers, a deep learning network architecture based on attention mechanisms and containing cross-attention layers. This module is initialized with a pre-trained large language model, QFormer, endowed with prior text understanding knowledge, to complete the processing of text tokens (text tokens, the smallest semantic unit) into speech tokens. During the generation phase, text and speech are generated synchronously at a rate of 40 speech tokens per second, and the speech synthesis duration is dynamically controlled based on the text length. Through multi-stage training, the speech synthesis module's capabilities are first enhanced using large-scale text-speech pairs, and then the model's speech and audio understanding capabilities are activated by constructing data and alignment modules, thus standardizing the speech output format.

[0037] This approach endows the model with powerful long-speech understanding and diverse generation capabilities, efficiently capturing fine-grained speech information while handling extremely long speech content with accurate responses. A synchronous generation mechanism ensures consistency between speech and text, and a multi-stage training strategy gives the model robust speech interaction performance. Ultimately, while retaining the textual capabilities of large language models, the model significantly improves its performance in complex life scenarios such as emotion recognition and empathetic dialogue, achieving natural, fluent, and variable-length speech dialogues. It can also quickly adapt to different multimodal large model frameworks, maximizing the retention of the original model's core dialogue capabilities.

[0038] This embodiment provides a method for training a large multimodal speech interaction model based on the modulation of speech acoustic features, such as... Figure 1 As shown, the specific steps include the following: Step S100: Obtain the text token of the text training sample, and construct a speech token containing the semantic features and speech acoustic features expressed by the text token based on the text token, so as to obtain pre-training data for converting the text token into a speech token.

[0039] In this embodiment, the construction of the pre-training data aims at high-quality speech generation. Specifically, through the design of a specialized speech-text dataset, the synergy of text semantics, cue word information, and speech features is achieved. This ensures that after the model outputs text tokens, the text-to-speech module can accurately recognize and process them, generating speech tokens with highly matched semantics and prosody. This provides training data support for subsequent pre-training of the basic model and the training of a large-scale multimodal speech interaction model. To achieve the above objectives, the construction of the dataset in this embodiment follows three core principles.

[0040] The first principle is the adaptation to the natural dialogue paradigm. The training data focuses on everyday voice interaction scenarios, retaining only text content that conforms to the logic of human spoken communication, and strictly eliminating non-dialogue elements such as mathematical symbols, code snippets, tables, and tree diagrams. This avoids special formats or non-communicative information from interfering with the semantic coherence and rhythmic naturalness of speech synthesis, ensuring the purity of text expression and interactive adaptability.

[0041] Secondly, there is the principle of ensuring purity in a single language. The training data adopts a single-language data construction mode, and separate datasets for English and Chinese are established to avoid the loss of speech synthesis quality caused by differences in pronunciation rules and conflicts in intonation and prosody when multiple languages ​​are mixed, as well as the problem of semantic mapping deviation between different languages, thereby improving the accuracy and naturalness of speech synthesis in single-language scenarios.

[0042] Thirdly, there is the principle of multi-dimensional data coverage. The training dataset covers multiple domains, multiple scenarios, and diverse text lengths. Specifically, the multiple domains include everyday chatter, work communication, education and tutoring, and life services; the multiple scenarios include family conversations, workplace meetings, customer service interactions, and social chatter; and the diverse text lengths include content ranging from short question-and-answer sentences to long text stories and explanatory content. This rich data distribution enhances the model's adaptability to the speech generation needs of different scenarios, improves the robustness and generalization of model training, and avoids model functional bias caused by limited data.

[0043] In one implementation, the step of obtaining text tokens from text training samples and constructing speech tokens containing semantic features and speech acoustic features expressed by the text tokens to obtain pre-training data for converting text tokens into speech tokens specifically includes the following steps: Step S110: Obtain initial training data for single voice, which includes speech format dialogue plain text data and long text dataset. The speech format dialogue plain text data includes a filtered general natural dialogue text dataset and a reconstructed dialogue text dataset. The reconstructed dialogue text dataset is obtained by reconstructing the plain text instruction fine-tuning data based on preset prompt words using a general large model. Step S120: Based on preset rules, filter special characters in the response round text content of the voice format dialogue plain text data and special characters in the long text dataset, and use symbol partitioning method to segment to obtain preprocessed single voice training initial data; Step S130: Perform token conversion on the initial data of the single voice training to obtain the text token, and extract the semantic features expressed by the text token; Step S140: Perform speech synthesis on the initial training data of the single voice to generate matching single voice speech samples, and extract speech acoustic features from the single voice speech samples; Step S150: Associate the text token, the semantic features expressed by the text token, and the speech acoustic features to obtain the speech token.

[0044] In this embodiment, initial training data for single-voice training is first constructed. Figure 2The process of constructing pre-training data is demonstrated. First, the initial pre-training data is divided into four categories, selecting four types of natural dialogue text datasets as the initial training data for single-voice speech. The first is the MuTual dataset, a two-person multi-turn dialogue dataset adapted from the National College Entrance Examination (NCEE) English listening comprehension questions. To more naturally simulate open-domain dialogue, MuTual further transforms additional questions in the listening comprehension questions into responses in the dialogue, with the turn counts of the model's responses used for speech synthesis pre-training. The second is the length-washed UltraChat dataset, containing 300K multi-turn dialogue data, with the turn counts of the model's output used as speech synthesis pre-training data, aligning subsequent training data with the text-to-speech data. The third is the TinyStories English dataset, a short story synthesis dataset generated by large models GPT-3.5 and GPT-4, and the TinyStories-zh Chinese story dataset generated by the large model qwen. This story dataset only contains long text content, i.e., a long text dataset, and all text in the data is used for speech synthesis training. Fourthly, there is the GPT-4 augmented dataset, which includes datasets of simulated everyday conversations directly generated using the GPT-4 model. This data, through adjusting prompts and calling the GPT-4 interface, constructs natural dialogues covering multiple aspects of everyday conversations. Therefore, based on the specific requirements of the generated dataset, prompts with clear dialogue generation logic and quality constraints are constructed. Specifically, these prompts mainly contain four types of key information. The first type of information is the specific definition of the task, instructing the large model to generate multi-turn dialogues that conform to real-world scenarios. The content must be reasonable, possess contextual logic, and the participants' statements must be concise and meaningful. The second type of information defines the dialogue types and topics, providing various dialogue types such as greetings, small talk, information exchange, discussion, emotional support, and decision-making, as well as multiple everyday topics such as weather, work, family, health, travel, and technology. The prompts will randomly select one or two dialogue types and a specific topic as the basis for generation. The third type of information is the generation rule constraints, specifying that the dialogue only includes two unnamed participants, the total number of rounds must be even, and both parties must speak an equal number of times. Repetition or overuse of simple synonymous words is prohibited. The fourth type of information is the output format specification, requiring the large model to return results in a JSON structure, containing three fields: dialogue type, dialogue topic, and dialogue content. The dialogue content field must individually label the speakers and their corresponding statements. By combining the above-mentioned prompt word elements, the scene adaptability and data standardization of the generated dialogue can be controlled, providing a text foundation for subsequent speech synthesis pre-training. In practical applications, Chinese or English prompt words can be selected according to the language preferences of the training data, and the specific expression details of the prompt words can be adjusted according to the scene emphasis.

[0045] In addition, several publicly available general fine-tuning datasets used by existing large-scale text language models were employed for constructing the speech dialogue data. These fine-tuning datasets specifically include the filtered Moss, OpenHermes, Alpaca, and GPTeacher English fine-tuning datasets, as well as some Belle Chinese instruction fine-tuning datasets. Specifically, Moss is an open-source Chinese multi-turn dialogue dataset with over 1.1 million data entries; OpenHermes is an English instruction fine-tuning dataset with over 1 million data entries; Alpaca contains over 50,000 cleaned English instruction pairs; GPTeacher is a modular English dataset generated by GPT-4; and Belle is a large-scale Chinese instruction dataset with over 3.5 million data entries. The input and output of this data were used to instruct the large-scale model interface for data reconstruction, aiming to construct high-quality text data that conforms to everyday dialogue patterns. The corresponding prompt words used in the constructed data were optimized for text-to-speech adaptability, aiming to rewrite the original speech interaction instruction data into speech format text adapted to the large-scale speech input-output model training. These prompt words mainly contain three key constraints. The first type of constraint aims to make the generated data conversational, ensuring the text closely resembles real human speech. Filler words can be appropriately inserted to enhance naturalness, but overuse should be avoided to prevent redundant expressions. The second type of constraint is synthesis compatibility, explicitly prohibiting content that text-to-speech models cannot process. For numerical information, it must be presented in English word spelling rather than Arabic numerals to ensure the feasibility of subsequent speech synthesis. The third type of constraint is conciseness, requiring the rewritten text to remain succinct, avoiding lengthy expressions, and conforming to the concise communication characteristics of everyday conversation. By reconstructing the original fine-tuned data using these cue word elements, the conversational quality and speech synthesis adaptability of the text data can be improved, making it more suitable for the actual needs of voice interaction scenarios. This provides a high-quality text foundation for subsequent voice token construction and model pre-training. Similarly, in practical applications, Chinese or English cue words can be selected based on the language preferences of the training data, and the specific details of the cue word expression can be adjusted according to the scenario emphasis.

[0046] Speech synthesis is performed using the rounds of model responses from such text datasets, and then the configuration of such text-speech pairs is input into the model for pre-training.

[0047] To further improve data quality, a multi-layered filtering mechanism was designed. This involved filtering out text containing special characters using rules, and segmenting long text data from the UltraChat and TinyStories datasets using a symbolic partitioning method. Specifically, a fixed length was used as a baseline, prioritizing ensuring that the segmented text ended with punctuation. If the text was excessively long and lacked punctuation, it was forcibly segmented and punctuation was added. Finally, based on the segmented data, speech synthesis was achieved by calling the Microsoft Edge text-to-speech interface, constructing three types of single-voice models: male voices for multiple languages, female voices adapted for English synthesis, and female voices adapted for Chinese synthesis. A retry mechanism was implemented in the synthesis process, handling failed cases in a loop until the maximum number of attempts was reached or synthesis was successful, ensuring data integrity and usability. Ultimately, the characteristic synthesis prompts for this type of single-voice training data adopted a structured parameter configuration format, containing two types of key information. The first type of key information is the language type parameter, with options covering English and Chinese, used to match the language scenarios of the training data. The second type of key information is the speaker parameter, providing three preset voice options: male voices supporting multiple languages, female voices adapted for English synthesis, and female voices adapted for Chinese synthesis, used to control the timbre characteristics of the speech samples. This standardized prompt format allows for the definition of speech synthesis configurations, aligning the generated speech samples with the text data and preset acoustic feature requirements, providing a standardized speech material foundation for the subsequent construction of speech tokens.

[0048] In one implementation, the step of obtaining text tokens from text training samples and constructing speech tokens containing semantic features and speech acoustic features expressed by the text tokens to obtain pre-training data for converting text tokens into speech tokens further includes the following steps: Step S160: Obtain initial training data for multi-person voices, which includes dialogue text, corresponding multi-person voice speech data, and corresponding speech acoustic feature annotations. Step S170: Clean the initial data of the multi-person voice training and perform token conversion to obtain the text token, and extract the semantic features expressed by the text token; Step S180: Associate the text token, the semantic features expressed by the text token, and the speech acoustic feature annotation to obtain the speech token.

[0049] In this embodiment, as Figure 2For the multi-voice emotional speech dataset portion, specifically the multi-speech training aspect, several high-quality open-source datasets were selected and applied to the pre-training process of the speech synthesis model. One such dataset is the Vccm dataset, a speech synthesis dataset optimized based on the TextrolSpeech dataset. TextrolSpeech contains 330 hours of speech data and 236,203 style description texts. Building upon this, the Vccm dataset was optimized for pitch distribution, label boundaries, and dataset partitioning, and the test set was re-selected. The Vccm dataset is constructed using the emotional data from the LibriTTS and TextrolSpeech datasets as a foundation, labeling each speech sample with five attributes: gender, volume, speech rate, pitch, and emotion. Through multi-dimensional data annotation and structured processing, a high-quality text-to-speech dataset with refined acoustic features is formed. LibriTTS, in particular, is a dataset of over 500 hours of male and female English speech covering over 2,000 speakers. Secondly, there is the Speechcraft dataset, a large-scale, emotion-rich bilingual speech dataset. It utilizes an automatic speech annotation system to generate natural language descriptions, covering over 2 million audio segments, and includes two versions of text prompts for annotation. The first version contains speech descriptions without transcripts and focuses on speech features; the second version contains speech instructions along with transcripts and directive information. This provides rich training resources for bilingual speech understanding, emotion recognition, and cross-modal research, and is suitable for developing speech processing models that more closely resemble real-world scenarios.

[0050] The training data for multi-voice speakers does not require speech reconstruction, but text cleaning is necessary. Because the dataset still contains many special characters, it significantly impacts the speech-to-text paradigm. Furthermore, some datasets use all uppercase letters and words to represent punctuation, requiring content reconstruction according to rules to ultimately obtain text content that conforms to the speaking pattern.

[0051] Ultimately, as Figure 2 As shown, by constructing pre-training initial data containing single-voice training data and multi-voice training data, the final pre-training synthetic data is obtained.

[0052] Step S200: Based on the multimodal input samples and the pre-training data, construct fine-tuning training data for speech understanding and dialogue generation, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations.

[0053] In this embodiment, to enhance the model's ability to understand user speech, a batch of accurate and efficient fine-tuning data needs to be constructed to train the speech understanding and normalization generation capabilities of the multimodal large model. The data to be constructed should enable the model to possess strong audio understanding capabilities while effectively utilizing contextual information for reasoning and outputting text content that matches the corresponding audio response. This part of the data involves the speech understanding alignment of the multimodal large model and the integration of various capabilities, requiring multiple types of data.

[0054] In one implementation, the fine-tuning training data includes speech understanding capability alignment data, fine-tuning data from multi-turn speech interaction dialogues, text-to-speech generation polishing fine-tuning data, and a pre-trained capability-preserving dataset. The step of constructing fine-tuning training data for speech understanding and dialogue generation based on multimodal input samples and the pre-training data specifically includes the following steps: Step S210: Obtain speech-to-text sample data, perform speech semantic annotation on the speech-to-text sample data, and construct the speech understanding ability alignment data based on the speech-to-text sample data and the corresponding speech semantic annotation, wherein the speech understanding ability alignment data is used for learning the speech understanding ability of the model. Step S220: Based on the multi-turn dialogue data in the pre-training data, obtain the context association logic of the multi-turn dialogue data, and construct the fine-tuning data of the voice interaction multi-turn dialogue based on the multi-turn dialogue data and the corresponding context association logic. The voice interaction multi-turn dialogue includes dialogue generation logic annotation, and the fine-tuning data of the voice interaction multi-turn dialogue is used for learning the context association logic of the model. Step S230: Obtain spoken language output sample data and construct the text speech generation polishing and fine-tuning data, wherein the text speech generation polishing and fine-tuning data is used for the model's spoken language learning; Step S240: Obtain multimodal input samples and construct the pre-trained capability preservation dataset, wherein the multimodal input samples include at least visual input samples, text input samples, and speech input samples, and the pre-trained capability preservation dataset is used for learning the multimodal recognition capability of the model.

[0055] In this embodiment, Figure 3The process of constructing fine-tuning data is demonstrated. The initial data for instruction fine-tuning consists of five parts: pre-trained stories, multi-voice data, speech-to-text data, pre-trained instruction fine-tuning data, joint capability fine-tuning data, and text and multimodal capability preservation data. These five parts, after processing, constitute the instruction fine-tuning synthetic data. After instruction training on the fine-tuning dataset, the model can largely retain the original visual multimodal dialogue capabilities. Furthermore, with a relatively small amount of fine-tuning data, it simultaneously achieves good speech recognition, speech understanding, and speech paradigm generation capabilities. Through data organization and training, the model can support high-quality long speech dialogues and achieve dialogue generation with controllable emotional timbre.

[0056] Specifically, the above five parts of data can be optimized and summarized into four parts: speech understanding ability alignment data, fine-tuning data of multi-turn dialogue in speech interaction, text speech generation polishing fine-tuning data, and pre-training capability preservation dataset, which are used to form fine-tuning training data.

[0057] The first part is the speech understanding alignment data. This part contains data related to the model's recognition of speech content, enabling the model to effectively understand speech information and transcribe it into text for output. These datasets are mainly selected from the CommonVoice and GigaSpeech datasets. The CommonVoice dataset is an open-source speech dataset from Mozilla, collecting over 100,000 hours of speech, primarily using its English data, including audio and text transcription, used for training open-source ASR (Automatic Speech Recognition) and research on language diversity. The GigaSpeech dataset is a large-scale speech dataset from Microsoft, containing over 1,000 hours of English speech, covering scenarios such as meetings and conversations, with fine annotations, used for training ASR models in complex scenarios. The prompts for this part of the data are the Chinese and English instructions to "convert speech content into text content".

[0058] The second part consists of fine-tuning data for multi-turn dialogues in voice interaction. This part includes multi-turn dialogue data collected and constructed during the pre-training phase. All data except the TinyStories dataset can be used in this stage. This dataset will undergo additional data augmentation to diversify its format. The Mutual dataset and the dataset constructed and refined using the GPT interface for the large model, due to their more colloquial text content, will only be used to construct the speech output data, not the text output data, to ensure the model can adapt to diverse inputs and to ensure that only the speech output content is subject to colloquial constraints without changing the model's original text output format. Therefore, these datasets will be constructed in two ways.

[0059] The first type of data is text input and audio output data. This type of data will generate corresponding voice responses based on the user's text commands. The corresponding prompt words are "Generate a voice response with the specified speaker's timbre based on the text content" in both Chinese and English.

[0060] The second type of data is voice input and voice output data. The model aligns voice and text information through the voice understanding encoder, enabling the model to effectively fuse the contextual content of voice features, and then obtain the corresponding voice response. The corresponding prompt words are the Chinese and English instructions to "listen to the voice content and generate the corresponding voice response with the specified speaker's timbre".

[0061] The third part is the text-to-speech generation polishing and fine-tuning data. This data is processed only into text input and speech output form to make the text corresponding to the model's speech output more conversational. It also enhances the breadth and accuracy of the conversational output content, ensuring the model maintains its output paradigm and avoids generating special characters or complex sentences that don't fit into spoken conversation. This ensures the accuracy of output segmentation and the robustness of the final speech synthesis module. The organization of this data is consistent with the format and prompts of the Chinese text input speech output in the second part of the data. The data comes from the VoiceAssistant dataset built using the mini-omni framework, which contains over 400,000 cleaned daily dialogues.

[0062] The fourth part is the pre-training capability preservation dataset. Part of this data is used to preserve the capabilities of the pre-trained model. This part input integrates the various parts of the pre-training data and organizes them into multiple subsets to ensure the quality of the generated results. The specific subsets are shown below.

[0063] The first subset is the text-to-speech capability preservation dataset. This data is used to maintain the model's text-to-speech capability. It is mainly extracted from the pre-training data and transformed into dialogue format. The corresponding prompts are Chinese and English instructions such as "Generate the corresponding speech form according to the text content and specified voice or speech parameters".

[0064] The second subset is the speech repetition dataset, used to recognize speech content and output it as speech, enhancing the model's intuitive capabilities in both understanding and generation. As above, this dataset is selected from the pre-training dataset, and its prompts are in Chinese and English: "Repeat the speech content and generate speech with the same content according to the specified sound or speech parameters."

[0065] The third sub-data set is the text and visual capability preservation dataset. The function of this dataset is to ensure that the model retains good multimodal visual capabilities and text dialogue capabilities. It includes multiple text command datasets and image-text command question-and-answer datasets.

[0066] The fourth subset is a multimodal emotion recognition dataset, primarily using the emova-sft-speech-231k dataset. This dataset is used to train the model's emotion generation capabilities, enabling the model to efficiently generate emotional speech content. The corresponding prompts are in Chinese and English: "Listen to the speech content and generate a corresponding speech response with the specified emotion, timbre, and pitch."

[0067] Step S300: Based on the pre-trained data, construct a basic model and perform pre-training, wherein the basic model is used to convert the input text token into a speech token.

[0068] In this embodiment, pre-training is divided into multiple stages. The first stage uses single-timbre data to activate the model's bilingual (Chinese and English) multilingual configuration generation capability, achieving realistic single-timbre speech generation. At this point, the model already possesses relatively complete text-to-speech capabilities, but the autoregressive module's understanding ability can further process more types of audio data. Therefore, in the second stage, female voice timbre data and various characteristic timbre data generated based on text instruction descriptions are further introduced, including discretized natural language descriptions for emotion (e.g., emotion: ..., pitch: ...) and natural language timbre description data. This not only enables the model to generate characteristic human voices but also further enhances the robustness of single-timbre synthesis through different timbres.

[0069] Training data organization methods such as Figure 4 The upper part is divided into three parts by special symbols: the first part is the text prompt words, which describe the features of the speech content; the second part is the speech prefix, which contains the timbre information of the previous sentence (speech token) to ensure timbre consistency; the third part is the text to be transcribed, which serves as the main context information. The model will convert it from text symbols to speech symbols through an autoregressive generation method.

[0070] In one implementation, constructing a base model and performing pre-training based on the pre-training data specifically includes the following steps: Step S310: Input the text token, the preceding voice token, and the preset prompt words based on the pre-trained data into the pre-constructed speech autoregressive module of the base model to obtain the predicted voice token. The speech autoregressive module includes 24 Transformer blocks. Step S320: Output the predicted speech token to a 4096-dimensional prediction head to obtain the predicted synthesized speech; Step S330: Iteratively calculate the loss function and backpropagate to optimize the parameters of the base model.

[0071] In this embodiment, the specific goal in the first stage of training is to train a model that generates speech tokens from text tokens, possessing basic text-to-speech capabilities. Specifically, the word embeddings of the text are obtained using the vocabulary mapping of the large model Qwen, which contains sufficient textual information. This textual information, after passing through a projection layer, is input into a speech autoregressive module initialized by the text model. The input to this module includes the text content to be converted to speech, cue words indicating the speech conversion style, and speech tokens input alongside the cue words. These speech tokens are used to ensure the consistency of the generated timbre. After the input data is processed by the 24-layer Transformer block of the autoregressive module, the model predicts subsequent speech tokens based on the preceding information and outputs them to a 4096-dimensional prediction head. Because the speech codec Wavtokenizer can encode and decode more complex acoustic information, combined with the more comprehension-enhancing autoregressive module, the model can effectively understand and generate various parasemantic information of the speech, thereby achieving more controllable speech generation.

[0072] In one implementation, when the base model generates a speech response, it generates synthesized speech of arbitrary length through block synthesis, specifically including the following steps: Step S340: Divide the input text corresponding to the speech to be generated into several semantically complete text blocks, with the period as the unique truncation position; Step S350: According to the order of the text blocks, input the text token corresponding to the first text block into the pre-trained base model, and generate the speech token and synthesized speech segment corresponding to the first text block through the speech autoregression module; Step S360: Use the voice token corresponding to the text block as a voice information prompt, and input it together with the text token of the next text block into the basic model to generate the corresponding voice token and synthesized voice segment; Step S370: Generate synthesized speech segments corresponding to all text blocks in the order of the text blocks, and concatenate all synthesized speech segments to obtain synthesized speech of arbitrary length.

[0073] In this embodiment, to address the limitation on speech length generation in open-source large language model frameworks such as LLaMA-Omni and Freeze-Omni, a method is adopted to segment the text into text blocks of similar length for speech synthesis. Foreground speech information is used to prompt subsequent generation, ensuring speech coherence. Specifically, only periods are used as truncation points for text blocks. This approach allows the model to generate speech responses of arbitrary length, unrestricted by model architecture, while ensuring the accuracy and robustness of long speech at any position.

[0074] Step S400: Based on the pre-trained base model, construct a large multimodal speech interaction model, and use the fine-tuned training data to train the large multimodal speech interaction model. The large multimodal speech interaction model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs, and output the speech responses.

[0075] In this embodiment, speech understanding is a core component of the speech dialogue model, and the completeness of its functionality directly affects the model's depth of understanding of user intent. Current mainstream models, when processing speech input, typically focus on extracting and compiling semantic information, while paying relatively little attention to prosodic features such as intonation and timbre. Based on the QwenAudio architecture, a large-scale audio language model, this invention enhances the encoder's representational capabilities and constructs a unified audio information processing framework, enabling the model to simultaneously capture semantic and prosodic features, achieving multi-dimensional perception of speech input.

[0076] Specifically, Whisper-large-v3 encoder is used as the speech encoder to construct the speech understanding alignment module of the multimodal large model. Experiments show that modifying the Whisper decoder to a Qformer and employing a mapping strategy of 400 tags per minute can efficiently compress speech information while preserving rich paralinguistic features, such as timbre, intonation variations, and emotional cues. These features, after being mapped to the text semantic space through a projection layer, can be effectively utilized by the model, significantly improving multimodal reasoning capabilities. The training process is as follows: Figure 3 The lower half indicates.

[0077] In one implementation, the construction of a large-scale multimodal voice interaction model based on the pre-trained base model, and the training of the large-scale multimodal voice interaction model using the fine-tuned training data, specifically includes the following steps: Step S410: Modify the Whisper decoder using the Qformer architecture to construct a large-scale multimodal voice interaction model. The structural expression of the large-scale multimodal voice interaction model is as follows:

[0078]

[0079]

[0080]

[0081]

[0082] in, For fixed-length query vectors, For vectors, For hidden layer output, For audio, For query, It constitutes a fixed-length query vector The Hidden layer output components It is the top-level output of the pre-trained audio encoder, Whisper encoder. For the multi-head self-attention layer, For multi-head cross-attention layer, For layer normalization, It is a multilayer perceptron. This is the corresponding output of the Whisper encoder. for The corresponding output content, for The corresponding output content, For fixed-length query vectors The multi-head self-attention computation performed, Represents the total number of initial vectors This represents the output of the cross-attention module, used to extract the main content of the input audio. The output of the cross-attention module The hidden layer output after processing by a multilayer perceptron is then subjected to four identical operations, and a learnable linear layer is applied to project the final output onto the representation space of a large language model. Step S420: Use the fine-tuned training data to train the multimodal voice interaction large model.

[0083] In this embodiment, in this architecture, the Whisper encoder encodes every 30 seconds of audio as... Each audio tag is assigned an audio tag, and the output of the cross-attention module is responsible for extracting the core content of the input audio. After passing through all layers of the Whisper decoder, the final output is projected into the representation space of the multimodal large language model through a learnable linear layer.

[0084] Compared to other speech understanding alignment methods, this architecture has three advantages. First, the Qformer architecture is advanced in design. Second, it uses a pre-trained Whisper architecture to initialize the encoder, which can efficiently extract audio information. Finally, it uses a pre-trained Whisper decoder to initialize the Qformer, which can significantly improve the speed and accuracy of the model's speech understanding alignment.

[0085] In terms of model output design, standardized generation of speech-to-text was achieved by constructing a specific dataset and introducing special tokens. Specifically, a fine-tuning dataset was built to activate the model's ability to standardize speech output in different scenarios. Simultaneously, multiple special tokens were introduced to separate different types of output information. <sosp>The text begins with a speech marker, followed by speech control information. Monotonic speech control information includes the specific language and speaker. In actual data construction, three types of voices are mainly used to achieve diversified control: male voices in multiple languages, female voices adapted to English, and female voices adapted to Chinese. Multi-voice, multi-timbre control information uses a description refined by a large model. <eop>Mark the end of the distinctive human voice description, followed by the actual text content to be converted into speech. <eosp>This serves as the end marker for speech generation; content outside this marker is not involved in speech conversion.

[0086] After training with the aforementioned fine-tuned dataset, the model possesses four core capabilities, including basic multimodal text-to-image interaction capabilities, end-to-end communication capabilities with voice input and output, hybrid communication capabilities with flexible switching between voice and text, and emotion recognition and precise control and generation of specific timbre.

[0087] Therefore, the innovative text-to-speech token alignment mechanism provided in this embodiment achieves efficient processing of text-to-speech synthesis training by constructing a dynamic mapping algorithm and an adaptive encoding framework. This method overcomes the limitations of traditional synthesis techniques in terms of length and timbre, not only accurately processing text input of arbitrary length to generate semantically coherent and rhythmically natural speech sequences, but also supporting distinctive speech synthesis with diverse timbres, emotions, and styles. While ensuring high fidelity in single-timbre synthesis, it achieves fine-grained control over speech acoustic features by introducing a conditional generation model and a cross-modal attention mechanism, effectively improving the flexibility and expressiveness of distinctive audio generation.

[0088] Meanwhile, addressing the resource bottleneck issue in transferring the voice interaction capabilities of large multimodal models, this embodiment provides a fast alignment method with low resource consumption and a corresponding dataset. By deeply mining the semantic relationships between multimodal data and combining contrastive learning and transfer learning techniques, a high-quality fine-tuning dataset containing text, speech, and emotion tags is constructed. This approach can quickly activate the speech understanding and generation potential of large multimodal models, endowing the model with emotional perception and expressive voice dialogue capabilities while significantly reducing computational resources and training time, thus significantly improving the model's adaptability and versatility in multimodal interaction scenarios.

[0089] Furthermore, with richer and more comprehensive datasets and more abundant resources, the understanding and generation of models can be further expanded, extending the single speech audio modality to a wider range of aspects such as ambient sound, music, and medical sound waves, thereby enabling the model to have broader and more comprehensive audio capabilities.

[0090] Based on the above embodiments, there are also variations with less satisfactory results. Specifically, a strategy that merges the two training phases into a single phase can be employed. However, this approach suffers from dual bottlenecks in resource utilization efficiency and model adaptability. This method directly discards the parameters and feature representations accumulated in the first phase of training, failing to effectively reuse previous training results, leading to redundant consumption of computational resources and a significant reduction in training efficiency. Furthermore, when the base model is updated and iterated, the lack of a modular training structure necessitates retraining the entire model from scratch, greatly increasing the cost of model transfer and optimization, and severely restricting the model's scalability and rapid adaptability.

[0091] Another less effective variation employs a fully end-to-end alignment training mode at the network level. While aiming to achieve direct mapping and collaborative optimization of multimodal information, this approach suffers from uncertainties in performance improvement. This method trains the model holistically during the final alignment stage, potentially leading to conflicts in gradient updates and imbalances in parameter adjustments. This makes it difficult for the model to accurately capture key semantic relationships between modalities during alignment. Compared to token-based direct mapping methods, this global training approach may weaken the model's alignment capabilities due to over-coupling, failing to fully leverage the strengths of each modality and consequently impacting the model's performance and generalization ability in speech dialogue tasks.

[0092] In summary, under the technical solutions of the above embodiments, this invention focuses on performance optimization and multimodal adaptation of voice dialogue systems, achieving multi-dimensional breakthroughs through innovative technical solutions. At the level of long text output, this invention proposes an efficient block synthesis strategy, effectively overcoming the problems of information loss and insufficient coherence in traditional voice dialogue models when processing long texts, and generating logically clear and semantically complete long text content. In the field of audio synthesis, this invention constructs a refined emotion and acoustic parameter control model. This model, based on deep neural networks, can accurately analyze users' diverse needs for emotion, timbre, pitch, and speech rate, and through advanced audio synthesis technology, maps these parameters into high-quality speech output, achieving a personalized and immersive voice interaction experience. Regarding multimodal model adaptation, this invention constructs a set of efficient fine-tuning data, enabling large multimodal models based on different frameworks to quickly integrate into the voice dialogue architecture proposed in this invention. While ensuring efficient adaptation, it maximizes the retention of the original model's core capabilities in text and multimodal dialogue tasks, significantly improving the system's versatility and scalability.

[0093] like Figure 5 As shown in the figure, this embodiment of the invention provides a multimodal speech interaction large model training system based on speech acoustic feature modulation. The system includes: a pre-training data construction module 10, a fine-tuning training data construction module 20, a basic model construction and pre-training module 30, and a multimodal speech interaction large model construction and training module 40.

[0094] Specifically, the pre-training data construction module 10 is used to obtain text tokens of text training samples, and based on the text tokens, construct speech tokens containing semantic features and speech acoustic features expressed by the text tokens, to obtain pre-training data for converting text tokens into speech tokens; the fine-tuning training data construction module 20 is used to construct fine-tuning training data for speech understanding and dialogue generation based on multimodal input samples and the pre-training data, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations; the basic model construction and pre-training module 30 is used to construct a basic model and perform pre-training based on the pre-training data, wherein the basic model is used to convert input text tokens into speech tokens; the multimodal speech interaction large model construction and training module 40 is used to construct a multimodal speech interaction large model based on the pre-trained basic model, and use the fine-tuning training data to train the multimodal speech interaction large model, wherein the multimodal speech interaction large model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs, and output the speech responses.

[0095] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 6 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal speech interaction large-scale model training method based on speech acoustic feature modulation. The display screen can be an LCD screen or an e-ink screen. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0096] Those skilled in the art will understand that Figure 6 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment, a terminal device is provided, including a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Obtain the text token of the text training sample, and construct a speech token containing the semantic features and speech acoustic features expressed by the text token based on the text token, so as to obtain pre-training data for converting the text token into a speech token; Based on the multimodal input samples and the pre-training data, fine-tuning training data for speech understanding and dialogue generation is constructed, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations; Based on the pre-trained data, a basic model is constructed and pre-trained, wherein the basic model is used to convert the input text token into a speech token; Based on the pre-trained base model, a large multimodal speech interaction model is constructed, and the fine-tuned training data is used to train the large multimodal speech interaction model. The large multimodal speech interaction model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs and output the speech responses.

[0098] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0099] In summary, this invention discloses a training method, system, terminal device, and medium for a large-scale multimodal speech interaction model based on speech acoustic feature modulation, relating to the field of multimodal speech interaction technology. The method includes: first, acquiring text tokens from text training samples; and based on these text tokens, constructing speech tokens containing semantic features and speech acoustic features expressed by the text tokens, obtaining pre-training data for converting text tokens into speech tokens. Next, based on multimodal input samples and the pre-training data, constructing fine-tuning training data for speech understanding and dialogue generation, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations. Following this, based on the pre-training data, constructing a basic model and performing pre-training, wherein the basic model is used to convert input text tokens into speech tokens. Finally, based on the pre-trained basic model, constructing a large-scale multimodal speech interaction model, and using the fine-tuning training data to train the large-scale multimodal speech interaction model, wherein the large-scale multimodal speech interaction model is used to modulate speech acoustic features for generating speech responses based on multimodal inputs and output the speech responses. This invention addresses the problems of insufficient speech diversity, difficulty in generating long speech, and high resource consumption in multimodal adaptation of existing models. It realizes acoustic features such as timbre and emotion, supports controllable and coherent generation of long speech, quickly activates the model's speech interaction capabilities, and retains the core multimodal capabilities, thereby improving the naturalness and generalization of speech interaction in large multimodal models.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / eosp> < / eop> < / sosp>

Claims

1. A method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation, characterized in that, The method includes: Obtain the text token of the text training sample, and construct a speech token containing the semantic features and speech acoustic features expressed by the text token based on the text token, so as to obtain pre-training data for converting the text token into a speech token; Based on the multimodal input samples and the pre-training data, fine-tuning training data for speech understanding and dialogue generation is constructed, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations; Based on the pre-trained data, a basic model is constructed and pre-trained, wherein the basic model is used to convert the input text token into a speech token; Based on the pre-trained base model, a large multimodal speech interaction model is constructed, and the fine-tuned training data is used to train the large multimodal speech interaction model. The large multimodal speech interaction model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs and output the speech responses.

2. The method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation according to claim 1, characterized in that, The process of obtaining text tokens from text training samples and constructing speech tokens containing semantic features and acoustic features expressed by the text tokens to obtain pre-training data for converting text tokens into speech tokens includes: Acquire initial training data for single voice, which includes speech format dialogue plain text data and long text dataset. The speech format dialogue plain text data includes a filtered general natural dialogue text dataset and a reconstructed dialogue text dataset. The reconstructed dialogue text dataset is obtained by reconstructing the plain text instruction fine-tuning data based on preset prompt words using a general large model. Based on preset rules, special characters in the response round text content of the voice format dialogue plain text data and special characters in the long text dataset are filtered out, and symbol partitioning is used for segmentation to obtain preprocessed single voice training initial data. The initial training data for the single voice is converted into a token to obtain the text token, and the semantic features expressed by the text token are extracted. Speech synthesis is performed on the initial training data of the single voice to generate matching single voice speech samples, and speech acoustic features are extracted from the single voice speech samples; The speech token is obtained by associating the text token, the semantic features expressed by the text token, and the speech acoustic features.

3. The method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation according to claim 2, characterized in that, The process of obtaining text tokens from text training samples and constructing speech tokens containing semantic features and speech acoustic features expressed by the text tokens to obtain pre-training data for converting text tokens into speech tokens also includes: Acquire initial training data for multi-voice dialogue, which includes dialogue text, corresponding multi-voice speech data, and corresponding speech acoustic feature annotations. The initial training data for the multi-person voice is cleaned and tokenized to obtain the text token, and the semantic features expressed by the text token are extracted. The speech token is obtained by associating the text token, the semantic features expressed by the text token, and the speech acoustic feature annotations.

4. The method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation according to claim 3, characterized in that, The fine-tuning training data includes speech understanding capability alignment data, fine-tuning data from multi-turn speech interaction dialogues, text-to-speech generation polishing fine-tuning data, and a pre-training capability-preserving dataset. Based on multimodal input samples and the pre-training data, fine-tuning training data for speech understanding and dialogue generation is constructed. This fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations, including: Acquire speech-to-text sample data, perform speech and semantic annotation on the speech-to-text sample data, and construct the speech understanding ability alignment data based on the speech-to-text sample data and the corresponding speech and semantic annotations, wherein the speech understanding ability alignment data is used for learning the speech understanding ability of the model. Based on the multi-turn dialogue data in the pre-training data, the context association logic of the multi-turn dialogue data is obtained, and based on the multi-turn dialogue data and the corresponding context association logic, the fine-tuning data of the voice interaction multi-turn dialogue is constructed. The voice interaction multi-turn dialogue includes dialogue generation logic annotation, and the fine-tuning data of the voice interaction multi-turn dialogue is used for learning the context association logic of the model. Obtain spoken language output sample data and construct the text speech generation polishing and fine-tuning data, wherein the text speech generation polishing and fine-tuning data is used for the model's spoken language learning; Obtain multimodal input samples and construct the pre-trained capability-preserving dataset, wherein the multimodal input samples include at least visual input samples, text input samples, and speech input samples, and the pre-trained capability-preserving dataset is used to learn the multimodal recognition capability of the model.

5. The method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation according to claim 1, characterized in that, The step of constructing a base model and pre-training it based on the pre-trained data includes: The text token, the preceding voice token, and the preset prompt words based on the pre-trained data are input into the pre-constructed speech autoregressive module of the base model to obtain the predicted voice token. The speech autoregressive module includes 24 Transformer blocks. The predicted speech token is output to a 4096-dimensional prediction head to obtain the predicted synthesized speech; The loss function is calculated iteratively, and the parameters of the base model are optimized through backpropagation.

6. The method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation according to claim 5, characterized in that, When the base model generates a speech response, it generates synthesized speech of arbitrary length through block synthesis, including: The input text corresponding to the speech to be generated is divided into several semantically complete text blocks, with a period as the unique truncation position. According to the order of the text blocks, the text token corresponding to the first text block is input into the pre-trained base model, and the speech token corresponding to the first text block and the synthesized speech segment are generated through the speech autoregression module. The voice token corresponding to the text block is used as a voice information prompt, and together with the text token of the next text block, it is input into the basic model to generate the corresponding voice token and synthesized voice segment; According to the order of the text blocks, generate the synthesized speech segments corresponding to all text blocks, and concatenate all the synthesized speech segments to obtain synthesized speech of arbitrary length.

7. The method for training a large-scale multimodal speech interaction model based on speech acoustic feature modulation according to claim 1, characterized in that, The process of constructing a large-scale multimodal speech interaction model based on the pre-trained base model, and training the large-scale multimodal speech interaction model using the fine-tuned training data, includes: The Whisper decoder is modified using the Qformer architecture to construct a large-scale multimodal voice interaction model. The structural expression of the large-scale multimodal voice interaction model is as follows: in, For fixed-length query vectors, For vectors, For hidden layer output, For audio, For query, It constitutes a fixed-length query vector The Hidden layer output components It is the top-level output of the pre-trained audio encoder Whisperencoder. For the multi-head self-attention layer, For multi-head cross-attention layer, For layer normalization, It is a multilayer perceptron. This is the corresponding output of the Whisper encoder. for The corresponding output content, for The corresponding output content, For fixed-length query vectors The multi-head self-attention computation performed, This represents the total number of initial vectors. This represents the output of the cross-attention module, used to extract the main content of the input audio. The output of the cross-attention module The hidden layer output after processing by a multilayer perceptron is then subjected to four identical operations, and a learnable linear layer is applied to project the final output onto the representation space of a large language model. The fine-tuned training data is used to train the large multimodal voice interaction model.

8. A training system for a large-scale multimodal speech interaction model based on speech acoustic feature modulation, characterized in that, The system includes: The pre-training data construction module is used to obtain text tokens of text training samples, and construct speech tokens containing semantic features and speech acoustic features expressed by the text tokens based on the text tokens, so as to obtain pre-training data for converting text tokens into speech tokens. The fine-tuning training data construction module is used to construct fine-tuning training data for speech understanding and dialogue generation based on multimodal input samples and the pre-training data, wherein the fine-tuning training data includes speech semantic understanding annotations and dialogue generation logic annotations. The basic model construction and pre-training module is used to construct a basic model and perform pre-training based on the pre-training data, wherein the basic model is used to convert the input text token into a speech token. The multimodal speech interaction large model construction and training module is used to construct a multimodal speech interaction large model based on the pre-trained base model, and to train the multimodal speech interaction large model using the fine-tuned training data. The multimodal speech interaction large model is used to adjust the speech acoustic features used to generate speech responses based on multimodal inputs, and to output the speech responses.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a multimodal speech interaction large model training program based on speech acoustic feature modulation stored in the memory and executable on the processor. When the processor executes the multimodal speech interaction large model training program based on speech acoustic feature modulation, it implements the steps of the multimodal speech interaction large model training method based on speech acoustic feature modulation as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a training program for a large multimodal speech interaction model based on speech acoustic feature modulation. When the training program is executed by a processor, it implements the steps of the training method for a large multimodal speech interaction model based on speech acoustic feature modulation as described in any one of claims 1-7.

Images