LLM-based text-to-speech dual-stream super-personalized speech generation method, device and equipment

Abstract

The application discloses a text-speech dual-flow super-personal speech generation method, device and equipment based on LLM, comprising the following steps: extracting a target latent acoustic representation of an input target speech prompt through a pre-trained speech encoder; mapping the target latent acoustic representation into a target acoustic embedding representation through a pre-trained projector module; inputting the target acoustic embedding representation and to-be-synthesized text into a pre-trained large language model to generate a plurality of target speech tokens; converting each target speech token into a corresponding target mel-frequency spectrum graph through a flow matching decoding module; and reconstructing the target mel-frequency spectrum graph based on probability density estimation to obtain a target speech. The application improves the accuracy and stability of speech generation, thereby improving the speech interaction experience of users, and can be widely applied in the technical field of speech synthesis.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a method, apparatus and device for generating hyperhuman speech based on LLM dual-stream text and speech. Background Technology

[0002] In human communication, in addition to explicit semantic content, there is a large amount of super-semantic information (Beyond-Semantic Speech, BoSS) transmitted through speech carriers. This includes paralinguistic features such as the speaker's emotional attitude, rhythm and stress, dynamics of the dialogue context (e.g., pauses, interruptions, contextual cues from background noise), and implicit intentions expressed through tone of voice. This super-semantic information plays a decisive role in the listener's judgment of dialogue intentions, emotional responses, and choice of interaction strategies, and is therefore a key element for high-quality human-computer speech interaction systems.

[0003] In recent years, the architecture of voice interaction systems has been evolving from the traditional ASR→LLM→TTS cascade paradigm to an end-to-end integrated model centered on a large-scale pre-trained text language model. Within this emerging architecture, research has expanded from simply pursuing "semantic intelligibility" to seeking higher-level capabilities such as "emotional interaction," "personalized expression," and "contextual sensitivity." However, building an end-to-end voice dialogue model with dialect understanding and emotional cognition capabilities often requires acquiring large-scale, finely annotated speech-text pair datasets covering multiple dialects and emotional states, resulting in extremely high costs for data collection and manual annotation. Given the relatively abundant text data and the rapid development of LLM-driven speech synthesis architectures, generative data augmentation can effectively alleviate this data scarcity challenge.

[0004] Specifically, in the field of speech generation, inspired by breakthroughs in text language modeling technology, numerous works in industry and academia have demonstrated that decoder-only LLMs possess strong instruction-following capabilities. Given voice prompts, they can achieve multi-dimensional control of speech features, including speaker features, ambient sound, emotional expression, and semantic content. However, this method still falls short in understanding complex contexts. Recent works such as OuteTTS, CosyVoice2, and Llasa have made significant explorations in model architecture, attempting to directly use pre-trained text language models as the base model and integrate speech tokens into the next-token prediction generation paradigm, achieving state-of-the-art speech generation results. These works reveal the potential of large-scale pre-trained models in cross-modal tasks, demonstrating their ability to effectively improve the understanding of input text during speech generation.

[0005] However, the above solutions still face many challenges in practical applications. When voice prompts are compressed and encoded using discrete tokens, acoustic features are often lost. Furthermore, the strong coupling between voice prompts and corresponding text, as well as the significant degradation of the original text understanding capabilities of language models, limit the performance of these technologies in real-world industry scenarios. These bottlenecks mean that the actual performance of existing solutions in practical applications still falls short of expectations.

[0006] The above problems urgently need to be addressed. Summary of the Invention

[0007] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0008] Therefore, one objective of this invention is to provide a text-to-speech dual-stream hyperhuman speech generation method based on LLM, which improves the accuracy and stability of speech generation, thereby enhancing the user's voice interaction experience.

[0009] Another objective of this invention is to provide a text-to-speech dual-stream hyperhuman speech generation device based on LLM.

[0010] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a method for generating hyperhuman-like speech using LLM-based dual-stream text and speech, comprising the following steps: Extract the target latent acoustic representation of the input target speech cues using a pre-trained speech encoder; The target latent acoustic representation is mapped to a target acoustic embedding representation through a pre-trained projector module; The target acoustic embedding representation and the text to be synthesized are input into a pre-trained large language model to generate multiple target speech tokens; The target speech tokens are converted into corresponding target Mel spectrograms by the stream matching decoding module, and the target speech is obtained by reconstructing the target Mel spectrograms based on probability density estimation.

[0011] Furthermore, in one embodiment of the present invention, the speech encoder, the projector module, and the large language model are pre-trained through the following steps: Construct multiple speech-text continuation pairs based on acoustic and / or semantic driving; With the parameters of the large language model fixed, modal alignment training is performed on the initialized speech encoder and projector module based on the speech-text continuation pair data to obtain the pre-trained speech encoder and projector module. Multiple speech generation training data are constructed based on quadruples; The parameters of the speech encoder and the projector module are fixed, and the underlying parameters of the large language model are also fixed. The large language model is then trained to generate speech based on the speech generation training data to optimize the top-level parameters, thereby obtaining a pre-trained large language model.

[0012] Furthermore, in one embodiment of the present invention, the construction of multiple speech-text continuation pairs based on acoustic driving and / or semantic driving specifically includes: Multiple first speech samples and corresponding first transcribed texts are obtained from the speech recognition corpus. A corresponding first continuation text is generated based on the first transcribed text using a prompting language model. The speech-text continuation pair data is generated based on the first speech samples and the first continuation text. And / or, Multiple first text phrases are obtained, and the first text phrases are converted into second speech samples through a speech synthesis model. Then, a corresponding second continuation text is generated based on the first text phrases through a prompting language model. Finally, the speech-text continuation pair data is generated based on the second speech samples and the second continuation text.

[0013] Furthermore, in one embodiment of the present invention, the large language model includes a decision module, a text generation module, and a speech synthesis module. The modal alignment training of the speech encoder and projector module initialized based on the speech-text continuation pair data specifically includes: The speech-text continuation pair data is divided into a first training dataset and a second training dataset according to the language type. The first training dataset includes multiple first speech prompts and corresponding continuation text labels, and the second training dataset includes multiple second speech prompts and corresponding continuation text labels. The first acoustic embedding representation of the first voice prompt is extracted by the speech encoder and the projector module, and the corresponding first text token is generated by the decision module and the text generation module based on the first acoustic embedding representation. A first text loss value is determined based on the first text token and the corresponding continuation text tag, and the parameters of the projector module are updated based on the first text loss value. The second acoustic embedding representation of the second voice prompt is extracted by the speech encoder and the projector module, and the corresponding second text token is generated by the decision module and the text generation module based on the second acoustic embedding representation. The second text loss value is determined based on the second text token and the corresponding continuation text tag, and the parameters of the speech encoder and the projector module are updated based on the second text loss value.

[0014] Furthermore, in one embodiment of the present invention, the construction of multiple speech generation training data based on quadruples specifically includes: Acquire text-to-speech input data and text-to-speech output data from historical speech generation scenarios; Based on the text-to-speech input data, determine text query samples and speech query samples, and based on the text-to-speech output data, determine the corresponding text response tags and speech response tags; The speech generation training data is constructed based on the text query samples, the speech query samples, the text response tags, and the speech response tags.

[0015] Furthermore, in one embodiment of the present invention, the large language model includes a decision module, a text generation module, and a speech synthesis module. The step of training the large language model to optimize top-level parameters based on the speech generation training data specifically includes: The speech generation training data is divided into a third training dataset and a fourth training dataset according to the language type. The third training dataset includes multiple first text query samples and corresponding text response labels and speech response labels. The fourth training dataset includes multiple first speech query samples and corresponding text response labels and speech response labels. The decision-making module, the text generation module, and the speech synthesis module generate a corresponding first speech token based on the first text query sample. A first speech loss value is determined based on the first speech token and the corresponding speech response tag, and the parameters of the speech synthesis module are updated based on the first speech loss value. The speech encoder and the projector module extract the third acoustic embedding representation of the first speech query sample, and the decision module, the text generation module and the speech synthesis module generate the corresponding second speech token based on the third acoustic embedding representation. The second speech loss value is determined based on the second speech token and the corresponding speech response tag, and the parameters of the speech synthesis module are updated based on the second speech loss value.

[0016] Furthermore, in one embodiment of the present invention, the speech synthesis module generates speech tokens using a multi-token prediction mechanism and updates parameters using an automatic gradient masking strategy based on prediction confidence. The method of generating voice tokens using a multi-token prediction mechanism is as follows: In each step, multiple subsequent speech tokens are predicted in parallel. The output token of the current step is concatenated with the input features of the next step as the conditional input of the speech synthesis module to guide the iterative generation of speech tokens. The parameter update using an automatic gradient masking strategy based on prediction confidence is specifically as follows: During parameter updates, the corresponding gradient contribution is determined based on the prediction confidence of each speech token, so that full gradient updates are applied to high-confidence samples and full masking is applied to low-confidence samples.

[0017] Furthermore, in one embodiment of the present invention, the large language model is post-training optimized through the following steps: Obtain preset voice prompt samples, synthesized text samples, and corresponding target text labels, and determine the reward model corresponding to different language types; The speech encoder, the projector module, and the large language model generate multiple candidate speech tokens based on the speech prompt samples and the synthesized text samples. The stream matching decoding module reconstructs each candidate speech token into a corresponding candidate audio, which is then input into the reward model for transcription to obtain the corresponding transcribed text. The reward score for each candidate speech token is determined based on the shortest edit distance between the transcribed text and the target text tag; The advantage function is calculated based on the reward score, and the large language model is fine-tuned and optimized based on the advantage function.

[0018] On the other hand, embodiments of the present invention provide an LLM-based dual-stream hyperhuman speech generation device, comprising: The acoustic representation extraction module is used to extract the target latent acoustic representation of the input target speech cues through a pre-trained speech encoder; An acoustic representation mapping module is used to map the target latent acoustic representation into a target acoustic embedding representation through a pre-trained projector module; The speech token generation module is used to input the target acoustic embedding representation and the text to be synthesized into a pre-trained large language model to generate multiple target speech tokens; The speech reconstruction module is used to convert each target speech token into a corresponding target Mel spectrogram through the stream matching decoding module, and reconstruct the target Mel spectrogram based on probability density estimation to obtain the target speech.

[0019] On the other hand, embodiments of the present invention provide an electronic device, the electronic device including a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein the computer program, when executed by the processor, implements the LLM-based dual-stream hyper-human speech generation method as described above.

[0020] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the LLM-based dual-stream hyper-human speech generation method as described above.

[0021] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the LLM-based dual-stream hyper-human speech generation method as described above.

[0022] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention extracts the latent acoustic representation of the target speech prompt from the input speech using a pre-trained speech encoder. A pre-trained projector module maps this latent acoustic representation to a target acoustic embedding representation. The target acoustic embedding representation and the text to be synthesized are then input into a pre-trained large language model to generate multiple target speech tokens. A stream matching decoding module converts each target speech token into a corresponding target Mel spectrogram. Based on probability density estimation, the target Mel spectrogram is reconstructed to obtain the target speech. This invention, through a pre-trained speech encoder and projector module, effectively associates linguistic features, prosodic information, and emotional cues with semantic text, achieving cross-modal alignment between speech and text. The pre-trained large language model maintains the ability to understand complex contexts during speech generation and supports real-time streaming speech synthesis, meeting diverse application needs and improving the accuracy and stability of speech generation, thereby enhancing the user's voice interaction experience. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the steps of a text-to-speech dual-stream hyperhuman speech generation method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the steps of a pre-trained speech encoder, a projector module, and a large language model provided in an embodiment of the present invention; Figure 3 A flowchart of step S201 provided in an embodiment of the present invention; Figure 4 A flowchart of step S202 provided in an embodiment of the present invention; Figure 5 A flowchart of step S203 provided in an embodiment of the present invention; Figure 6 A flowchart of step S204 provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating a step-by-step process for post-training optimization of a large language model, as provided in an embodiment of the present invention. Figure 8 A schematic diagram of the structure of the LLM-based dual-stream hyper-human speech generation device provided in an embodiment of the present invention; Figure 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0027] The LLM-based hyper-human speech generation method provided in this application can be applied to terminals, servers, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a machine vision-based driver status monitoring method, but is not limited to the above forms.

[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0029] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0030] like Figure 1The diagram shown is a flowchart of one step of the LLM-based dual-stream hyperhuman speech generation method provided in an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a method for generating hyperhuman-like speech using LLM-based dual-stream text and speech, specifically including the following steps: S101. Extract the target latent acoustic representation of the input target speech cues through a pre-trained speech encoder; S102. The latent acoustic representation of the target is mapped to the target acoustic embedding representation through a pre-trained projector module; S103. Input the target acoustic embedding representation and the text to be synthesized into a pre-trained large language model to generate multiple target speech tokens; S104. The target speech token is converted into the corresponding target Mel spectrogram through the stream matching decoding module, and the target speech is reconstructed based on the probability density estimation.

[0031] Specifically, in this embodiment of the invention, the latent acoustic representation of the input speech prompt is first extracted using a pre-trained Whisper-Small speech encoder. Then, a projector module containing a three-layer CNN architecture and a linear transformation layer is used to map these latent features into an embedding representation compatible with a large language model. Subsequently, the large language model iteratively generates discrete speech tokens under the dual conditions of adapting to acoustic embedding and text input. Finally, the speech tokens are converted into Mel spectrograms using a stream matching model, and waveform synthesis is completed based on probability density estimation.

[0032] It can be recognized that the embodiments of the present invention, through a pre-trained speech encoder and projector module, achieve effective association between language features, prosodic information, emotional cues and semantic text, and complete cross-modal alignment of speech and text. Through a pre-trained large language model, it can maintain the ability to understand complex contexts during speech generation and supports real-time streaming speech synthesis, which can meet diverse application needs, improve the accuracy and stability of speech generation, and thus improve the user's voice interaction experience.

[0033] like Figure 2 The diagram shown is a flowchart illustrating the steps of a pre-trained speech encoder, projector module, and large language model provided in an embodiment of the present invention. (Refer to...) Figure 2 As an optional implementation, the speech encoder, projector module, and large language model are pre-trained through the following steps: S201. Construct multiple speech-text continuation pairs based on acoustic and / or semantic driving; S202. Fix the parameters of the large language model, and perform modal alignment training on the initialized speech encoder and projector module based on the speech-text continuation data to obtain the pre-trained speech encoder and projector module. S203. Construct multiple speech generation training data based on quadruples; S204. Fix the parameters of the speech encoder and projector modules, and fix the underlying parameters of the large language model. Train the large language model to generate speech based on the speech generation training data to optimize the top-level parameters and obtain the pre-trained large language model.

[0034] Specifically, the LLM-based dual-stream hyperhuman speech generation method (GOAT-TTS) of this invention trains the speech encoder, projector module, and large language model through modality alignment training, speech generation training, and post-training optimization mechanisms, ultimately achieving "being able to understand what is heard, understand deeply, and speak well." Details are as follows: 1) Modal Alignment Training: Unlike existing speech prompt encoding methods that rely on discrete labels, this invention utilizes a speech encoder and a projector module to extract continuous representations of the prompt speech. In this process, paralinguistic information such as timbre features, prosodic patterns, and emotional cues can be simultaneously captured and mapped to an embedding space compatible with semantic text representations, establishing a bidirectional correspondence between continuous acoustic embeddings and semantic representations. Through this deep alignment, the language model can directly parse speech prompts during the inference stage, reducing reliance on precise speech-to-text transcription and thus improving flexibility and adaptability in industrial applications.

[0035] 2) Speech Generation Training: To avoid disrupting the original knowledge structure and contextual understanding capabilities of the pre-trained language model, this invention employs a hierarchical parameter freezing method in the generation branch. Only the top layers of the pre-trained model are fine-tuned for task orientation, while the lower-level parameters remain frozen. This strategy achieves efficient optimization for speech generation tasks while maintaining the original semantic understanding capabilities. Furthermore, a multi-token prediction (MTP) method is introduced to mitigate the frequency differences between text and speech tokens over time. By predicting multiple speech tokens simultaneously at the decoding end, not only are the model inference steps reduced and latency shortened, but streaming input and output are also naturally supported, enabling the system to achieve real-time speech generation and enhancing compatibility and scalability in practical applications.

[0036] 3) Post-training optimization mechanism: After the model completes basic training, this invention introduces a post-training fine-tuning strategy based on GRPO (Group Relative Policy Optimization) to improve the performance of synthesized speech in terms of intelligibility and consistency with human preferences, thereby further improving the actual usability of the system.

[0037] The aforementioned architecture not only overcomes existing technical challenges, but also provides crucial underlying technical support for the comprehensive upgrade of the intelligent voice interaction experience.

[0038] like Figure 3 The diagram shown is a flowchart of step S201 provided in an embodiment of the present invention. (Refer to...) Figure 3 Furthermore, as an optional implementation, multiple speech-text continuation pairs are constructed based on acoustic and / or semantic driving, specifically including: S2011. Obtain multiple first speech samples and corresponding first transcribed texts based on the speech recognition corpus. Generate corresponding first continuation texts based on the first transcribed texts using a prompting language model. Generate speech-text continuation pair data based on the first speech samples and the first continuation texts. And / or, S2012. Obtain multiple first text phrases, convert the first text phrases into second speech samples through a speech synthesis model, and generate corresponding second continuation texts based on the first text phrases through a prompting language model. Then, generate speech-text continuation pair data based on the second speech samples and the second continuation texts.

[0039] Specifically, in existing large-scale speech generation models, prompts typically require discretization encoding. This process leads to the loss of some original acoustic features, including key information such as language, prosody, and emotion, resulting in a decrease in the natural prosody of the generated speech, unclear emotional expression, and insufficient sound quality. Furthermore, during model inference, users often need to provide both the speech and its corresponding text simultaneously. This strong dependency limits the system's deployment flexibility and reduces the model's adaptability and robustness to arbitrary speech prompts.

[0040] To overcome the above problems, this invention proposes a cross-modal alignment method based on speech-text continuation pairs. By introducing joint modeling of continuous acoustic embedding and semantic text representation during training, the model is able to understand speech and perform continuation. Specifically, this invention employs the following two types of data construction strategies: 1) Acoustic diversity-driven speech-semantic data construction: Utilizing a large-scale Automatic Speech Recognition (ASR) corpus (covering Mandarin, English, and multiple dialects), a prompting language model generates semantically coherent continuation text based on transcribed text. This approach establishes a correspondence between acoustic features in speech (such as prosody, timbre, and emotion) and semantic information in the text during training, enabling the model to maintain semantic coherence and preserve the acoustic properties of the speech prompts during text continuation.

[0041] 2) Semantic diversity-driven speech-semantic data construction: Text-to-speech (TTS) is used to convert semantically rich and structurally coherent short text sentences into speech. Simultaneously, a language model generates continuation content based on the corresponding text, thus constructing speech-semantic aligned data with semantic diversity and syntactic integrity. During this process, natural language descriptors are appended to the input text to annotate non-linguistic information such as dialect categories and emotional states, thereby explicitly guiding the model to learn and recognize these paralinguistic features.

[0042] Multiple speech-text continuation pairs can be constructed using at least one of the two data construction strategies mentioned above, thereby providing a data foundation for subsequent modality alignment training.

[0043] like Figure 4 The diagram shown is a flowchart of step S202 provided in an embodiment of the present invention. (Refer to...) Figure 4 As an optional implementation, the large language model includes a decision module, a text generation module, and a speech synthesis module. Based on the speech-text continuation data, the initialized speech encoder and projector module are trained for modal alignment, specifically including: S2021. Based on the language type, the speech-text continuation pair data is divided into a first training dataset and a second training dataset. The first training dataset includes multiple first speech prompts and corresponding continuation text labels, and the second training dataset includes multiple second speech prompts and corresponding continuation text labels. S2022. Extract the first acoustic embedding representation of the first voice prompt through the speech encoder and projector module, and generate the corresponding first text token based on the first acoustic embedding representation through the decision module and text generation module; S2023. Determine the first text loss value based on the first text token and the corresponding continuation text tag, and update the parameters of the projector module based on the first text loss value; S2024. Extract the second acoustic embedding representation of the second voice prompt through the speech encoder and projector module, and generate the corresponding second text token based on the second acoustic embedding representation through the decision module and text generation module; S2025. Determine the second text loss value based on the second text token and the corresponding continuation text label, and update the parameters of the speech encoder and projector module based on the second text loss value.

[0044] Specifically, at the training method level, cross-modal alignment is divided into two steps. The first step uses Mandarin and English corpora as the main data to train the Projector module to establish robust semantic parsing capabilities, and alleviates the training instability problem through gradient control in the optimization process. The second step is based on a balanced dataset covering multiple languages, dialects and sentiment categories to further optimize the speech encoder and Projector module so that they achieve unified alignment in the semantic representation space.

[0045] Furthermore, throughout the cross-modal alignment process, the parameters of the pre-trained language model remain frozen, with only the speech-side modules updated. This ensures that the original semantic understanding capabilities are not compromised when acoustic conditions are introduced. This method integrates multilingual, multi-dialect, and emotional information into a unified embedding space, enabling the model to directly parse cue speech using continuous acoustic embeddings, rather than relying on discretized tokens and precise transcription. This preserves more original acoustic features and significantly improves the flexibility and generalization ability for practical applications.

[0046] Specifically, based on language type, the speech-text continuation pair data is divided into a first training dataset (mainly composed of Mandarin and English corpora) and a second training dataset (covering multiple languages, dialects, and sentiment categories). The first training dataset includes multiple first speech prompts and corresponding continuation text labels, while the second training dataset includes multiple second speech prompts and corresponding continuation text labels. The first acoustic embedding representation of the first speech prompt is extracted using a speech encoder and projector module. A decision module and text generation module generate the corresponding first text token based on the first acoustic embedding representation. The first text loss value is determined based on the first text token and the corresponding continuation text label, and the parameters of the projector module are updated based on the first text loss value, completing the first round of modal alignment training. The second acoustic embedding representation of the second speech prompt is extracted using a speech encoder and projector module. A decision module and text generation module generate the corresponding second text token based on the second acoustic embedding representation. The second text loss value is determined based on the second text token and the corresponding continuation text label, and the parameters of the speech encoder and projector module are updated based on the second text loss value, completing the second round of modal alignment training.

[0047] like Figure 5 The diagram shown is a flowchart of step S203 provided in an embodiment of the present invention. (Refer to...) Figure 5 As an optional implementation, multiple speech generation training data are constructed based on quadruples, specifically including: S2031. Obtain text-to-speech input data and text-to-speech output data from historical speech generation scenarios; S2032. Determine text query samples and voice query samples based on text-to-speech input data, and determine corresponding text response labels and voice response labels based on text-to-speech output data; S2033. Construct speech generation training data based on text query samples, speech query samples, text response labels, and speech response labels.

[0048] Specifically, the speech generation training phase utilizes a large-scale real-world dataset (approximately 150,000 hours) for training, aiming to learn how to generate high-fidelity, expressive speech token sequences. To achieve the ability to generate speech driven by prompts while maintaining consistent speaking style, the training data is organized in the form of a quadruple: <Text Query (TQ), Speech Query (SQ), Text Response (TA), Speech Response (SA)>. The speech query and speech response are required to maintain consistency in speaker identity and speaking style to ensure that the generated speech inherits the speaker and dialogue characteristics of the prompts.

[0049] like Figure 6 The diagram shown is a flowchart of step S204 provided in an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, the large language model includes a decision module, a text generation module, and a speech synthesis module. The large language model is trained on speech generation training data to optimize the top-level parameters, specifically including: S2041. Based on the language type, the speech generation training data is divided into a third training dataset and a fourth training dataset. The third training dataset includes multiple first text query samples and corresponding text response labels and speech response labels. The fourth training dataset includes multiple first speech query samples and corresponding text response labels and speech response labels. S2042. Generate the corresponding first voice token based on the first text query sample through the decision module, text generation module and speech synthesis module; S2043. Determine the first speech loss value based on the first speech token and the corresponding speech response tag, and update the parameters of the speech synthesis module based on the first speech loss value; S2044. Extract the third acoustic embedding representation of the first speech query sample through the speech encoder and projector module, and generate the corresponding second speech token based on the third acoustic embedding representation through the decision module, text generation module and speech synthesis module. S2045. Determine the second speech loss value based on the second speech token and the corresponding speech response tag, and update the parameters of the speech synthesis module based on the second speech loss value.

[0050] Specifically, the large language model in this embodiment of the invention includes a decision-making module (thinker), a text generation module (texter), and a speech synthesis module (speaker), with speech generation training mainly targeting the speech synthesis module.

[0051] The training process consists of two steps. The first step is joint optimization driven by mixed data (cold start phase). For Mandarin and English data, a triple input format is used.<TQ,TA,SA> Training was conducted to enhance the model's stability in text-to-speech token mapping across mainstream languages; for dialect data, a triplet input format was adopted.<SQ,TA,SA> Training was conducted to learn dialect-following capabilities while generating speech. This stage involved joint optimization using a mix of Mandarin / English and dialect data, aiming to obtain a cold-start model that could stably generate mainstream languages ​​while also possessing dialect-following capabilities. This is in contrast to directly using...<SQ,TA,SA> Triple training, the step-by-step training method described above, makes the model (especially in cross-Chinese / English generation scenarios) more stable. The second step, prompt-driven fine-tuning, builds upon the cold-start model and uniformly adopts...<SQ,TA,SA> The triplet format is used to continue training to enhance the model's cue-driven generation capabilities, enabling the model to more reliably follow the speaking style and paralinguistic features of any cue speech to complete the speech generation of the target text.

[0052] Specifically, based on language type, the speech generation training data is divided into a third training dataset (Mandarin and English data) and a fourth training dataset (dialect data). The third training dataset (data format)<TQ,TA,SA> This includes multiple first text query samples and their corresponding text response labels and speech response labels, and a fourth training dataset (data format).<SQ,TA,SA> The training process includes multiple first speech query samples and corresponding text response tags and speech response tags. A decision module, text generation module, and speech synthesis module generate a corresponding first speech token based on the first text query samples. A first speech loss value is determined based on the first speech token and the corresponding speech response tag, and the parameters of the speech synthesis module are updated based on the first speech loss value. Simultaneously, a third acoustic embedding representation of the first speech query samples is extracted through a speech encoder and projector module. A corresponding second speech token is generated based on the third acoustic embedding representation through the decision module, text generation module, and speech synthesis module. A second speech loss value is determined based on the second speech token and the corresponding speech response tag, and the parameters of the speech synthesis module are updated based on the second speech loss value. The first training step is completed through these steps.

[0053] For the second step of training, a unified approach will be adopted.<SQ,TA,SA> The training data in triplet format is used to perform prompt-driven fine-tuning of the large language model after the first step of training. The process is similar to the training process of the fourth training dataset, and will not be described in detail here.

[0054] As an optional implementation, the speech synthesis module uses a multi-token prediction mechanism to generate speech tokens and uses an automatic gradient masking strategy based on prediction confidence to update parameters. A multi-token prediction mechanism is used to generate voice tokens, specifically: In each step, multiple subsequent speech tokens are predicted in parallel. The output token of the current step is concatenated with the input features of the next step as the conditional input of the speech synthesis module to guide the iterative generation of speech tokens. The parameters are updated using an automatic gradient masking strategy based on prediction confidence, specifically as follows: During parameter updates, the corresponding gradient contribution is determined based on the prediction confidence of each speech token, so that full gradient updates are applied to high-confidence samples and full masking is applied to low-confidence samples.

[0055] Specifically, the embodiments of the present invention employ the following parameter optimization strategies for speech generation training of large language models: 1) Modular Freeze and Targeted Fine-Tuning: Freeze the parameters of the speech encoder, Projector module, and bottom N layers of LLM, and only perform task-oriented fine-tuning on the parameters of the top K layers of LLM to predict speech tokens.

[0056] 2) Multi-token prediction mechanism: Multiple subsequent speech tokens are predicted in parallel at each step. The output token of the current step is embedded and concatenated with the input features of the next step and used as the conditional input of the subsequent LLM, thereby guiding the iterative generation of speech tokens.

[0057] 3) Automatic gradient masking based on prediction confidence: Full gradient updates are applied to high-confidence samples, while low-confidence samples are completely masked to avoid noisy samples from adversely affecting model parameters. Specifically, this invention employs a combination of modular freezing and targeted fine-tuning to balance the semantic understanding of the pre-trained language model with its adaptability to speech generation tasks. The LLM has a total of M layers, divided into a bottom N layers (containing the decision module and text generation module) and a top K layers (corresponding to the speech synthesis module). During the speech generation training phase, the parameters of the speech encoder, the Projector module, and the bottom N layers of the LLM are frozen, while only the parameters of the top K layers of the LLM are fine-tuned for task-oriented purposes to predict speech tokens. This hierarchical fine-tuning strategy aims to focus on optimizing high-level representations related to speech generation without compromising the original contextual understanding capabilities of the pre-trained language model.

[0058] To improve inference efficiency and streaming generation capabilities, this invention introduces a multi-token prediction mechanism at the generation stage: within the decoding process, multiple subsequent speech tokens are predicted in parallel at each step. The output token of the current step is embedded and concatenated with the input features of the next step, serving as the conditional input for subsequent LLM, thereby guiding the iterative generation of speech tokens. This mechanism can reduce the number of decoding steps, alleviate the frequency differences between text tokens and speech tokens on the time scale, and naturally support end-to-end streaming synthesis.

[0059] Furthermore, to improve model robustness and pronunciation stability, this invention employs a confidence-based gradient masking strategy: when calculating parameter updates, the gradient contribution of a single speech token is determined based on the model's prediction confidence; that is, high-confidence samples receive a full gradient update, while low-confidence samples are completely masked, thus preventing noise samples from adversely interfering with model parameters. This selective gradient update mechanism reduces the amplification of error patterns, significantly improving pronunciation stability and overall speech accuracy.

[0060] like Figure 7 The diagram shows a flowchart illustrating a step-by-step process for post-training optimization of a large language model according to an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, the large language model undergoes post-training optimization through the following steps: S301. Obtain preset voice prompt samples, synthesized text samples and corresponding target text labels, and determine the reward model corresponding to different language types; S302. Generate multiple candidate speech tokens based on speech prompt samples and synthesized text samples using a speech encoder, projector module, and large language model. S303. The stream matching decoding module reconstructs each candidate speech token into the corresponding candidate audio, and inputs it into the reward model for transcription to obtain the corresponding transcribed text. S304. Determine the reward score for each candidate speech token based on the shortest edit distance between the transcribed text and the target text tag; S305. Calculate the advantage function based on the reward score, and fine-tune and optimize the large language model based on the advantage function.

[0061] Specifically, in existing technologies, supervised fine-tuning (SFT) enables models to generate semantically coherent speech, but its optimization goal is to approximate the joint probability distribution of training samples, essentially a compression and inductive process. This process inevitably produces "illusion" problems, such as repetition, omissions, or incompleteness of speech content, leading to a decrease in the perceived quality of the generated speech. Existing research has shown that introducing reinforcement learning can significantly improve the naturalness and intelligibility of synthesized audio. In the field of text generation, the practice of models such as the GPT-O series and DeepSeek-R1 also demonstrates that reinforcement learning has become the dominant paradigm, its advantage being that it directly optimizes the final goal rather than merely fitting the data distribution. Inspired by this, this invention introduces a GRPO-based reinforcement learning method in the speech generation stage. However, unlike text generation, speech generation directly outputs speech tokens and lacks readily available quality assessment tools; therefore, constructing an effective reward signal becomes a key challenge.

[0062] To address this problem, this invention employs a pre-trained Automatic Speech Recognition (ASR) model as a reward model to transcribe the final synthesized audio and generate a reward signal. In its specific implementation: 1) For Chinese corpora, the Paraformer model is used as the reward model; 2) For English and mixed Chinese-English corpora, the Whisper-large-v3 model is used as the reward model.

[0063] To avoid memory overflow (OOM) during training due to excessively long computational chains and high model coupling, this invention encapsulates the stream matching model and reward model as independent services, deployed on different servers. During the training phase, the speech generation model obtains reward scores via remote invocation, thereby ensuring the stability and scalability of the training process.

[0064] In the training process, the system samples a set of candidate speech token sequences, numbered G, from the input prompt speech and target text through a speech generation branch. To enhance exploration capabilities and data utilization efficiency, this invention introduces a dynamic perturbation strategy for hyperparameters (such as top-k, top-p, etc.) during the sampling phase, resulting in greater output diversity. Specifically, assuming top-k is 25 and top-p is 0.8 during inference, half (G / / 2) of the samples will have top-k ∈ [15, 35] and top-p ∈ [0.6, 1.0], while the other half will be fixed to match the inference. Subsequently, these speech token sequences are reconstructed into waveform audio by a stream matching model and input into a reward model for transcription, yielding the corresponding transcribed text.

[0065] In designing the reward function, this invention employs a non-linear normalization method to restrict the reward value to the (0,1) interval. Specifically, the generated audio is transcribed using Paraformer or Whisper-large-v3, and the shortest edit distance is calculated between the transcribed audio and the target text. The result is used as the primary reward score.

[0066] in, This represents the transcribed text corresponding to the i-th generated speech. This represents the target text corresponding to the i-th generated speech, where k is a hyperparameter set to 0.4.

[0067] The reward score, after normalization, is transformed into an advantage function, which is used to optimize the policy model.

[0068] in, It outputs the voice token. The corresponding reward points, It is a vector consisting of the reward scores of all outputs in the same group.

[0069] Finally, a reinforcement learning method based on GRPO is used to fine-tune and optimize the large language model according to the obtained advantage function. The specific process will not be elaborated here.

[0070] In some optional embodiments, the present invention can also flexibly introduce other reward models to achieve multi-dimensional supervision. For example, an emotion recognition model can be used to evaluate whether the generated speech is consistent with a specified emotion label; a speaker similarity model can be used to evaluate the similarity score between the generated speech and the prompt speech in terms of timbre, etc.

[0071] During the inference phase, the input speech query SQ is used as a prompt speech. The speech encoder and the Projector module extract the continuous acoustic embedding. The continuous acoustic condition and the representation of the text to be synthesized TA are jointly input into the LLM (the bottom N layers are kept frozen and the top K layers are used for generation). A series of speech tokens SA are output through multi-token prediction. The generated speech tokens are input into the stream matching decoding module to reconstruct the Mel spectrum and synthesize the final waveform.

[0072] It should be noted that the process of converting each target speech token into a corresponding target Mel spectrogram through the stream matching decoding module, and reconstructing the target Mel spectrogram based on probability density estimation to obtain the target speech is existing technology, and will not be described in detail in this embodiment of the invention.

[0073] The implementation process of the present invention has been described above. The present invention will be further explained below in conjunction with test experiments.

[0074] To verify the stability and robustness of the model in multilingual and complex scenarios, the standard test set SEED was selected for evaluation. This test set includes a Chinese subset (test-zh), an English subset (test-en), and a high-difficulty Chinese subset (test-hard). In comparative experiments, the GOAT-TTS model described in this invention was compared with existing open-source zero-shot speech synthesis models such as FireRedTTS, MaskGCT, and CosyVoice2. The experimental results are shown in Table 1 below.

[0075]

[0076] Table 1 As shown in Table 1, under non-streaming inference conditions, GOAT-TTS ranks third in performance for the Chinese subset (test-zh), second only to CosyVoice2 and FireRedTTS; in the English subset (test-en) and the high-difficulty subset (test-hard), GOAT-TTS ranks third and second respectively, indicating that the model can maintain strong robustness to prompts in diverse scenarios.

[0077] Under streaming inference conditions, GOAT-TTS outperforms CosyVoice2-S on an overall scale in English and high-difficulty subsets, validating the significant advantages of its native streaming architecture in speech generation.

[0078] Model fine-tuned using the GRPO method It outperforms all evaluation metrics, surpassing both the baseline model and the untuned model: compared to the untuned model, it reduces character error rate (CER) by 49.7% on the test-zh subset, word error rate (WER) by 17.4% on the test-en subset, and CER by 22% on the test-hard subset.

[0079] As can be seen, the present invention exhibits high stability and robustness under both non-streaming and streaming inference conditions, and further improves the accuracy of speech synthesis by introducing GRPO fine-tuning.

[0080] In the validation experiment of dialect speech generation, Central Plains Mandarin and Southwestern Mandarin from the open-source KeSpeech dataset were selected as target dialects. The experimental procedure is as follows: speech prompts were randomly sampled from the target dialect training set; the GOAT-TTS model proposed in this invention and the contrast model CosyVoice2 were used to generate the corresponding dialect speech respectively; the synthesized speech was used as training corpus to train the Automatic Speech Recognition (ASR) model to evaluate the effectiveness of data generated by different speech synthesis methods. In the experiment, the ASR model was based on Zipformer. The comparison results are shown in Table 2 below, where GT represents the model trained on the original data, and the baseline represents the initial ASR model.

[0081]

[0082] Table 2 As shown in Table 2, the ASR model trained using dialect data generated by GOAT-TTS has a significantly higher recognition accuracy than the model trained using CosyVoice2 data. On the Central Plains Mandarin test set, the word error rate (WER) is reduced by 21.0%; on the Southwestern Mandarin test set, the word error rate is reduced by 33.4%.

[0083] The above results demonstrate that the model of this invention outperforms existing open-source models in dialect generation accuracy and can generate speech data that better matches the characteristics of the target dialect. Furthermore, using the dialect speech generated by the GOAT-TTS model provided by this invention to enhance the baseline dialect ASR model, although there is still a certain gap compared to the model trained using real dialect data, the synthetic data can improve the performance of the baseline model by more than 59% in the absence of real large-scale dialect data. Therefore, this invention can significantly improve the recognition ability of the dialect ASR model in the absence of real dialect data, thus proving that its generated speech has significant advantages in terms of dialect feature fidelity and usability.

[0084] It is understood that the embodiments of the present invention achieve effective association between language features, prosodic information, emotional cues and semantic text through pre-trained speech encoder and projector module, and complete cross-modal alignment of speech and text. Through pre-trained large language model, it can maintain the ability to understand complex contexts during speech generation and supports real-time streaming speech synthesis, which can meet diverse application needs, improve the accuracy and stability of speech generation, and thus improve the user's voice interaction experience.

[0085] like Figure 8 The diagram shown is a structural schematic of the LLM-based dual-stream hyperhuman speech generation device provided in an embodiment of the present invention. (Refer to...) Figure 8 This invention provides an LLM-based dual-stream hyperhuman speech generation device, comprising: The acoustic representation extraction module is used to extract the target latent acoustic representation of the input target speech cues through a pre-trained speech encoder; The acoustic representation mapping module is used to map the latent acoustic representation of the target to the target acoustic embedding representation through a pre-trained projector module; The speech token generation module is used to input the target acoustic embedding representation and the text to be synthesized into a pre-trained large language model to generate multiple target speech tokens; The speech reconstruction module is used to convert each target speech token into a corresponding target Mel spectrogram through the stream matching decoding module, and reconstruct the target Mel spectrogram based on probability density estimation to obtain the target speech.

[0086] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0087] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned LLM-based dual-stream hyper-human speech generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0088] like Figure 9 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 9 This invention provides an electronic device, comprising: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to implement the LLM-based dual-stream hyper-human speech generation method of the embodiments of this invention. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0089] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0090] like Figure 10 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 10 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1001, which can be executed by one or more processors to implement the above-described LLM-based dual-stream hyper-human speech generation method.

[0091] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0092] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the above-described LLM-based dual-stream hyper-human speech generation method.

[0093] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0094] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0096] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0097] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0098] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0099] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0101] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0102] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0104] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0105] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for generating hyperhuman-like speech using LLM-based dual-stream text and speech, characterized in that, Includes the following steps: Extract the target latent acoustic representation of the input target speech cues using a pre-trained speech encoder; The target latent acoustic representation is mapped to a target acoustic embedding representation through a pre-trained projector module; The target acoustic embedding representation and the text to be synthesized are input into a pre-trained large language model to generate multiple target speech tokens; The target speech tokens are converted into corresponding target Mel spectrograms by the stream matching decoding module, and the target speech is obtained by reconstructing the target Mel spectrograms based on probability density estimation.

2. The method for generating hyper-human speech based on LLM using both text and speech streams according to claim 1, characterized in that, The speech encoder, the projector module, and the large language model are pre-trained through the following steps: Construct multiple speech-text continuation pairs based on acoustic and / or semantic driving; With the parameters of the large language model fixed, modal alignment training is performed on the initialized speech encoder and projector module based on the speech-text continuation pair data to obtain the pre-trained speech encoder and projector module. Multiple speech generation training data are constructed based on quadruples; The parameters of the speech encoder and the projector module are fixed, and the underlying parameters of the large language model are also fixed. The large language model is then trained to generate speech based on the speech generation training data to optimize the top-level parameters, thereby obtaining a pre-trained large language model.

3. The method for generating hyper-human speech based on LLM using both text and speech streams according to claim 2, characterized in that, The construction of multiple speech-text continuation pairs based on acoustic and / or semantic driving specifically includes: Multiple first speech samples and corresponding first transcribed texts are obtained from the speech recognition corpus. A corresponding first continuation text is generated based on the first transcribed text using a prompting language model. The speech-text continuation pair data is generated based on the first speech samples and the first continuation text. And / or, Multiple first text phrases are obtained, and the first text phrases are converted into second speech samples through a speech synthesis model. Then, a corresponding second continuation text is generated based on the first text phrases through a prompting language model. Finally, the speech-text continuation pair data is generated based on the second speech samples and the second continuation text.

4. The method for generating hyper-human speech based on LLM using both text and speech streams according to claim 2, characterized in that, The large language model includes a decision module, a text generation module, and a speech synthesis module. The modal alignment training of the initialized speech encoder and projector module based on the speech-text continuation data specifically includes: The speech-text continuation pair data is divided into a first training dataset and a second training dataset according to the language type. The first training dataset includes multiple first speech prompts and corresponding continuation text labels, and the second training dataset includes multiple second speech prompts and corresponding continuation text labels. The first acoustic embedding representation of the first voice prompt is extracted by the speech encoder and the projector module, and the corresponding first text token is generated by the decision module and the text generation module based on the first acoustic embedding representation. A first text loss value is determined based on the first text token and the corresponding continuation text tag, and the parameters of the projector module are updated based on the first text loss value. The second acoustic embedding representation of the second voice prompt is extracted by the speech encoder and the projector module, and the corresponding second text token is generated by the decision module and the text generation module based on the second acoustic embedding representation. The second text loss value is determined based on the second text token and the corresponding continuation text tag, and the parameters of the speech encoder and the projector module are updated based on the second text loss value.

5. The method for generating hyper-human speech based on LLM using both text and speech streams according to claim 2, characterized in that, The construction of multiple speech generation training data based on quadruples specifically includes: Acquire text-to-speech input data and text-to-speech output data from historical speech generation scenarios; Based on the text-to-speech input data, determine text query samples and speech query samples, and based on the text-to-speech output data, determine the corresponding text response tags and speech response tags; The speech generation training data is constructed based on the text query samples, the speech query samples, the text response tags, and the speech response tags.

6. The method for generating hyper-human speech based on LLM using both text and speech streams according to claim 2, characterized in that, The large language model includes a decision module, a text generation module, and a speech synthesis module. The step of training the large language model to optimize its top-level parameters based on the speech generation training data specifically includes: The speech generation training data is divided into a third training dataset and a fourth training dataset according to the language type. The third training dataset includes multiple first text query samples and corresponding text response labels and speech response labels. The fourth training dataset includes multiple first speech query samples and corresponding text response labels and speech response labels. The decision-making module, the text generation module, and the speech synthesis module generate a corresponding first speech token based on the first text query sample. A first speech loss value is determined based on the first speech token and the corresponding speech response tag, and the parameters of the speech synthesis module are updated based on the first speech loss value. The speech encoder and the projector module extract the third acoustic embedding representation of the first speech query sample, and the decision module, the text generation module and the speech synthesis module generate the corresponding second speech token based on the third acoustic embedding representation. The second speech loss value is determined based on the second speech token and the corresponding speech response tag, and the parameters of the speech synthesis module are updated based on the second speech loss value.

7. The method for generating hyperhuman-like speech based on LLM using both text and speech streams according to claim 6, characterized in that, The speech synthesis module uses a multi-token prediction mechanism to generate speech tokens and uses an automatic gradient masking strategy based on prediction confidence to update parameters. The method of generating voice tokens using a multi-token prediction mechanism is as follows: In each step, multiple subsequent speech tokens are predicted in parallel. The output token of the current step is concatenated with the input features of the next step as the conditional input of the speech synthesis module to guide the iterative generation of speech tokens. The parameter update using an automatic gradient masking strategy based on prediction confidence is specifically as follows: During parameter updates, the corresponding gradient contribution is determined based on the prediction confidence of each speech token, so that full gradient updates are applied to high-confidence samples and full masking is applied to low-confidence samples.

8. A method for generating hyper-human speech based on LLM according to any one of claims 2 to 7, characterized in that, The large language model is post-training optimized through the following steps: Obtain preset voice prompt samples, synthesized text samples, and corresponding target text labels, and determine the reward model corresponding to different language types; The speech encoder, the projector module, and the large language model generate multiple candidate speech tokens based on the speech prompt samples and the synthesized text samples. The stream matching decoding module reconstructs each candidate speech token into a corresponding candidate audio, which is then input into the reward model for transcription to obtain the corresponding transcribed text. The reward score for each candidate speech token is determined based on the shortest edit distance between the transcribed text and the target text tag; The advantage function is calculated based on the reward score, and the large language model is fine-tuned and optimized based on the advantage function.

9. A text-to-speech dual-stream hyper-human speech generation device based on LLM, characterized in that, include: The acoustic representation extraction module is used to extract the target latent acoustic representation of the input target speech cues through a pre-trained speech encoder; An acoustic representation mapping module is used to map the target latent acoustic representation into a target acoustic embedding representation through a pre-trained projector module; The speech token generation module is used to input the target acoustic embedding representation and the text to be synthesized into a pre-trained large language model to generate multiple target speech tokens; The speech reconstruction module is used to convert each target speech token into a corresponding target Mel spectrogram through the stream matching decoding module, and reconstruct the target Mel spectrogram based on probability density estimation to obtain the target speech.

10. An electronic device, characterized in that, The electronic device includes a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the computer program is executed by the processor, it implements the LLM-based hyper-human speech generation method as described in any one of claims 1 to 8.

Images