An end-to-end voice interaction method without text guidance

Abstract

The application relates to an end-to-end voice interaction method without text guidance, which comprises the following steps: acquiring a voice signal input by a user, inputting the voice signal into a pre-trained voice model, and outputting a reconstructed voice signal; the voice model comprises a voice encoder, a multi-modal processing module and a voice decoder; the step of outputting the reconstructed voice signal comprises the following steps: converting the input voice signal into a voice token sequence by using the voice encoder; performing cross-modal semantic fusion and alignment of text-semantics by using the multi-modal processing module, and obtaining a probability distribution of the voice token by deep optimization; and decoding the probability distribution of the voice token by using the voice decoder to obtain the reconstructed voice signal, so that an end-to-end voice interaction process is realized; and during the training process of the voice model, a text model and / or text data are used to participate in the training. Compared with the prior art, the application has the advantages of not needing text guidance and the like.

Description

Technical Field

[0001] This invention relates to the field of voice interaction, and in particular to an end-to-end voice interaction method that does not require text guidance. Background Technology

[0002] Against the backdrop of rapid development in artificial intelligence, voice dialogue has become an important means of human-computer interaction. Traditional voice dialogue systems typically employ a cascaded pipeline architecture. This involves first transcribing the user's speech input into text using an automatic speech recognition module, then having a text model understand the text and generate a text response, and finally synthesizing the text response into speech output using a text-to-speech module. While this architecture fully utilizes the reasoning capabilities of the text model, it has significant drawbacks: during the speech-to-text process, paralinguistic information such as rhythm, emotion, and tone in the original speech signal is inevitably lost, resulting in a lack of naturalness and expressiveness in the system's output; simultaneously, because the system can only produce responses that can be represented by text, it cannot generate nonverbal cues such as laughter, hesitation, or sighs, severely limiting the system's expressive capabilities; furthermore, the three-modal conversion process introduces significant response latency, impacting the user's interactive experience.

[0003] To address these issues, end-to-end speech modeling methods have emerged in recent years. Early explorations demonstrated that speech could be directly modeled, but these methods were primarily limited to experimental dialogue continuation tasks and were difficult to scale to fully functional dialogue assistant systems. Subsequent research turned to text-guided generation as a compromise. These methods, by directly accepting speech input, preserve paralinguistic information to some extent, but they still rely on intermediate text representations during the generation phase, such as... Figure 2 Figures (a) and (b) show the architecture of a traditional cascaded voice interaction system. In this system, the input audio is first converted to text through speech recognition, then the text model generates a response, and finally, the text response is synthesized into a voice response. Figure (b) shows the architecture of an existing text-guided end-to-end streaming voice interaction model. The input speech is encoded and directly understood by the large model, but a text draft of the corresponding content needs to be generated before generating a voice response. Currently, this design creates fundamental bottlenecks: First, the text-guided process introduces additional latency and computational overhead, reducing system efficiency; second, due to the lack of a natural text-equivalent representation for non-verbal speech, the system's expressive power remains limited; more importantly, due to the inherent differences between speech and text modalities, current methods often significantly degrade the model's original text processing capabilities while introducing speech capabilities, severely impairing the model's reasoning ability and knowledge reserves.

[0004] While some works support direct speech generation, its performance is significantly weaker than text-guided generation. This reflects a core challenge currently facing the technology: narrowing the performance gap between text-guided generation and direct speech generation is crucial for achieving true end-to-end voice interaction. Furthermore, existing methods generally suffer from low knowledge density in speech data. Since the knowledge density of natural speech corpora is far lower than that of text corpora, it is difficult to support large-scale, knowledge-intensive speech pre-training, further limiting the performance improvement potential of direct speech generation methods. In summary, existing technologies face technical bottlenecks in achieving true end-to-end voice interaction, urgently requiring a solution that can maintain the reasoning and knowledge capabilities of text models while natively understanding and generating high-quality speech. Summary of the Invention

[0005] The purpose of this invention is to provide an end-to-end voice interaction method that directly generates speech without text guidance.

[0006] The objective of this invention can be achieved through the following technical solutions: An end-to-end voice interaction method without text guidance includes the following steps: The process involves acquiring user-input speech signals, inputting them into a pre-trained speech model, and outputting reconstructed speech signals. The speech model includes a speech encoder, a multimodal processing module, and a speech decoder. The steps for outputting the reconstructed speech signals include: The input speech signal is converted into a speech token sequence using the speech encoder. The speech token sequence is input into the multimodal processing module for cross-modal semantic fusion and alignment of text and semantics, and the probability distribution of speech tokens is obtained through deep optimization. The probability distribution of the voice token is decoded using the voice decoder to obtain the reconstructed voice signal, thereby realizing the end-to-end voice interaction process. The speech model training process utilizes a text model and / or text data together for training.

[0007] Furthermore, the speech encoder employs a fully causal convolutional and Transformer neural network architecture.

[0008] Furthermore, the speech model is trained using a two-stage pre-training process: the first stage is the speech pre-training stage, and the second stage is the supervised fine-tuning stage.

[0009] Furthermore, the training steps in the speech pre-training phase include: Select a pre-trained text model as the base model; Construct a large-scale corpus containing mixed speech-text data, wherein the large-scale corpus includes two types of speech data: real speech data filtered using speech activity detection technology and speech data synthesized from text. The large-scale corpus is divided into two groups to obtain two sets of training data: initial interleaved speech-text training data and unsupervised speech training data. For the real speech data and text-synthesized speech data in the initial interleaved speech-text training data, a connection-time classification algorithm is used to align them with the text content in time. The time-aligned real speech and the text-synthesized speech data are segmented to obtain real speech segments and text-synthesized speech segments respectively. The real speech segments and text-synthesized speech segments are then arranged alternately to obtain the final interleaved speech-text training data. The basic model is introduced into the speech model to form a large speech-text model; Freeze all parameters of the text model in the speech-text large model, and use the final interleaved speech-text training data and unsupervised speech training data to complete the training process of the speech pre-training stage.

[0010] Furthermore, the training steps in the supervised fine-tuning phase include: The original question-and-answer text dataset is obtained, rewritten into speech, and then audio is synthesized using various text-to-speech methods to obtain the initial speech question-and-answer dataset. The rewriting process removes symbols or formats that are not suitable for audio synthesis, converts mathematical formulas into a readable form, simplifies lengthy content, and assigns tone and style. The word error rate of each voice question and answer data in the initial voice question and answer dataset is automatically compared with the corresponding original question and answer text. Voice question and answer data with a word error rate greater than a preset threshold are removed to obtain the final voice question and answer dataset. Unfreeze some or all parameters of the text model in the speech-text large model, fine-tune it using the final speech question-answering dataset, continuously introduce question-answering text data, and dynamically adjust the learning rate through a cosine learning rate scheduling strategy until the supervised fine-tuning training process is completed.

[0011] Furthermore, during the supervised fine-tuning phase, the speech-text large model employs the following four input-output modal configurations during training: (1) Question asked by voice → Answered by voice; (2) Question by voice → Answer by text; (3) Text question → Voice answer; (4) Text question → Text answer.

[0012] Furthermore, the network structure of the speech-text large model includes an encoder connected in series, a multimodal processing module composed of multiple Transformers, and a decoder. The encoder includes a parallel text encoder and a speech encoder. The text encoder converts the input text data into a sequence of text tokens, and the speech encoder converts the input speech data into a sequence of speech tokens. The multimodal processing module includes a shared layer and two parallel branches: a text processing branch consisting of several Transformer layers and a speech processing branch consisting of several Transformer layers. The decoder includes a text decoder and the speech decoder.

[0013] Furthermore, in the multimodal processing module, the shared layer is used to receive text token sequences and speech token sequences. Through training methods using self-attention mechanism, next token prediction, and cross-entropy loss, the speech model can achieve cross-modal semantic fusion and alignment of text and semantics without explicit alignment annotation, thereby obtaining aligned text token sequences and aligned speech token sequences respectively. The text processing branch is used to optimize the aligned text token sequence to obtain the probability distribution of the text tokens; The speech processing branch is equivalent to optimizing the aligned speech token sequence to obtain the probability distribution of the speech tokens.

[0014] Furthermore, the layer structure partitioning between the shared layer and the two parallel branches in the multimodal processing module is performed through the following steps: Given a text and its corresponding speech sequence, convert it into a text token sequence and a speech token sequence, respectively. In the multimodal processing module, the hidden state of each layer of text tokens and the hidden state of each layer of voice tokens are obtained layer by layer from the text token sequence and the voice token sequence. Calculate the cosine similarity matrix between the hidden state of the text token and the hidden state of the voice token at each layer; The cosine similarity matrix is ​​used as the input of the dynamic time warping algorithm. The start and end points of the aligned text token sequence and the aligned speech token sequence are constrained, and the path is limited to satisfy the time monotonicity and continuity constraints. The cumulative matching cost of each position in the cosine similarity matrix is ​​calculated step by step using dynamic programming. Then, the cosine similarity matrix is ​​backtracked to select the path with the optimal cumulative matching cost as the optimal alignment path between speech and text. The multimodal processing module is divided according to the optimal alignment path to obtain a shared layer and two parallel branches.

[0015] Furthermore, the speech decoder employs a stream-matching-based generative model architecture, and the step of obtaining the reconstructed speech signal includes: The aligned speech token sequence is mapped to a high-dimensional Mel spectrum representation by a continuous normalization stream, and then converted into a time-domain waveform by a vocoder as the reconstructed speech signal.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) By introducing a text model and / or text data during the training process, this invention ensures that the speech model can integrate the language understanding capabilities and knowledge reserves of the text model. Thus, in practical applications, only the trained speech model is needed to generate speech directly from speech without any text guidance. (2) The present invention trains the speech model through a two-stage pre-training process. In the first stage of training, interwoven speech and text training data is constructed to train only the parameters of the speech model, thereby achieving speech alignment while freezing the text backbone. In the second stage of training, the parameters of the text model are unfrozen, which enables cross-modal deep adaptation while preserving text knowledge. Additional pure text pre-training data is introduced during training to maintain the language ability of the model.

[0017] (3) The two-stage pre-training process of this invention, through freeze-thaw progressive training, successfully reduces the damage to text capabilities while introducing speech capabilities, achieving a good balance between the two modal capabilities. Since the main knowledge reserves are still retained in the pre-trained text model, this invention avoids dependence on large-scale knowledge-intensive speech datasets. Through the alignment mechanism, reasoning ability, world knowledge, and generalization ability are transferred from the text backbone to the speech modality, significantly reducing the cost of data collection and annotation.

[0018] (4) This invention utilizes the previous shared layers for deep multimodal fusion by performing modal segmentation at appropriate positions within a multi-layer Transformer architecture. This design, which fuses the two token sequences before separating them, fully leverages the existing shared layers to optimize the generation characteristics of each modality. This design allows the reasoning ability and knowledge reserves of the text model to be effectively transferred to the speech modality, while achieving high quality in both text and speech generation, avoiding compromises between the two modalities by a single output head. By determining the optimal segmentation position through similarity analysis, the branching is ensured only after the speech and text representations have reached optimal alignment, thereby maximizing the effect of cross-modal knowledge transfer.

[0019] (5) This invention designs a semantically guided speech codec. The speech encoder uses automatic speech recognition as the sole training objective and improves it to a fully causal architecture. The speech decoder is based on a streaming matching architecture and reduces latency by compressing the processing block size. This design enables the speech encoder to maintain excellent semantic capture capabilities under a fully streaming architecture, and the speech decoder to achieve high-quality speech reconstruction at lower frame rates. At the same time, the single codebook design significantly reduces the length of the token sequence, making autoregressive generation more efficient. Both the codec and the decoder support full streaming processing, and can start processing and generation without waiting for complete speech segments, significantly reducing end-to-end latency.

[0020] (6) This invention achieves direct speech generation without text guidance. The speech model can directly generate speech output from speech input without relying on any intermediate text representation. This improvement significantly reduces response latency, enabling a smoother real-time dialogue experience. Because it is not limited by text representation, it can generate non-verbal sounds such as laughter, hesitation, and sighs, making the dialogue more natural and expressive. It also reduces the number of modality transitions, lowering computational complexity and energy consumption. This invention successfully achieves performance close to that of text-guided methods while overcoming the inherent limitations of text-guided methods in terms of latency and expressiveness, establishing a new technological paradigm for end-to-end speech interaction.

[0021] (7) This invention supports native multimodal interaction. A single model can simultaneously process text and speech input and output, supporting multiple modal combinations. This design allows users to freely choose the input and output modalities according to the actual scenario. A single model can adapt to multiple application scenarios, significantly improving the system's versatility. By jointly training multiple modal combinations during the supervised fine-tuning stage, the model can smoothly switch between text and speech, achieving true multimodal understanding and generation. The unified framework simplifies system development, deployment, and maintenance. Compared to traditional solutions that require the integration of multiple independent modules, the single-model architecture of this invention is more concise and efficient. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the implementation process of an embodiment of the present invention; Figure 2 This is a schematic diagram of the two-stage pre-training process of the present invention; Figure 3 The diagrams are comparative diagrams of the prior art and the method of this embodiment of the present invention, wherein (a) is a flowchart of the first prior art for realizing voice interaction, (b) is a flowchart of the second prior art for realizing voice interaction, and (c) is a flowchart of the method of this embodiment for realizing voice interaction. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0024] This embodiment provides an end-to-end voice interaction method without text guidance. This method, through a modality-based layer segmentation architecture and a frozen pre-training strategy, enables a large language model to natively understand and generate speech while maintaining its original text processing capabilities, thereby achieving true end-to-end voice interaction. This embodiment implements end-to-end voice interaction through the following sections.

[0025] 1. Overview of Interaction Methods This invention first systematically analyzes the hidden state similarity at different levels of a pre-trained speech-text large model, revealing that in deep neural networks, speech and text representations gradually merge in the shallow layers, reach the highest alignment in the intermediate layers, and then gradually separate in the deeper layers. For example... Figure 3 As shown, in this embodiment, a large speech-text model is constructed using a speech model and a text model. Its network structure includes a serially connected encoder, a multimodal processing module consisting of multiple Transformers, and a decoder. The encoder includes a parallel text encoder (i.e.,...). Figure 3 The text embedding layer in the text and the speech encoder (i.e. Figure 3 The text encoder converts the input text data into a text token sequence, and the speech encoder converts the input speech data into a speech token sequence. The multimodal processing module includes a shared layer and two parallel branches. The two parallel branches are text processing branches composed of several Transformer layers (i.e.,...). Figure 3 The speech processing branch consists of the text layer and several Transformer layers (i.e., the text layer in the text processing layer) and several Transformer layers. Figure 3 The speech layer in the code), the decoder includes a text decoder (i.e., Figure 3The text output header and the speech decoder (i.e., the text output header) Figure 3 (The voice output head in the middle).

[0026] Based on this discovery, this invention innovatively proposes a modality-based layer segmentation architecture. At an appropriate depth, the Transformer branches the network into text processing and speech processing branches, enabling the preceding shared layers to achieve deep cross-modal fusion, while subsequent dedicated layers can be optimized for the characteristics of their respective modalities. Simultaneously, this invention employs a two-stage frozen pre-training strategy. In the first stage, the parameters of the pre-trained text model are completely frozen, training only the newly introduced speech-related components (i.e., the speech model) to establish stable speech-text alignment. In the second stage, the parameters of the text model are selectively unfrozen, either partially or completely, for joint training, while continuously injecting high-quality text data to prevent text capability degradation. This design effectively transfers the reasoning ability and knowledge reserves of the pre-trained text model to the speech modality, avoiding the capability degradation problem commonly found in existing methods.

[0027] In terms of speech representation, this invention presents a semantically oriented speech codec. The speech encoder uses automatic speech recognition as the sole training objective to maximize the preservation of semantic content and is improved to a fully causal architecture to support true streaming processing. The speech decoder is based on a stream matching architecture and reduces latency by compressing the processing block size.

[0028] Through the above technical solution, the embodiments of the present invention can generate speech directly from speech without any text guidance by using only a trained speech model, while maintaining text processing capabilities close to those of the original text model, thus establishing a new technical paradigm for end-to-end voice interaction.

[0029] 2 Modality-based layer partitioning architecture (see Figure 2 ) One of the core innovations of this invention is its modality-based layer segmentation architecture. To determine the optimal segmentation position, this invention first conducts an in-depth analysis of existing large-scale speech-text models.

[0030] Specifically, for a given sentence and its corresponding speech sequence, this embodiment of the invention calculates the cosine similarity matrix between the hidden states of the text token and the hidden states of the speech token at each layer of the large model, and finds the optimal alignment path through a dynamic time warping algorithm. The steps of this dynamic time warping algorithm to achieve path alignment are as follows: Using the cosine similarity matrix as input to the dynamic time warping algorithm, constraints are set on the start and end points between the aligned text token sequences and the aligned speech token sequences. The paths are also constrained to satisfy temporal monotonicity and continuity. A dynamic programming approach is employed to progressively calculate the cumulative matching cost at each position within the similarity matrix. During the calculation, a one-to-many matching relationship is allowed between the aligned text token sequences and the speech token sequences to accommodate variations in speech duration and rate. Subsequently, backtracking is performed from the end of the matrix to the start position, selecting the path with the optimal cumulative cost as the optimal alignment path between the speech and text, thus achieving automatic alignment of the speech token sequences and text token sequences in the temporal dimension.

[0031] The analysis results show that the similarity between speech and text representations steadily increases in the earlier layers, fluctuates and gradually stabilizes in the middle layers, and then decreases significantly in the last few layers. This phenomenon reveals an important pattern: speech and text representations gradually merge and achieve depth alignment in the early and middle parts of the large model, but gradually separate again at the top layer of the large model to adapt to the generation requirements of their respective modalities.

[0032] Based on the above findings, embodiments of the present invention perform modal segmentation at appropriate locations within a multi-layer Transformer architecture.

[0033] Specifically, the first few layers of the large model serve as shared layers, processing both text and speech inputs simultaneously. The primary function of these shared layers is to spatially align text and speech through cross-modal semantic fusion and alignment. Following the shared layers, the hidden states are routed to two parallel branches: the text processing branch continues to predict the probability distribution of text tokens through several Transformer layers, while the speech processing branch predicts the probability distribution of speech tokens through several more parallel Transformer layers. This fusion-then-separation design fully leverages the preceding shared layers for deep multimodal fusion while preserving subsequent dedicated layers to optimize the generation characteristics of their respective modalities. Importantly, both the text processing and speech processing branches are initialized from the same pre-trained text model backbone, ensuring that the speech processing branch inherits the language understanding capabilities and knowledge reserves of the text model.

[0034] Compared to existing technologies, the layer segmentation architecture of this invention has significant advantages. Traditional methods generate multiple vector quantization tokens at each time step, or simply extend speech tokens into the text vocabulary. Neither of these methods considers the alignment characteristics of speech and text representations at different model depths. This invention, by performing modal segmentation at an appropriate depth, ensures sufficient cross-modal knowledge transfer while avoiding the performance loss caused by forced alignment. This allows the model to natively represent speech modalities while maintaining textual capabilities.

[0035] 3. Speech Codec Design The speech codec design of this invention follows four core objectives: to achieve low bit rate representation of a single codebook to improve the efficiency of autoregressive generation, to maximize semantic content to facilitate knowledge transfer from text to speech, to preserve sufficient paralinguistic details to achieve high-fidelity speech reconstruction, and to support full streaming operations to meet the needs of low-latency dialogue systems.

[0036] In terms of speech encoder design, this invention employs automatic speech recognition as the sole training objective. Existing research indicates that discrete speech tokens optimized primarily for reconstruction tasks are often unsuitable for learning large language models. This invention, however, focuses the speech encoder's training objective on semantic content extraction, converting the original speech signal into a discrete token sequence and ensuring the generated token sequence possesses good language modeling properties. The speech encoder utilizes a fully causal convolutional and Transformer neural network architecture, achieving true streaming processing capabilities. While maintaining the integrity of semantic information, it significantly reduces the length of the token sequence, alleviating the computational burden of autoregressive generation.

[0037] In terms of speech decoder design, this embodiment of the invention adopts a stream matching architecture. Stream matching is a generative model based on continuous normalized streams, capable of achieving high-quality speech synthesis. The specific steps are as follows: the speech decoder uses a stream matching generative model architecture, mapping discrete speech tokens to a high-dimensional Mel-spectral representation through a continuous normalized stream, and then converting them into time-domain waveforms through a vocoder. This embodiment of the invention makes key improvements to meet the needs of real-time dialogue systems: the attention mask of the speech encoder is changed from block causality to full causality, enabling audio input to support token-level streaming processing with a latency of no more than 80 milliseconds; the stream matching block size of the speech decoder is compressed to approximately 5 tokens, and the speech decoder is specifically trained for streaming output, allowing the speech decoder to adapt to smaller output blocks and adding smooth transitions between different output blocks. This allows the speech decoder to output smooth, coherent, high-quality audio with a latency of no more than 400ms, significantly reducing latency while maintaining reconstruction quality, making the speech decoder particularly suitable for streaming dialogue systems requiring high fidelity and low response latency. The speech codec of this invention supports streaming output and can immediately start generating corresponding audio segments after receiving enough speech tokens, thereby achieving low-latency real-time dialogue. Its effectiveness has been verified on multiple benchmark tests, and it has achieved excellent levels in semantic preservation, reconstruction quality and streaming processing capabilities.

[0038] 4. Multimodal processing module design The multimodal processing module is the core component of this invention, employing a multi-layered Transformer architecture. Based on the modality-based layer segmentation architecture described in Section 2, the shared layer and two-branch structure are divided. The first few layers are shared layers, accepting both text and speech token sequences as input. Through training using self-attention, next-token prediction, and cross-entropy loss, the speech model achieves cross-modal semantic fusion and alignment without explicit alignment annotations, resulting in aligned text and speech token sequences. After the shared layers, the network branches into two parallel paths: the text processing branch contains several Transformer layers to optimize the aligned text token sequence and generate a probability distribution for the text tokens; the speech processing branch contains several Transformer layers to optimize the aligned speech token sequence and generate a probability distribution for the speech tokens. Both branches are initialized from the same pre-trained text model backbone. The input to the multimodal processing module can be plain text, plain speech, or a mixed sequence of text and speech. The output can be a text token sequence, a speech token sequence, or a combination of both. The specific input and output modalities are controlled by system prompts.

[0039] 5 Two-stage pre-training strategy (see...) Figure 2 ) The embodiments of the present invention employ a two-stage pre-training strategy to introduce speech modalities. The core idea of ​​this strategy is to gradually build speech understanding and generation capabilities while maintaining the capabilities of the pre-trained text model.

[0040] In the data preparation phase, this embodiment of the invention collected a large-scale amount of raw audio data from the Internet, filtered non-speech content through a speech activity detection pipeline, and obtained high-quality speech data. This data is divided into two categories based on its source type: interleaved speech-text pre-training data (also known as woven speech-text training data) mainly comes from podcasts, as podcasts typically provide clearer recordings and more distinct pronunciations; unsupervised speech pre-training data mainly comes from video content, which, although more diverse and noisy, has better robustness to challenging acoustic conditions. For the interleaved task, this embodiment of the invention applies automatic speech recognition to obtain text transcription, and then uses word alignment technology to segment the audio into random-length segments, constructing sequences by interleaving the two modalities. To alleviate the inherent low knowledge density problem in natural speech corpora, this embodiment of the invention also synthesizes additional interleaved data from high-quality text corpora, converting the text to audio through a text-to-speech system, generating large-scale synthetic speech-text pairs to enrich the training corpus.

[0041] The training objective of the first stage is to establish speech alignment while freezing the text backbone. In this embodiment of the invention, the model is initialized from a pre-trained text model backbone, freezing all its parameters and training only the newly introduced speech-related components (i.e., the speech model), including the speech encoder, the speech-specific Transformer layer in the shared layers, the speech processing branch, and the speech decoder. This stage of training employs standard optimizer and learning rate scheduling strategies, primarily to initialize speech parameters and establish a stable alignment with the pre-trained text representation, laying the foundation for subsequent joint training. Specifically, as... Figure 2 As shown in Figure (a), the specific training steps for this first stage include: Step 1-1: First, select a pre-trained text model (such as Qwen-3-8B) as the base model and use it as the core structural framework for multimodal fusion.

[0042] Steps 1-2 involve constructing a large-scale corpus containing mixed speech and text data. The data sources include both real speech data and text-synthesized speech data. The real speech data is collected over the network and then filtered for non-speech segments using a speech activity detection (VAD) module, resulting in approximately 4 million hours of pure speech data. The synthesized data is generated using high-quality text corpora (such as FineWeb-Edu) via the CosyVoice 2 system to enhance the speech knowledge density.

[0043] Steps 1-3 involve grouping the corpus into "interwoven speech-text training data" and "unsupervised speech training data." The interwoven data primarily comes from podcast corpora and is used for speech-text alignment; the unsupervised data primarily comes from video corpora and is used to improve the model's robustness in complex acoustic environments.

[0044] Steps 1-4 employ the Connected Temporal Classification (CTC) algorithm for temporal alignment of speech and text, dividing the audio into short segments of 3–6 seconds and alternating speech and text tags to obtain the final interleaved speech-text training data. This enables cross-modal sequence fusion.

[0045] Steps 1-5, entering the first stage of pre-training (Stage I): Freeze all parameters of the text model, and train only the speech-related modules using the final interleaved speech-text training data and unsupervised speech training data, including the speech encoder, the speech-specific Transformer layer in the shared layers, the speech processing branch, and the speech decoder. This stage of training achieves preliminary alignment between speech features and text features. Training uses the AdamW optimizer with an initial learning rate of 4×10⁻⁶. -4 Batch size 2.2M tokens, weight decay 0.1, maximum context length 14,336 tokens.

[0046] The training objective of the second stage is to achieve cross-modal adaptation while preserving textual knowledge. In this stage, embodiments of the invention unfreeze some or all model parameters (such as...). Figure 2 (as shown in Figure (b)) to allow for deep adaptation across modalities. Due to the risk of text capability degradation caused by unfreezing text parameters, this embodiment of the invention incorporates additional plain text pre-training data during training to maintain the model's language capabilities. The second stage involves joint training on both speech and text datasets, achieving a good balance between speech and text capabilities through appropriate learning rate scheduling and batch size settings. Since high-quality supervised fine-tuning data for voice assistants is extremely scarce in natural environments, this embodiment of the invention employs a synthetic approach to construct such data. The construction process begins with existing open-source text-supervised fine-tuning datasets, covering high-quality question-answering data from multiple domains. Specifically, the training steps in the second stage include: Step 2-1: First, select several open-source text supervision datasets (such as OpenHermes-2.5, Magpie series, COIG-CQIA, RefGPT-Fact, etc.) as basic question-answering templates.

[0047] Step 2-2, data adaptation, is the first step in supervised fine-tuning of data construction. This embodiment utilizes a language model interface (such as the GPT-5 API) to rewrite the original question-and-answer text into speech. The rewriting process includes removing symbols or formats unsuitable for speech synthesis, converting mathematical formulas into a readable form, filtering out instances that cannot be effectively presented as speech, shortening excessively long responses to make them more suitable for spoken expression (simplifying verbose content), correcting obvious factual errors, and assigning appropriate intonation and style to the speech synthesis.

[0048] Steps 2-3 involve audio synthesis using multiple text-to-speech (seedTTS) systems. To improve robustness, a multi-speaker strategy is employed to generate speech for the user end to enhance generalization, while a single, stable voice is used for the assistant end to establish a unified identity.

[0049] Steps 2-4 involve quality screening of the generated speech. The SenseVoice-Small automatic speech recognition system is used to compare the word error rate (WER) between the generated speech and the original text, eliminating samples with a WER ≥ 0.2. After screening, a large number of high-quality question-answer pairs are obtained for supervised fine-tuning, ultimately resulting in approximately 1.5 million speech question-answer samples, including approximately 650,000 English samples and approximately 860,000 Chinese samples.

[0050] In steps 2-5, the pre-trained speech-text large model is loaded with the aforementioned multimodal speech-question-answering data for supervised fine-tuning. Some or all model parameters are unfrozen to achieve deep fusion of speech and text. To prevent a decline in text capabilities, a small amount of high-quality question-and-answer text data (used for synthesizing speech-question-answering data) is mixed in during training. This high-quality question-and-answer text data accounts for approximately 10% of the total batches, and the learning rate is dynamically adjusted using a cosine learning rate scheduling strategy. In the experiment, the AdamW optimizer was used during training with an initial learning rate of 1×10⁻⁶. -5 Gradually decrease to 1×10 -6 The weights were reduced by 0.1, the batch size was increased to 8, the batch size was increased to 2.8M tokens, and the maximum context length was 10,240 tokens. After training for 2 epochs in this stage, the speech model achieved stable speech understanding and generation capabilities while retaining the original text inference performance.

[0051] In steps 2-6, the large model alternates between four input-output modal configurations during training: (1) Question asked by voice → Answered by voice; (2) Question by voice → Answer by text; (3) Text question → Voice answer; (4) Text question → Text answer.

[0052] This multimodal interaction design enables large models to achieve voice and text interaction in both unimodal and crossmodal modes within a unified framework. It allows them to accept text or voice as input and generate text or voice as output within the unified framework, thus forming a true bimodal dialogue capability.

[0053] The two-stage pre-training strategy of this invention achieves a smooth transition from the pre-trained text model to the speech model through a freeze-thaw progressive training, and effectively prevents the degradation of text capabilities, providing key technical support for building a true end-to-end voice interaction system.

[0054] 5. End-to-end voice interaction process In summary, through the content of Sections 1-4 above, a well-trained speech-text model is obtained. In actual end-to-end voice interaction processes, only this trained speech model is needed. For example... Figure 3 As shown in Figure (c), the specific implementation steps include: The system acquires user-input speech signals and inputs them into a trained speech model, where a speech encoder converts them into a speech token sequence. This speech token sequence is then fed into the shared layer of the multimodal processing module for further processing. The output of the shared layer enters the speech processing branch to generate a probability distribution of the speech tokens. Finally, the speech decoder reconstructs the speech signal from the probability distribution of the speech tokens as the final response output. The entire process requires no intermediate text representation, achieving true end-to-end voice interaction.

[0055] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0057] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0058] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0060] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0061] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An end-to-end voice interaction method without text guidance, characterized in that, Includes the following steps: The process involves acquiring user-input speech signals, inputting them into a pre-trained speech model, and outputting reconstructed speech signals. The speech model includes a speech encoder, a multimodal processing module, and a speech decoder. The steps for outputting the reconstructed speech signals include: The input speech signal is converted into a speech token sequence using the speech encoder. The speech token sequence is input into the multimodal processing module for cross-modal semantic fusion and alignment of text and semantics, and the probability distribution of speech tokens is obtained through deep optimization. The probability distribution of the voice token is decoded using the voice decoder to obtain the reconstructed voice signal, thereby realizing the end-to-end voice interaction process. The speech model training process utilizes a text model and / or text data together for training.

2. The end-to-end voice interaction method without text guidance according to claim 1, characterized in that, The speech encoder employs a fully causal convolutional and Transformer neural network architecture.

3. The end-to-end voice interaction method without text guidance according to claim 1, characterized in that, The speech model is trained using a two-stage pre-training process: the first stage is the speech pre-training stage, and the second stage is the supervised fine-tuning stage.

4. The end-to-end voice interaction method without text guidance according to claim 3, characterized in that, The training steps in the speech pre-training phase include: Select a pre-trained text model as the base model; Construct a large-scale corpus containing mixed speech-text data, wherein the large-scale corpus includes two types of speech data: real speech data filtered using speech activity detection technology and speech data synthesized from text. The large-scale corpus is divided into two groups to obtain two sets of training data: initial interleaved speech-text training data and unsupervised speech training data. For the real speech data and text-synthesized speech data in the initial interleaved speech-text training data, a connection-time classification algorithm is used to align them with the text content in time. The time-aligned real speech and the text-synthesized speech data are segmented to obtain real speech segments and text-synthesized speech segments respectively. The real speech segments and text-synthesized speech segments are then arranged alternately to obtain the final interleaved speech-text training data. The basic model is introduced into the speech model to form a large speech-text model; Freeze all parameters of the text model in the speech-text large model, and use the final interleaved speech-text training data and unsupervised speech training data to complete the training process of the speech pre-training stage.

5. The end-to-end voice interaction method without text guidance according to claim 4, characterized in that, The training steps in the supervised fine-tuning phase include: The original question-and-answer text dataset is obtained, rewritten into speech, and then audio is synthesized using various text-to-speech methods to obtain the initial speech question-and-answer dataset. The rewriting process removes symbols or formats that are not suitable for audio synthesis, converts mathematical formulas into a readable form, simplifies lengthy content, and assigns tone and style. The word error rate of each voice question and answer data in the initial voice question and answer dataset is automatically compared with the corresponding original question and answer text. Voice question and answer data with a word error rate greater than a preset threshold are removed to obtain the final voice question and answer dataset. Unfreeze some or all parameters of the text model in the speech-text large model, fine-tune it using the final speech question-answering dataset, continuously introduce question-answering text data, and dynamically adjust the learning rate through a cosine learning rate scheduling strategy until the supervised fine-tuning training process is completed.

6. The end-to-end voice interaction method without text guidance according to claim 5, characterized in that, During the supervised fine-tuning phase, the speech-text large model employs the following four input-output modal configurations during training: (1) Question asked by voice → Answered by voice; (2) Question by voice → Answer by text; (3) Text question → Voice answer; (4) Text question → Text answer.

7. The end-to-end voice interaction method without text guidance according to claim 4, characterized in that, The network structure of the speech-text large model includes an encoder connected in series, a multimodal processing module consisting of multiple Transformer layers, and a decoder. The encoder includes a parallel text encoder and a speech encoder. The text encoder converts the input text data into a sequence of text tokens, and the speech encoder converts the input speech data into a sequence of speech tokens. The multimodal processing module includes a shared layer and two parallel branches: a text processing branch consisting of several Transformer layers and a speech processing branch consisting of several Transformer layers. The decoder includes a text decoder and the speech decoder.

8. The end-to-end voice interaction method without text guidance according to claim 7, characterized in that, In the multimodal processing module, the shared layer is used to receive text token sequences and speech token sequences. Through training methods using self-attention mechanism, next token prediction and cross-entropy loss, the speech model can achieve cross-modal semantic fusion and alignment of text and semantics without explicit alignment annotation, and obtain aligned text token sequences and aligned speech token sequences respectively. The text processing branch is used to optimize the aligned text token sequence to obtain the probability distribution of the text tokens; The speech processing branch is equivalent to optimizing the aligned speech token sequence to obtain the probability distribution of the speech tokens.

9. The end-to-end voice interaction method without text guidance according to claim 8, characterized in that, The layer structure partitioning between the shared layer and the two parallel branches in the multimodal processing module is performed through the following steps: Given a text and its corresponding speech sequence, convert it into a text token sequence and a speech token sequence, respectively. In the multimodal processing module, the hidden state of each layer of text tokens and the hidden state of each layer of voice tokens are obtained layer by layer from the text token sequence and the voice token sequence. Calculate the cosine similarity matrix between the hidden state of the text token and the hidden state of the voice token at each layer; The cosine similarity matrix is ​​used as the input of the dynamic time warping algorithm. The start and end points of the aligned text token sequence and the aligned speech token sequence are constrained, and the path is limited to satisfy the time monotonicity and continuity constraints. The cumulative matching cost of each position in the cosine similarity matrix is ​​calculated step by step using dynamic programming. Then, the cosine similarity matrix is ​​backtracked to select the path with the optimal cumulative matching cost as the optimal alignment path between speech and text. The multimodal processing module is divided according to the optimal alignment path to obtain a shared layer and two parallel branches.

10. The end-to-end voice interaction method without text guidance according to claim 1, characterized in that, The speech decoder employs a stream-matching-based generative model architecture, and the steps for obtaining the reconstructed speech signal include: The aligned speech token sequence is mapped to a high-dimensional Mel spectrum representation by a continuous normalization stream, and then converted into a time-domain waveform by a vocoder as the reconstructed speech signal.

Images