A method and system for training an end-to-end speech translation model

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a training method and a training system of an end-to-end speech translation model.The method of the present application comprises: providing a pre-trained large language model capable of processing wordpiece sequences; constructing a plurality of training data instances, each instance comprising an input speech signal and a corresponding target wordpiece sequence; for each training data instance, providing the wordpiece representation of the input speech signal as input to the pre-trained large language model, and optimizing the model parameters to predict the corresponding target wordpiece sequence in an autoregressive manner to train the model, so that the model learns a thinking chain including speech recognition, text generation and speech synthesis in a single end-to-end differentiable framework.The method combines the structured reasoning advantages of traditional stage connection systems and the performance advantages of end-to-end training, and at the same time reduces the dependence on massive labeled data through post-training of the thinking chain, saving computing resources.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a training method and system for an end-to-end speech translation model. Background Technology

[0002] With the rapid development of artificial intelligence technology, voice interaction systems have become an important means of human-computer interaction, widely used in intelligent customer service, smart homes, in-vehicle systems, and professional auxiliary tools. Existing voice processing technologies can be mainly divided into two categories: traditional cascaded systems and end-to-end systems, but both of these technical approaches have inherent drawbacks.

[0003] Traditional voice dialogue systems typically employ a cascaded architecture, which consists of multiple independently optimized modules connected in series. A typical process is as follows: the Automatic Speech Recognition (ASR) module converts the user's speech into text; the Natural Language Understanding (NLU) and Dialogue Management module performs intent recognition and semantic analysis on the text and generates corresponding text responses; finally, the Text-to-Speech (TTS) module converts the text responses into speech output.

[0004] While this modular architecture is logically clear, it leads to the accumulation and propagation of errors at each level. Any recognition error in the automatic speech recognition module is unconditionally passed to the downstream natural language understanding module, causing biases in subsequent semantic understanding and decision-making. These errors cannot be corrected during processing, ultimately severely impacting the accuracy and fluency of the output. Furthermore, the cascaded system compresses rich speech signals into a single text representation during processing, losing a significant amount of non-textual information crucial for a deeper understanding of user intent, thus limiting the naturalness and depth of the interaction.

[0005] To address the issues of error accumulation and information loss in cascaded systems, end-to-end models have been proposed in the research community. These models typically employ a single deep neural network to directly map the input speech signal to the final output, such as directly translating source language speech into target language text or directly generating response speech. Through joint optimization, end-to-end models can learn the complex mapping from acoustic features to semantic representations, avoiding the rigid constraints of intermediate text representations, thereby reducing information loss and error propagation.

[0006] However, existing end-to-end speech models, by directly maximizing the posterior probability distribution, lack explicit and structured intermediate reasoning processes. This results in semantically inconsistent responses when dealing with tasks requiring multi-step logic, domain knowledge, or complex reasoning, making it difficult to provide professional and accurate guidance. For example, in specialized scenarios such as hospital logistics management, the system not only needs to understand requests but also their underlying logic, invoke relevant procedures, and generate structured solutions—tasks that traditional end-to-end models struggle to accomplish.

[0007] In summary, existing technologies still have significant gaps in realizing efficient and truly end-to-end voice dialogue systems with complex reasoning capabilities. On the one hand, cascaded systems have clear reasoning structures but accumulate errors; on the other hand, traditional end-to-end systems offer superior performance but lack reasoning capabilities. There is a need for a training method and system that combines the structured reasoning advantages of cascaded systems with the end-to-end differentiability of end-to-end models, and achieves this in a data-efficient manner. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a training method and system for an end-to-end speech translation model. This method is based on a pre-trained text-speech large language model and employs a thought chain post-training strategy to enable the model to learn structured reasoning capabilities while maintaining end-to-end differentiability.

[0009] This invention provides a training method for an end-to-end speech translation model, comprising the following steps: providing a pre-trained large language model capable of processing lexical sequences; constructing multiple training data instances, each instance including an input speech signal and a corresponding target lexical sequence, wherein the target lexical sequence sequentially includes: a first text lexical sequence representing the text transcription of the input speech signal, a second text lexical sequence representing the text response to the text transcription, and an audio lexical sequence representing the synthesized speech version of the text response; for each training data instance, providing the lexical representation of the input speech signal as input to the pre-trained large language model, and optimizing the model parameters to predict the corresponding target lexical sequence in an autoregressive manner to train the model, thereby enabling the model to learn a thought chain including speech recognition, text generation, and speech synthesis within a single end-to-end differentiable framework.

[0010] Preferably, the target lexical sequence further includes: a first special task identifier lexical placed before the first text lexical sequence, a second special task identifier lexical placed before the second text lexical sequence, and a third special task identifier lexical placed before the audio lexical sequence.

[0011] Preferably, the special task identifier lexical is used to indicate to the model the type of lexical sequence to be generated subsequently. Specifically, the lexical representation of the input speech signal and the first special task identifier lexical are provided as an initial context to the large language model to induce the model to autoregressively generate the first text lexical sequence; the generated first text lexical sequence and the second special task identifier lexical are appended to the initial context to form an updated context, and the updated context is provided to the large language model to induce the model to autoregressively generate the second text lexical sequence; the generated second text lexical sequence and the third special task identifier lexical are appended to the updated context to form a final context, and the final context is provided to the large language model to induce the model to autoregressively generate the audio lexical sequence.

[0012] Preferably, the method further includes using a unified lexer to convert the input speech signal, the text lexes, and the audio lexes into a common lexical vocabulary. Specifically, this includes: for the input speech signal and the audio lexes, processing the corresponding continuous audio waveforms through an audio encoder and converting the audio waveforms into discrete audio lexes using residual vector quantization; for the text lexes, processing the corresponding text strings through a text lexer and converting the text strings into discrete sub-word text lexes using byte-pair encoding.

[0013] Preferably, the step of using residual vector quantization to convert the audio waveform into discrete audio tokens specifically includes: converting the audio waveform into an initial continuous feature vector sequence through the audio encoder; for each feature vector in the sequence, processing is performed through a multi-stage quantization process, wherein each stage includes: comparing the input vector of the current stage with a preset codebook to find the closest codebook vector; using the index of the found codebook vector as the discrete audio token output of the current stage; calculating the difference between the input vector of the current stage and the found codebook vector to generate a residual vector for the next stage; and combining the discrete audio token index sequences generated in all stages to form the final audio token sequence.

[0014] Preferably, the step of using byte-pair encoding technology to convert the text string into discrete sub-word text units specifically includes: initializing a vocabulary using all individual characters in a text corpus, wherein the words in the corpus are first split into character sequences and appended with a special word ending symbol; iteratively performing a predetermined number of merging operations, or until the target vocabulary size is reached, wherein each merging operation includes: identifying the pair of adjacent units that occur most frequently in the current representation of the corpus; merging the identified unit pairs into a new, single sub-word unit; adding the new sub-word unit to the vocabulary; and replacing all occurrences of the identified unit pairs in the corpus with the new sub-word unit.

[0015] Preferably, the pre-trained large language model is a Transformer model that contains only a decoder.

[0016] Preferably, the post-training process employs a teacher-mandated strategy and uses the cross-entropy loss function to calculate the prediction loss for the next lexical term.

[0017] Another aspect of the present invention provides a training system for an end-to-end speech translation model, comprising a data input module, a data storage module, a unified lexicalization module, a data construction module, and a training module; the data input module is used to receive speech signals, target text, and target speech; the data storage module is used to store a pre-trained model and a training dataset; the unified lexicalization module is used to convert the speech signals, target text, and target speech into a unified, discrete lexical space; the data construction module is used to assemble training instances, wherein each instance includes a lexicalized input speech signal and a target lexical sequence; the training module is used to provide the training instances to the pre-trained large language model, perform a post-training process, which includes calculating a next lexical prediction loss covering the entire target lexical sequence, and updating the parameters of the model through backpropagation.

[0018] Compared to existing technologies, the advantages of this invention lie in the fact that by introducing two intermediate inference steps—automatic speech recognition transcription and text response—into the training objective, the model can learn a structured thought process. This results in a more rigorous and semantically coherent final speech response, making it particularly suitable for vertical fields requiring specialized knowledge and complex decision-making.

[0019] The entire training process is completed within a single model using a single loss function. This means that during backpropagation, the error gradient from the final speech output can be directly passed to and adjust the model parameters responsible for the automatic speech recognition and text generation steps. This end-to-end differentiability solves the error accumulation problem caused by hard boundaries between modules in traditional cascaded systems.

[0020] The post-training strategy leverages the powerful foundational capabilities of large-scale pre-trained models (such as speech recognition, language understanding, and speech synthesis). Through post-training via the thought chain, these general capabilities can be transferred to specific complex dialogue tasks with only a relatively small amount of labeled data, reducing reliance on massive amounts of labeled data and saving computational resources. Attached Figure Description

[0021] Figure 1 This is a system architecture diagram of a training system for an end-to-end speech translation model proposed in this invention; Figure 2 This is a flowchart of a training method for an end-to-end speech translation model proposed in this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] Example 1: Refer to Figure 1 This illustrates the overall architecture of a training system for an end-to-end speech translation model according to an embodiment of the present invention. The system aims to fine-tune a large, pre-trained speech-language model using a thought chain post-training method to enable it to acquire the structured reasoning capabilities required for handling complex dialogue tasks.

[0024] At the core of the system is a pre-trained large-scale speech model. In a preferred embodiment, this model is a large language model based on the Transformer architecture and containing only decoders. This model has undergone large-scale multi-task pre-training and possesses powerful fundamental capabilities, including but not limited to: speech signal recognition as text (ASR), generating coherent text paragraphs based on text instructions (language modeling), and text synthesis into natural language (TTS). The model can process and generate word sequences that mix audio and text within its unified representation space.

[0025] The system also includes the following modules: Input module: responsible for receiving raw audio waveform data, such as voice input from the user.

[0026] Data storage module: used to store the pre-trained large speech model and training dataset.

[0027] Unified Lexicalization Module: Responsible for converting all types of data received by the system (including input speech, target text, and target speech) into discrete lexical sequences that can be understood by the pre-trained model.

[0028] The data construction module is responsible for building structured training instances for post-training of the thought chain based on the original dataset. For each training sample, it generates a data pair containing the input speech and the target sequence of the thought chain.

[0029] Training module: Responsible for receiving data processed by the data construction module and the unified lexicalization module, performing forward propagation of the model, calculating the loss function based on the next lexical prediction, and updating the weights of the pre-trained large speech model through the backpropagation algorithm.

[0030] Output module: After the model training is completed or during the inference phase, it is responsible for decoding the discrete audio words generated by the model into continuous speech waveforms, which serve as the final output of the system.

[0031] Example 2:

[0032] This embodiment provides a detailed description of the unified lexicalization module. This module is used to map three different types of information—audio, text, and task-specific identifiers—to a unified lexical space, thereby constructing a hybrid sequence that can guide the model in thought chain reasoning.

[0033] The raw speech waveform received by the input module First, the audio signal is processed using an audio encoder (e.g., a pre-trained audio codec model). This encoder uses residual vector quantization to segment the continuous audio signal into frames, generating one or more discrete audio tokens for each frame. All generated discrete audio token index sequences are then combined to form the final audio token sequence. Specifically, the audio signal is converted into an initial continuous feature vector sequence by the audio encoder. For each feature vector in the sequence, a multi-stage quantization process is performed, where each stage includes: comparing the input vector of the current stage with a preset codebook to find the closest codebook vector; using the index of the found codebook vector as the discrete audio token output of the current stage; calculating the difference between the input vector of the current stage and the found codebook vector to generate a residual vector for the next stage; and combining the discrete audio token index sequences generated in all stages to form the final audio token sequence.

[0034] The target text (such as ASR transcription results or text responses) is processed using a standard text lexer, such as byte-pair encoding. This process breaks down the text string into a series of sub-word text lexical units. Specifically: a vocabulary is initialized using all individual characters in a text corpus, where words are first split into character sequences and appended with a special suffix; a predetermined number of merge operations are iteratively performed, or until the target vocabulary size is reached, wherein each merge operation includes: identifying the most frequent pair of adjacent lexical units in the current representation of the corpus; merging the identified lexical pair into a new, single sub-word lexical unit; adding the new sub-word lexical unit to the vocabulary; and replacing all occurrences of the identified lexical pairs in the corpus with the new sub-word lexical unit.

[0035] Furthermore, the unified lexicalization module's vocabulary includes a set of predefined special lexical units that do not correspond to any specific speech or text content, but rather serve as meta-instructions to guide the model's generation process. For example, The identifier prompts the model to generate the text results of the speech recognition next; The identifier prompting model should generate an understood text response based on the preceding context; The identifier then prompts the model to begin generating audio lexical units corresponding to the final text response.

[0036] This unified lexicalization module allows a complete training target sequence to be constructed into a seamless, hybrid-type lexical stream. For example, for a training sample, its target sequence, after lexicalization, will appear as follows: In the form of, and It is a text word sequence. It is an audio word sequence. This structured data representation is the foundation for implementing post-training of the thought chain.

[0037] Example 3:

[0038] This embodiment describes the process of the post-training method of the thought chain in this scheme. This method changes the traditional training objective, no longer directly maximizing... Instead, it breaks down the problem and trains a model to predict a complete sequence that includes intermediate reasoning steps, i.e., maximizing the joint probability. This is achieved by modeling a structured target sequence in an autoregressive manner.

[0039] The method specifically includes the following steps: S301, Data preparation: First, prepare a voice input containing triples. ASR transcription and text response The dataset can come from annotations of real-world scenarios or can be synthesized semi-automatically using existing toolchains (e.g., using a high-performance ASR model and a powerful text LLM), thereby reducing the cost of data annotation.

[0040] S302. Constructing Training Sequences: For each data triplet, the data construction module and the unified lexicalization module work together to construct input-output pairs for model training. The model's input is the lexicalized speech signal. The model needs to predict complete thought chain sequences that have been lemmatized and structured. ,Right now .

[0041] S303, Model Training: The training module performs the following operations: transforms the lexicalized input speech... Feed it into a pre-trained large-scale speech model.

[0042] The model generates the target sequence in an autoregressive manner. During training, a teacher-forced strategy is used, meaning that at each time step, the model predicts the next lexical unit based on the true prefix, rather than its own prediction from the previous step.

[0043] The computational model predicts the lexical distribution and the actual target sequence. The loss between word terms. Typically, the standard cross-entropy loss function is used, which is applied to the target sequence. The summation is performed on all the lexical units.

[0044] S304, Parameter Update: Based on the calculated loss, the gradient is calculated using the backpropagation algorithm, and the model parameters are updated.

[0045] By processing the entire structured sequence in step S303 During training, the model learns that when it sees... When prompted, it must learn to perform ASR tasks; when it sees When encountering the previously generated ASR text, it must learn to perform natural language processing and text generation tasks; finally, when seeing... Given a complete text context, it must learn to perform TTS tasks. Furthermore, because the entire process is driven by a single model and a single loss function, it is fully end-to-end differentiable. This means that if the final synthesized speech... There is an error in the process, and its gradient can be propagated backtracking to adjust the generator responsible for generating the algorithm. even The model parameters are determined to achieve global optimization.

[0046] Example 4: This example specifically demonstrates the inference data flow of the model trained by the method of this invention in a practical application. This process is a step-by-step autoregressive generation process, clearly reflecting the thought chain learned by the model.

[0047] S401: The system receives user voice input. The unified lexicalization module then converts it into an audio lexical sequence.

[0048] S402: Lexicalization and task prompt The model is input as the initial context. The model then begins to generate ASR transcription results word-by-word. This continues until a sequence terminator is generated.

[0049] S403: The generated... and the new task prompt It is then appended to the current context. Based on this richer context, the model continues to generate text responses word by word. .

[0050] S404: The generated... and the final task prompt Attached to the context. The model generates the final audio lexical sequence based on the complete context (including the original speech information, ASR transcription, and text response). .

[0051] S405: The output module will output the audio word sequence. The audio is decoded into a final waveform and played back to the user.

[0052] This process makes the model's "thinking" process transparent and controllable. The intermediate outputs at each step ( and All of these are inspectable, which greatly facilitates system debugging and iteration.

[0053] Example 5: This example illustrates an exemplary computing device that can be used to implement the training system described in this invention or to run a trained model. The computing device can be one or more servers, a workstation, or a distributed computing cluster. The computing device includes at least one processor (such as a CPU, GPU, or dedicated AI accelerator), memory (such as RAM, ROM), storage device (such as a hard disk, solid-state drive), and a network interface. The memory stores the operating system and application programs. In this example, the application programs may include instructions for implementing functions such as a data construction module, a unified lexicalization module, and a training module. The processor executes these instructions to complete the training method described in this invention. The storage device can be used to persistently store the pre-trained model, training dataset, and additional data generated during training.

[0054] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0055] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for an end-to-end speech translation model, characterized in that, Includes the following steps: Provides a pre-trained large language model capable of processing word sequences; Multiple training data instances are constructed, each instance including an input speech signal and a corresponding target word sequence, wherein the target word sequence includes, in order: a first text word sequence representing the text transcription of the input speech signal, a second text word sequence representing the text response to the text transcription, and an audio word sequence representing the synthesized speech version of the text response; For each training data instance, the lexical representation of the input speech signal is provided as input to the pre-trained large language model for post-training, and the model parameters are optimized to predict the corresponding target lexical sequence in an autoregressive manner to train the model. This allows the model to learn a thought chain including speech recognition, text generation, and speech synthesis within a single end-to-end differentiable framework. The target lexical sequence further includes: a first special task identifier lexical placed before the first text lexical sequence, a second special task identifier lexical placed before the second text lexical sequence, and a third special task identifier lexical placed before the audio lexical sequence. The method also includes using a unified lexer to convert the input speech signal, the text lexical units, and the audio lexical units into a common lexical vocabulary, specifically including: For the input speech signal and the audio words, the corresponding continuous audio waveform is processed by an audio encoder, and the audio waveform is converted into discrete audio words using residual vector quantization technology. For each text lexicon, the corresponding text string is processed by a text lexiconizer, and byte-pair encoding is used to convert the text string into discrete sub-word text lexicons. The special task identifier lexicon is used to indicate to the model the type of the subsequent lexicon sequence to be generated, specifically: The lexical representation of the input speech signal and the first special task identifier lexical are provided as initial context to the large language model, so as to enable the model to generate the first text lexical sequence autoregressively. The first text lexical sequence and the second special task identifier lexical sequence generated above are appended to the initial context to form an updated context, and the updated context is provided to the large language model to enable the model to generate the second text lexical sequence autoregressively. The second text lexical sequence and the third special task identifier lexical sequence generated above are appended to the updated context to form a final context, and the final context is provided to the large language model to enable the model to generate the audio lexical sequence autoregressively.

2. The training method for an end-to-end speech translation model according to claim 1, characterized in that, The step of converting the audio waveform into discrete audio words using residual vector quantization technology specifically includes: The audio waveform is converted into an initial continuous feature vector sequence by the audio encoder; For each feature vector in the sequence, a multi-stage quantization process is performed, where each stage includes: The input vector at the current stage is compared with a preset codebook to find the closest codebook vector; The index of the found codebook vector is used as the output of the discrete audio lexical for the current stage; Calculate the difference between the input vector at the current stage and the found codebook vector to generate a residual vector for the next stage; The discrete audio lexical index sequences generated in all stages are combined to form the final audio lexical sequence.

3. The training method for an end-to-end speech translation model according to claim 2, characterized in that, The step of using byte-pair encoding technology to convert the text string into discrete sub-word text units specifically includes: A vocabulary is initialized using all the individual characters in a text corpus, wherein the words in the corpus are first split into character sequences and appended with a special word ending symbol; Iteratively perform the merge operation a predetermined number of times, or until the target vocabulary size is reached, wherein each merge operation includes: In the current representation of the corpus, identify the pair of adjacent word elements that appear most frequently; The identified word pairs are merged into a new, single sub-word word; Add the new sub-word lexicographical element to the vocabulary; In the corpus, all occurrences and identified word pairs are replaced with the new sub-words.

4. The training method for an end-to-end speech translation model according to claim 1, characterized in that, The pre-trained large language model is a Transformer model that only contains a decoder.

5. The training method for an end-to-end speech translation model according to claim 1, characterized in that, The post-training process employs a teacher-mandated strategy and uses the cross-entropy loss function to calculate the prediction loss for the next word.

Images