Time-length-controllable end-to-end speech translation method and translation system

Abstract

The invention discloses an end-to-end speech translation method and translation system capable of controlling duration, which effectively reduces the translation delay by introducing a speech end-to-end scheme. By constructing a brand new token and introducing a token alignment scheme, the length of a translation result is effectively controlled; by controlling the token duration and adding a length control variable, cross-language timbre and rhythm cloning is guided, so that relatively excellent cross-language timbre and rhythm cloning is achieved.

Description

Technical Field

[0001] This invention relates to the field of speech translation technology, and in particular to an end-to-end speech translation method and system with controllable duration. Background Technology

[0002] As an important application at the intersection of artificial intelligence and natural language processing, the development of speech translation models reflects humanity's unremitting efforts to overcome language barriers and achieve barrier-free communication.

[0003] From the early days of simply connecting three independent modules—ASR (Automatic Speech Recognition), MT (Machine Translation), and TTS (Text-to-Speech)—to forming a "cascaded system," to the recent cascading of speech-to-text translation models and speech synthesis models, and finally to the latest end-to-end neural network model for speech-to-speech translation.

[0004] As an important branch of artificial intelligence and speech synthesis, speech cloning technology is mainly characterized by large models and zero-shot technology, specifically the model's ability to learn and continue writing from the input speech.

[0005] In scenarios that combine voice cloning with multilingual voice translation, existing voice translation technologies can only translate the content, but cannot inherit information related to the speaker (such as timbre, rhythm, etc.), nor can they control the translated content and the length of the speech. Moreover, existing voice cloning technology is also unable to effectively clone voices with timbre, rhythm, etc., across languages.

[0006] Existing voice cloning technology has the following main problems: 1) The ASR-MT-TTS cascaded solution has a relatively large overall translation delay; 2) Traditional MT solutions cannot control the length of translation results, often resulting in translations that are too long or too short; 3) Traditional TTS voice cloning cannot effectively synchronize timbre and rhythm because the pronunciation and rhythm rules of different languages ​​are inconsistent after crossing languages. Summary of the Invention

[0007] The purpose of this invention is to provide an end-to-end speech translation method and system with controllable duration. This invention has the advantage of achieving relatively better cross-language timbre and prosody cloning translation.

[0008] The technical solution of this invention: an end-to-end speech translation method with controllable duration, comprising the following steps: S1, Voice input; S2. Semantic Extraction: Extract the input speech sequence using a semantic extractor; S3. Speaker Information Extraction: Extract speaker reference information using a speaker information extractor; S4. Input the speech sequence into the LLM model for prediction and output the token sequence; S5. Output the token sequence, combine it with the speaker reference information, and input the TTS LLM model to predict the speech token sequence. S6. Speech generation: The speech token sequence is converted into translated speech using a vocoder and then pronounced.

[0009] In the aforementioned end-to-end speech translation method with controllable duration, the semantic extraction described in S2 is as follows: The encoder's capabilities are trained using a multi-task training framework, guided by the partitioning of token, speaker, and language information. After training, the output of linear & layer normalization is used to build a semantic extractor to extract the input speech sequence that conforms to the LLM tokenizer.

[0010] In the aforementioned end-to-end speech translation method with controllable duration, the semantic extractor is an encoder-decoder architecture, as detailed below: The encoder uses a neural network architecture model based on the conformer structure, the conformer encoder, and the decoder uses a neural network architecture model based on the transformer structure, the transformer decoder. Linear & Layer Normalization is provided between the conformer encoder and the transformer decoder.

[0011] In the aforementioned end-to-end speech translation method with controllable duration, the speaker information extractor in S3 is a sound codec, specifically as follows: The speaker information extractor uses the codec module in the SoundStream structure to obtain an implicit representation vector containing speaker information after RVQ from the input speech. This implicit representation vector is used as the speaker reference information input to the TTS LLM model.

[0012] In the aforementioned end-to-end speech translation method with controllable duration, the LLM model described in S4 is used to predict the teoken sequence of the target language, as detailed below: The LLM model takes into account data including translation task control, input language, target language, target length, and input token sequence. The translation task control is set to predefined values; The input language and target language are obtained by learning the model's emb vectors; The target length constructs a tokenizer with phonetic elements. Based on the phonetic correspondence, it is constructed through a linear structure and combined with token sequence information to learn the correspondence between token sequence lengths in different languages.

[0013] In the aforementioned end-to-end speech translation method with controllable duration, the method for training and obtaining the target length is as follows: , , , in Here, is the weight matrix, a and b represent the input language and output language, seq represents the token sequence, Lemb represents the language representation vector, and len represents the length representation vector. For training purposes.

[0014] In the aforementioned end-to-end speech translation method with controllable duration, the speech generation described in S6 specifically includes: Speech generation is achieved using a vocoder based on a flow-matching + HiFi-GAN structure; Flow-matching enables the conversion from TTS LLM predicted token sequences to speech Mel spectrograms; HiFi-GAN enables the conversion of speech mel spectrum to speech signal.

[0015] An end-to-end speech translation system with controllable duration is provided. The system includes a speaker information extractor, a semantic extractor, an LLM model, a TTS LLM model, and a vocoder. The system applies the above translation methods to achieve cross-language translation.

[0016] Compared with the prior art, this application has the following advantages: 1) Voice input, voice output, end-to-end voice translation solution, effectively reducing translation latency; 2) A tokenizer with phonetic elements is proposed, and a token alignment algorithm between different languages ​​is introduced to control the translation length; 3) A speech cloning algorithm using a length-controlled token sequence is proposed to improve the ability to clone speech timbre and prosody across languages.

[0017] In summary, this application effectively reduces translation latency by introducing an end-to-end speech solution; effectively controls the length of the translation result by constructing a novel tokenizer and introducing a token alignment scheme; and guides cross-lingual timbre and prosody cloning by controlling token duration and incorporating a length control variable, thereby achieving superior cross-lingual timbre and prosody cloning. Therefore, this invention has the advantage of being able to achieve relatively better cross-language timbre and prosody cloning translation. Attached Figure Description

[0018] Figure 1 This is a flowchart of the translation method of the present invention; Figure 2 This is a structural diagram of the semantic extractor of the present invention; Figure 3 This is a schematic diagram of the LLM model of the present invention; Figure 4 This is a schematic diagram of the speaker information extractor of the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0020] Example. An end-to-end speech translation method with controllable duration, such as... Figure 1 As shown, the process includes the following: S1, Voice input; S2. Semantic Extraction: Extract the input speech sequence using a semantic extractor; S3. Speaker Information Extraction: Extract speaker reference information using a speaker information extractor; S4. Input the speech sequence into the LLM model for prediction and output the token sequence; S5. Output the token sequence, combine it with the speaker reference information, and input the TTS LLM model to predict the speech token sequence. S6. Speech generation: The speech token sequence is converted into translated speech using a vocoder and then pronounced.

[0021] The semantic extraction described in S2 is as follows: The encoder's capabilities are trained using a multi-task training framework, guided by the partitioning of token, speaker, and language information. After training, the output of linear & layer normalization is used to build a semantic extractor to extract the input speech sequence that conforms to the LLM tokenizer.

[0022] The semantic extractor is an encoder-decoder architecture, such as... Figure 2 As shown, the specific content is as follows: The encoder uses a neural network architecture model based on the conformer structure, the conformer encoder, and the decoder uses a neural network architecture model based on the transformer structure, the transformer decoder. Linear & Layer Normalization is provided between the conformer encoder and the transformer decoder.

[0023] In the speaker information extraction described in S3, the speaker information extractor is a sound codec, such as... Figure 4 The specific content is as follows: The speaker information extractor uses the codec module in the SoundStream structure to obtain an implicit representation vector containing speaker information after RVQ from the input speech. This implicit representation vector is used as the speaker reference information input to the TTS LLM model.

[0024] The LLM model described in S4 is used to predict teoken sequences in the target language, such as Figure 3 The specific details are as follows: The LLM model takes into account data including translation task control, input language, target language, target length, and input token sequence. The translation task control is set to predefined values; The input language and target language are obtained by learning the model's emb vectors; The target length constructs a tokenizer with phonetic elements. Based on the phonetic correspondence, it is constructed through a linear structure and combined with token sequence information to learn the correspondence between token sequence lengths in different languages.

[0025] The training method for obtaining the target length is as follows: , , , in Here, is the weight matrix, a and b represent the input language and output language, seq represents the token sequence, Lemb represents the language representation vector, and len represents the length representation vector. For training purposes.

[0026] The speech generation described in S6 specifically includes: Speech generation is achieved using a vocoder based on a flow-matching + HiFi-GAN structure; Flow-matching enables the conversion from TTS LLM predicted token sequences to speech Mel spectrograms; HiFi-GAN enables the conversion of speech mel spectrum to speech signal.

[0027] An end-to-end speech translation system with controllable duration is provided. The system includes a speaker information extractor, a semantic extractor, an LLM model, a TTS LLM model, and a vocoder. The system applies the above translation methods to achieve cross-language translation.

Claims

1. An end-to-end speech translation method with controllable duration, characterized in that, The process includes the following steps: S1, Voice input; S2. Semantic Extraction: Extract the input speech sequence using a semantic extractor; S3. Speaker Information Extraction: Extract speaker reference information using a speaker information extractor; S4. Input the speech sequence into the LLM model for prediction and output the token sequence; S5. Output the token sequence, combine it with the speaker reference information, and input the TTS LLM model to predict the speech token sequence. S6. Speech generation: The speech token sequence is converted into translated speech using a vocoder and then pronounced.

2. The end-to-end speech translation method with controllable duration according to claim 1, characterized in that, The semantic extraction described in S2 is as follows: The encoder's capabilities are trained using a multi-task training framework, guided by the partitioning of token, speaker, and language information. After training, the output of linear & layer normalization is used to build a semantic extractor to extract the input speech sequence that conforms to the LLM tokenizer.

3. The end-to-end speech translation method with controllable duration according to claim 2, characterized in that, The semantic extractor uses an encoder-decoder architecture, as detailed below: The encoder uses a neural network architecture model based on the conformer structure, the conformer encoder, and the decoder uses a neural network architecture model based on the transformer structure, the transformer decoder. Linear and LayerNormalization are implemented between the conformer encoder and the transformer decoder.

4. The end-to-end speech translation method with controllable duration according to claim 1, characterized in that, In the speaker information extraction described in S3, the speaker information extractor is a sound codec, specifically: The speaker information extractor uses the codec module in the SoundStream structure to obtain an implicit representation vector containing speaker information after RVQ from the input speech. This implicit representation vector is used as the speaker reference information input to the TTS LLM model.

5. The end-to-end speech translation method with controllable duration according to claim 1, characterized in that, The LLM model described in S4 is used to predict teoken sequences in the target language, and the specific details are as follows: The LLM model takes into account data including translation task control, input language, target language, target length, and input token sequence. The translation task control is set to predefined values; The input language and target language are obtained by learning the model's emb vectors; The target length constructs a tokenizer with phonetic elements. Based on the phonetic correspondence, it is constructed through a linear structure and combined with token sequence information to learn the correspondence between token sequence lengths in different languages.

6. The end-to-end speech translation method with controllable duration according to claim 5, characterized in that, The training method for obtaining the target length is as follows: ， ， ， in Here, is the weight matrix, a and b represent the input language and output language, seq represents the token sequence, Lemb represents the language representation vector, and len represents the length representation vector. For training purposes.

7. The end-to-end speech translation method with controllable duration according to claim 1, characterized in that, The speech generation described in S6 specifically includes: Speech generation is achieved using a vocoder based on a flow-matching + HiFi-GAN structure; Flow-matching enables the conversion from TTS LLM predicted token sequences to speech Mel spectrograms; HiFi-GAN enables the conversion of speech mel spectrum to speech signal.

8. A controllable-duration end-to-end speech translation system according to any one of claims 1-7, characterized in that: The system includes a speaker information extractor, a semantic extractor, an LLM model, a TTS LLM model, and a vocoder; the system applies the above translation methods to achieve cross-language translation.

Images