Audio translation method and device based on large language model

Abstract

The invention discloses a speech translation method and device based on a large language model, and the method comprises the steps: collecting a training corpus, and obtaining an audio and a corresponding text in a training stage, so as to extract the spectrum features of the audio; semantic features and global acoustic features of the spectrum features are extracted and coded; matching the semantic feature codes of the corresponding texts through a large number of translations to train a semantic feature translation large model so as to generate translated semantic feature codes from the semantic feature codes; training a vocoder basic model by using global acoustic feature codes and semantic feature codes of audios corresponding to a large number of texts; and finely adjusting the vocoder basic model through global acoustic feature information and semantic features of a small amount of audio of the target speaker to obtain a vocoder. According to the method, the problems of high processing delay, weak context understanding ability, complex system integration and the like of a traditional speech translation mode are solved, smooth and accurate translation audio can be synthesized even if the corpus of a certain language of a target speaker is few, and the consistency of timbres can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, in particular to cross-language speech translation, and more specifically to an audio translation method and device based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence, speech recognition, and natural language processing technologies, real-time speech translation systems are showing broad application prospects in scenarios such as cross-language communication, international conferences, and education and training. Currently, most speech translation systems typically use a three-stage process: speech recognition - text translation - speech synthesis. However, this traditional process suffers from high processing latency, poor contextual understanding, and complex system integration, making it difficult to meet the requirements of real-time communication scenarios with high timeliness and accuracy. Furthermore, existing technologies still face significant challenges in multi-speaker environments, dialect recognition, and support for low-resource languages.

[0003] Therefore, there is an urgent need for a real-time speech translation method and device that can reduce latency, improve translation accuracy, and have stronger adaptability to meet the growing demand for cross-language voice communication. Summary of the Invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a speech translation method and device based on a large language model to solve the problem of poor translation effect caused by information loss between modules due to too many steps in the current sound translation process, and improves the accuracy of speech translation through a large model.

[0005] The present invention solves the above problems through the following technical solutions:

[0006] A speech translation method based on a large language model, with a training phase and an inference phase;

[0007] The training phase includes the following steps:

[0008] S11. Collect training corpus, including audios of different speakers and corresponding texts, and extract spectral features of the audios through a spectral feature model;

[0009] S12. Train a semantic feature extraction model using the spectral features of text and audio, and remove the influence of speaker characteristics by gradient inversion of the speaker classification model during training.

[0010] S13, extracting and encoding features from the spectral features using the trained semantic feature extraction model to obtain semantic feature codes;

[0011] S14. extracting and encoding global acoustic features from the spectral features using a global acoustic feature extraction model to obtain a global acoustic feature code;

[0012] S15. Training a semantic feature translation model by translating a large number of semantic feature codes of the corresponding texts, so as to generate translated semantic feature codes from the semantic feature codes by the semantic feature translation model;

[0013] S16. Using global acoustic feature encoding and semantic feature encoding of a large amount of text corresponding to audio to train a vocoder basic model;

[0014] S17. Fine-tune the vocoder base model using global acoustic feature information and semantic features of a small amount of audio of the target speaker to obtain a vocoder.

[0015] As a further improvement, in S11, language labels and speaker labels of different speakers are also obtained, wherein the language labels are used to train the semantic feature translation model, and the speaker labels are used to train the semantic feature extraction model.

[0016] As a further improvement, in S11, the training corpus includes but is not limited to a public speech synthesis training dataset or a self-recorded speech synthesis training dataset; the extracted audio spectrum features include at least linear spectrum features and Mel spectrum features.

[0017] As a further improvement, in S12, the specific method is:

[0018] The semantic feature extraction model is trained based on the spectral features and text information of the training audio combined with semantic features. The network structure of the semantic feature extraction model includes an attention mechanism and a TDNN model.

[0019] During the training of the semantic feature extraction model, the latent variable features of the encoded text information are subjected to gradient reversal recognition by the speaker classification model, and combined with the speaker label, the error adjustment network is fed back through the classification loss function to decouple the speaker features from the semantic features.

[0020] As a further improvement, in S13, the semantic features extracted from the hidden layer of the semantic feature extraction model are encoded. Encoding refers to generating a format required for the large translation model.

[0021] As a further improvement, in S15, the semantic feature translation large model network structure includes but is not limited to the qwen and llama large model network structures.

[0022] As a further improvement, in S16, the specific method is:

[0023] The extracted semantic feature codes and global acoustic feature codes are used as input, and the audio is output through the deep learning network in the decoder. The real audio is compared and the error is fed back through the loss function to train the vocoder basic model; the decoder network structure includes but is not limited to RNN and Transformer network structures.

[0024] As a further improvement, in S16, the vocoder basic model includes but is not limited to a deep learning-based upsampling model and an adversarial neural network.

[0025] As a further improvement, in S17, the specific method is: before use, a small amount of audio is recorded through a given text to extract the semantic feature coding and global acoustic feature coding of the audio, and the recorded audio signal is combined as input. The adversarial training network is used to train the network by using the difference between the generated signal and the real signal as the loss function to fine-tune the basic model of the vocoder.

[0026] As a further improvement, the following steps are included in the inference phase:

[0027] S21, extracting target spectrum features through a spectrum feature model;

[0028] S22, extracting the semantic features of the target from the target spectrum features through a semantic feature extraction model and encoding the target to obtain a semantic feature code;

[0029] S23, extracting and encoding global acoustic features in the target audio through a global acoustic feature extraction model;

[0030] S24, reasoning the corresponding semantic feature coding through the semantic feature translation large model to generate translated semantic feature coding;

[0031] S25. Combining the translated semantic feature encoding with the global acoustic feature encoding, and generating new audio through a vocoder.

[0032] At the same time, the present invention solves the above problems through the following technical solutions:

[0033] A speech translation device based on a large language model to implement the above method includes:

[0034] An audio feature extraction unit, used to process the audio and extract spectrum features;

[0035] A semantic feature encoding and extraction unit, configured to extract and encode semantic features from spectral features;

[0036] A global acoustic feature extraction unit, used to extract and encode global acoustic features from spectral features;

[0037] A large model translation unit, used to convert the semantic feature encoding of the original speech into the semantic feature encoding of the target speech;

[0038] The vocoder unit converts the semantic feature codes generated by the conversion into audio signals in combination with the global acoustic feature codes.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] (1) The present invention solves the problems of high processing delay, weak context understanding ability, and complex system integration in traditional speech translation methods. Even if the target speaker has very little corpus in a certain language, it can synthesize fluent and accurate translation audio of the language and ensure the consistency of the timbre. It is also applicable to the field of cross-language speech translation, but is not limited to this field.

[0041] (2) The present invention adds audio spectrum features to the input of a large model. By extracting the spectrum features and inputting them into the large model, direct audio-to-audio translation is achieved. In addition, a large language model and an acoustic model are simultaneously synthesized to achieve faster translation through audio, realizing real-time translation from the speaker's language to the target language, thereby improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a speech translation method based on a large language model according to the present invention is shown;

[0043] Figure 2 This is a structural block diagram of a speech translation device based on a large language model of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] Example 1:

[0046] Combined with attachment Figure 1 As shown, a speech translation method based on a large language model includes: a training phase and an inference phase;

[0047] The training phase includes the following steps:

[0048] S11. Collect training corpus, including audios of different speakers and corresponding texts, as well as language tags and speaker tags, and extract spectral features of the audios using a spectral feature model;

[0049] Among them, the language label is used to train the semantic feature translation large model, and the speaker label is used to train the semantic feature extraction model.

[0050] Optionally, the training corpus includes but is not limited to publicly available speech synthesis training data sets or self-recorded speech synthesis training data sets; the text information of the speech synthesis training data set includes but is not limited to Chinese, English, etc., and the phonetic symbols used in the text include but are not limited to IPA (International Phonetic Alphabet). The extracted audio spectral features include but are not limited to linear spectral features and Mel spectral features.

[0051] For example, the obtained audio is used to extract 80-dimensional Mel spectral features through a window with a window length of 0.05s and a sliding distance of 0.015s, or 513-dimensional linear spectral features are extracted through Fourier transform.

[0052] S12. Use the text and audio spectral features obtained through S11 to train the semantic feature extraction model, and during the training process, remove the influence of the speaker feature through the gradient reversal of the speaker classification model;

[0053] Specifically, the training is carried out by using the spectral features and text information of the training audio combined with semantic features as the input of the semantic feature extraction model. The network structure of the semantic feature extraction model includes but is not limited to using an attention mechanism and a TDNN (Time Delay Neural Network) model;

[0054] During the process of training the semantic feature extraction model, the latent variable features of the encoded text information are subjected to gradient reversal through the speaker classification model, and the error is fed back through the classification loss function to adjust the network. That is, recognition is carried out through a gradient-reversed speaker classification model composed of a convolutional neural network and a fully connected network. Combining the speaker label, the error is fed back through the loss function to adjust the network, so that the speaker feature and the semantic feature are decoupled.

[0055] S13. Use the trained semantic feature extraction model to extract features from the spectral features and encode them to obtain semantic features and encode;

[0056] Specifically, the spectral features of the target audio are used to extract semantic features through the hidden layer of the semantic feature extraction model trained in S12, and the format required by the semantic feature translation large model is generated through encoding to obtain semantic feature encoding.

[0057] For example, the semantic feature text generated by the hidden layer of the semantic feature extraction model is 512-dimensional. According to the encoding input format of the selected translation large model such as qwen2.5, etc., the latent variable is quantized into <sematic_token_(digital number)> through the encoder.

[0058] S14. Use the global acoustic feature extraction model to extract the global acoustic features in the audio and encode them;

[0059] Specifically, the global acoustic features of the target audio are extracted by the global acoustic feature extraction model and encoded to obtain the global acoustic feature encoding. The global acoustic feature extraction model includes, but is not limited to, CNN, RNN, and Transformer. The encoding is performed according to the standard format of the large model input, such as <global_token_(numerical number)>.

[0060] It also includes: extracting the global feature variables from the audio of the translation target and the audio corresponding to the translated target, and encoding the feature variables.

[0061] S15. Train the semantic feature translation large model through the semantic feature encoding of a large number of translation-matched corresponding texts, so as to generate the translated semantic feature encoding through the semantic feature translation large model.

[0062] The semantic features extracted from a large number of content-matched audio include the semantic feature encoding extracted from the target speech combined with the language label and the semantic feature encoding corresponding to the translated audio to train the semantic feature translation large model.

[0063] Optionally, the network structure of the semantic feature translation large model includes, but is not limited to, large model network structures such as qwen and llama.

[0064] S16. Train the vocoder basic model using the global acoustic feature encoding and semantic feature encoding of a large number of audio corresponding to texts.

[0065] Specifically, the semantic feature encoding and global acoustic feature encoding extracted through S15 are used as inputs and combined with the original audio signal to train the vocoder basic model.

[0066] Optionally, the vocoder basic model includes, but is not limited to, deep learning upsampling models (such as Wavernn and WaveNet) based on deep learning, as well as adversarial neural networks such as hifigan.

[0067] The specific method is: using the semantic feature encoding and global acoustic feature encoding extracted through S15 as inputs, outputting audio through the deep learning network in the decoder, and comparing the real audio to feedback errors through the loss function to train the vocoder basic model. The network structure of the decoder includes, but is not limited to, RNN and Transformer network structures.

[0068] S17. Fine-tune the vocoder basic model with the global acoustic feature information and semantic features of a small amount of audio of the target speaker to obtain the vocoder.

[0069] Specifically, before use, a small amount of audio is recorded with given text to extract the semantic features and global acoustic feature encoding of the audio. The recorded audio signal is combined as input, and an adversarial training network is used to train the network by using the difference between the generated signal and the real signal as the loss function to fine-tune the basic model of the vocoder.

[0070] The inference phase includes the following steps:

[0071] S21. Extract target spectrum features through spectrum feature model

[0072] S22, extracting the semantic features of the target from the target spectrum features through a semantic feature extraction model and encoding the target to obtain a semantic feature code;

[0073] S23, extracting and encoding global acoustic features in the target audio through a global acoustic feature extraction model;

[0074] S24, reasoning the corresponding semantic feature coding through the semantic feature translation large model to generate translated semantic feature coding;

[0075] S25, combining the translated semantic feature code with the global acoustic feature code to generate new audio through a vocoder;

[0076] It is understandable that the spectral feature model parameters of the inference stage are obtained in the training stage, and the network structure is consistent; the semantic feature processing method of the inference stage is consistent with that of the training stage, and the global acoustic feature processing method of the inference stage is consistent with that of the training stage; the semantic features generated in the inference stage are generated by the large model according to the semantic features of the audio to generate the semantic features of the corresponding translation results and encode them, the global acoustic features of the speaker in the inference stage are expanded according to the length of the semantic feature encoding input by the vocoder and combined with the semantic feature encoding, the basic model parameters of the vocoder are obtained in the training stage, and the corresponding audio is synthesized by the vocoder.

[0077] Example 2:

[0078] Refer to the attached Figure 2 , a speech translation device based on a large language model, comprising:

[0079] The audio feature extraction unit 100 is used to process the audio and extract spectral features;

[0080] Semantic feature encoding and extraction unit 200, used to extract semantic features from spectral features and encode them;

[0081] A global acoustic feature extraction unit 300 is used to extract and encode global acoustic features from spectral features;

[0082] The large model translation unit 400 is used to convert the semantic feature coding of the source speech into the semantic feature coding of the target speech;

[0083] The vocoder unit 500 converts the semantic feature codes generated by the conversion into audio signals in combination with the global acoustic feature codes.

[0084] Embodiment 2 of the present invention provides a speech translation device based on a large language model. By extracting and encoding the semantic features of the target audio, and adding gradient inversion to remove speaker features, the speech translation system can more accurately translate the target audio using the semantic features in the large model.

[0085] It should be noted that the modules (or units) in this embodiment are logical in nature. In specific implementation, multiple modules (or units) can be combined into one module (or unit), and one module (or unit) can also be split into multiple modules (or units).

[0086] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0087] Although the present invention is described herein with reference to illustrative embodiments of the present invention, the above embodiments are merely preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A speech translation method based on a large language model, characterized in that: It has a training phase and an inference phase; The training phase includes the following steps: S11. Collect training corpus, including audios of different speakers and corresponding texts, and extract spectral features of the audios using a spectral feature model; S12. Train a semantic feature extraction model using the spectral features of text and audio, and remove the influence of speaker characteristics by gradient inversion of the speaker classification model during training. S13, extracting and encoding features from the spectral features using the trained semantic feature extraction model to obtain semantic feature codes; S14. extracting and encoding global acoustic features from the spectral features using a global acoustic feature extraction model to obtain a global acoustic feature code; S15. Training a semantic feature translation model by translating a large number of semantic feature codes of the corresponding texts, so as to generate translated semantic feature codes from the semantic feature codes by the semantic feature translation model; S16. Using global acoustic feature encoding and semantic feature encoding of a large amount of text corresponding to audio to train a vocoder basic model; S17. Fine-tune the vocoder base model using global acoustic feature information and semantic features of a small amount of audio of the target speaker to obtain a vocoder.

2. The speech translation method based on a large language model according to claim 1, characterized in that: In the above S11, language tags and speaker tags of different speakers are also obtained, wherein the language tags are used to train the semantic feature translation model, and the speaker tags are used to train the semantic feature extraction model.

3. The speech translation method based on a large language model according to claim 1, characterized in that: In S11, the training corpus includes but is not limited to a public speech synthesis training dataset or a self-recorded speech synthesis training dataset; the extracted audio spectrum features include at least linear spectrum features and Mel spectrum features.

4. The speech translation method based on a large language model according to claim 1, characterized in that: In said S12, the specific method is: The semantic feature extraction model is trained based on the spectral features and text information of the training audio combined with semantic features. The network structure of the semantic feature extraction model includes an attention mechanism and a TDNN model. During the training of the semantic feature extraction model, the latent variable features of the encoded text information are subjected to gradient reversal recognition by the speaker classification model, and combined with the speaker label, the error adjustment network is fed back through the classification loss function to decouple the speaker features from the semantic features.

5. The speech translation method based on a large language model according to claim 1, characterized in that: In the above S13, the semantic features extracted from the hidden layer of the semantic feature extraction model are encoded. Encoding refers to generating a format required for the large translation model.

6. The speech translation method based on a large language model according to claim 1, characterized in that: In said S16, the specific method is: The extracted semantic feature codes and global acoustic feature codes are used as input, and the audio is output through the deep learning network in the decoder. The real audio is compared and the error is fed back through the loss function to train the vocoder basic model; the decoder network structure includes but is not limited to RNN and Transformer network structures.

7. The speech translation method based on a large language model according to claim 6, characterized in that: In S16, the vocoder basic model includes but is not limited to a deep learning-based upsampling model and an adversarial neural network.

8. The speech translation method based on a large language model according to claim 1, characterized in that: In said S17, the specific method is: before use, a small amount of audio is recorded through a given text to extract the semantic feature coding and global acoustic feature coding of the audio, the recorded audio signal is combined as input, and an adversarial training network is used to train the network by using the difference between the generated signal and the real signal as the loss function to fine-tune the basic model of the vocoder.

9. A speech translation method based on a large language model according to any one of claims 1 to 8, characterized in that: The inference phase includes the following steps: S21, extracting target spectrum features through a spectrum feature model; S22, extracting the semantic features of the target from the target spectrum features through a semantic feature extraction model and encoding the target to obtain a semantic feature code; S23, extracting and encoding global acoustic features in the target audio through a global acoustic feature extraction model; S24, reasoning the corresponding semantic feature coding through the semantic feature translation large model to generate translated semantic feature coding; S25. Combining the translated semantic feature encoding with the global acoustic feature encoding, and generating new audio through a vocoder.

10. A speech translation device based on a large language model, characterized in that: include: An audio feature extraction unit, used to process the audio and extract spectrum features; A semantic feature encoding and extraction unit, configured to extract and encode semantic features from spectral features; A global acoustic feature extraction unit, used to extract and encode global acoustic features from spectral features; A large model translation unit, used to convert the semantic feature encoding of the original speech into the semantic feature encoding of the target speech; The vocoder unit converts the semantic feature codes generated by the conversion into audio signals in combination with the global acoustic feature codes.

Images