Audio translation methods, devices, electronic equipment, and storage media
By standardizing the voiceprint of audio in speech translation scenarios and using a trained speech translation model for audio synthesis and translation, the problem of inconsistent speech translation results from different speakers is solved, thus improving the accuracy of speech translation.
Patent Information
- Application Number
- CN202311273188.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-09-27
AI Technical Summary
In existing speech translation tasks, the translation results of the same speech vary depending on the speaker, resulting in poor speech translation accuracy.
By performing voiceprint standardization processing on speech translation scenarios, the original language audio of the initial voiceprint is obtained, the original language text is extracted, and audio synthesis and speech-to-text translation are performed based on the target voiceprint. The voiceprint standardization of the audio is achieved by using a trained speech translation model.
It improves the accuracy of speech translation and avoids differences in translation results when different people speak the same words.
Smart Images

Figure CN117334199B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech translation technology, and more specifically, to an audio translation method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the rapid development of artificial intelligence technology, natural language processing technology is being applied more and more widely. Speech translation is an important task in natural language processing, which is to translate the original language audio into the target language text. However, current speech translation tasks suffer from poor accuracy. Summary of the Invention
[0003] In view of the above problems, this application proposes an audio translation method, apparatus, electronic device, and storage medium to solve the above problems.
[0004] In a first aspect, embodiments of this application provide an audio translation method, the method comprising: acquiring a first original language audio corresponding to an initial voiceprint; extracting original language text corresponding to the first original language audio, and performing audio synthesis on the original language text based on a target voiceprint to obtain a second original language audio corresponding to the target voiceprint; and performing speech-to-text translation on the second original language audio to obtain the target language text.
[0005] Secondly, embodiments of this application provide an audio translation device, the device comprising: an original language audio acquisition module, configured to acquire a first original language audio corresponding to an initial voiceprint; a target language text acquisition module, configured to extract the original language text corresponding to the first original language audio, and perform audio synthesis on the original language text based on the target voiceprint to obtain a second original language audio corresponding to the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text.
[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs the above-described method.
[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the above-described method.
[0008] The audio translation method, apparatus, electronic device, and storage medium provided in this application embodiment acquire a first original language audio corresponding to an initial voiceprint, extract the original language text corresponding to the first original language audio, and synthesize the original language text based on the target voiceprint to obtain a second original language audio corresponding to the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text. By standardizing the voiceprint of the audio for the speech translation scenario, the problem of different translation results corresponding to different people speaking the same speech can be avoided, thereby improving the accuracy of speech translation. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 A flowchart illustrating an embodiment of the audio translation method provided in this application is shown;
[0011] Figure 2 A flowchart illustrating an embodiment of the audio translation method provided in this application is shown;
[0012] Figure 3 This application shows Figure 2 The flowchart of step S220 of the audio translation method shown is illustrated.
[0013] Figure 4 This illustration shows a schematic diagram of audio translation using a speech translation model provided in an embodiment of this application;
[0014] Figure 5 This application shows Figure 3 The flowchart of step S221 of the audio translation method shown is illustrated.
[0015] Figure 6 A flowchart illustrating an embodiment of the audio translation method provided in this application is shown;
[0016] Figure 7 A schematic diagram of the structure of the speech translation model provided in an embodiment of this application is shown;
[0017] Figure 8 A schematic diagram of the voiceprint conversion model provided in an embodiment of this application is shown;
[0018] Figure 9 A schematic diagram of the structure of the speech recognition model provided in an embodiment of this application is shown;
[0019] Figure 10 This paper shows a schematic diagram of the speaker recognition model provided in an embodiment of the present application;
[0020] Figure 11 A schematic diagram of the decoder provided in an embodiment of this application is shown.
[0021] Figure 12 A block diagram of an audio translation device according to an embodiment of this application is shown.
[0022] Figure 13 A block diagram of an electronic device for performing an audio translation method according to an embodiment of this application is shown;
[0023] Figure 14 An embodiment of this application shows a storage unit for storing or carrying program code that implements the audio translation method according to an embodiment of this application. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0025] With the rapid development of artificial intelligence technology, the application of natural language processing (NLP) technology is becoming increasingly widespread. Speech translation is a crucial task within NLP, involving translating audio from a source language into text in a target language. Current solutions typically employ cascaded systems. The audio file of the source language first passes through an automatic speech recognition (ASR) system to convert the audio into text, which is then input into a machine translation (MT) system to translate it into text in the target language. End-to-end speech translation systems are gaining increasing attention and becoming a technological hotspot. Compared to cascaded systems, end-to-end systems offer numerous advantages: relatively simpler models, easier maintenance, lower latency, and reduced error propagation, gradually becoming a replacement for cascaded systems.
[0026] As introduced above, cascaded systems are a cascade of ASR and MT systems. They suffer from drawbacks such as error propagation, high computational cost, high latency, and relatively complex maintenance and deployment. This has led to the growing attention and promising alternative of end-to-end systems. Currently, end-to-end systems mainly employ the following technical solutions: pre-training, multi-task joint training, modality mitigation, and data augmentation. Pre-training and multi-task joint training primarily address the lack of training data for speech translation by using additional data to achieve knowledge transfer. Modality mitigation aims to bring the speech representation of the original language closer to the corresponding text representation, enabling the speech-to-text (ST) translation system to approach the translation capabilities of the MT system. However, none of these solutions have solved the speaker voiceprint robustness problem, where different people uttering the same speech often produce different translation results, leading to poor accuracy in speech translation.
[0027] To address the aforementioned problems, the inventors, through long-term research, discovered and proposed the audio translation method, apparatus, electronic device, and storage medium provided in the embodiments of this application. By standardizing the voiceprint of the audio for speech translation scenarios, the problem of different people producing different translations of the same speech can be avoided, thereby improving the accuracy of speech translation. The specific audio translation method will be described in detail in the subsequent embodiments.
[0028] Please see Figure 1 , Figure 1 A flowchart illustrating an embodiment of the audio translation method provided in this application is shown. This method standardizes the voiceprint of audio for speech translation scenarios, thereby avoiding different translation results for the same speech spoken by different people, and thus improving the accuracy of speech translation. In a specific embodiment, this audio translation method is applied to, for example... Figure 12 The audio translation device 200 and the electronic device 100 equipped with the audio translation device 200 are shown. Figure 13 The following will use an electronic device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the electronic device used in this embodiment may include smartphones, tablets, wearable electronic devices, servers, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown will be explained in detail. The audio translation method may specifically include the following steps:
[0029] Step S110: Obtain the first original language audio corresponding to the initial voiceprint.
[0030] In this embodiment, a first original language audio can be obtained. This first original language audio may correspond to an initial voiceprint.
[0031] In some implementations, the first original language audio corresponding to the initial voiceprint can be picked up by a microphone, the first original language audio corresponding to the initial voiceprint can be obtained from a server via a network, the first original language audio corresponding to the initial voiceprint can be obtained from other devices via a network, or the first original language audio corresponding to the initial voiceprint can be obtained locally, etc., without limitation.
[0032] As an implementable approach, the first original language audio can correspond to a first speaker; that is, the first original language audio can be obtained by capturing the first speaker during their speech. It is understood that the voiceprint corresponding to the first speaker can be identified as the initial voiceprint, and the language corresponding to the first speaker can be identified as the original language. For example, during the first speaker's speech, a microphone can be used to capture the first speaker's voice to obtain the first original language audio corresponding to the initial voiceprint.
[0033] In some implementations, the original language may include Chinese, English, Japanese, Korean, etc., and is not limited thereto.
[0034] Step S120: Extract the original language text corresponding to the first original language audio, and synthesize the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text.
[0035] In this embodiment, if a first original language audio is obtained, the original language text corresponding to the first original language audio can be extracted. If the original language text is obtained, audio synthesis can be performed on the original language text based on the target voiceprint to obtain a second original language audio corresponding to the target voiceprint. If the second original language audio is obtained, speech-to-text translation can be performed on the second original language audio to obtain the target language text.
[0036] In some implementations, extracting the original language text corresponding to the first original language audio may include: inputting the first original language audio into a speech recognition model to obtain the original language text corresponding to the first original language audio output by the speech recognition model. Optionally, the speech recognition model may include a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, or a Transformer model. Optionally, when obtaining the original language text corresponding to the first original language audio output by the speech recognition model, the original language text may be corrected or ambiguous according to the context, etc., which is not limited here. Optionally, before inputting the first original language audio into the speech recognition model, the first original language audio may also be preprocessed, for example, the first original language audio may be digitized, converting the first original language audio into a form that can be accepted by the model (e.g., converting it into a spectrogram using Fourier transform (FFT), dividing the first original language audio into some short segments, etc., which is not limited here. Optionally, before inputting the first original language audio into the speech recognition model, feature extraction can be performed on the first original language audio, and the extracted features can be input into the speech recognition model. For example, Mel-frequency cepstral coefficients (MFCC), linear predictive coding (LPC), cepstral distance (I-vectors), or more advanced features such as spectrograms can be extracted from the first original language audio.
[0037] In one approach, the electronic device can pre-acquire the target voiceprint and store it locally. Based on this, when the original language text corresponding to the first original language audio is extracted, the target voiceprint can be obtained locally, and the original language text can be synthesized based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0038] As another method, the electronic device can obtain the target voiceprint from a server connected to the communication link. Based on this, if the original language text corresponding to the first original language audio is extracted, the target voiceprint can be obtained from the server, and the original language text can be synthesized based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0039] As another approach, after extracting the original language text corresponding to the first original language audio, a standard audio can be obtained, and voiceprint recognition can be performed on the standard audio to obtain a target voiceprint. Then, based on the target voiceprint, audio synthesis is performed on the original language text to obtain a second original language audio corresponding to the target voiceprint. This standard audio can correspond to a second speaker; that is, the standard audio can be obtained by capturing data during the second speaker's speech.
[0040] Optionally, the target voiceprint can be referenced from the user's corresponding voiceprint. This reference user can be a robot or a real person; there are no restrictions. Furthermore, the reference user's occupation, gender, age, and location are also not limited.
[0041] It is understandable that the voiceprints corresponding to the first and second original language audios are different, but the corresponding audio content is the same.
[0042] In some implementations, after extracting the original language text corresponding to the first original language audio, an initial original language audio can be synthesized from the original language text using a speech synthesis engine. Finally, the target voiceprint and the initial original language audio are combined to obtain the second original language audio. Optionally, before processing the original language text using a speech synthesis engine (such as a TTS engine), the original language text can be preprocessed, for example, by performing word segmentation, part-of-speech tagging, and grammar detection, to ensure the correctness and readability of the original language text.
[0043] In some implementations, speech-to-text translation of second source language audio to obtain target language text may include: If the second source language audio is obtained, speech can be converted into text using speech recognition technology. Speech recognition technology analyzes the speech signal, identifies the sounds and intonations, and converts it into text. If the converted text is obtained, machine translation technology can be used to translate the text into the target language. Machine translation technology utilizes large corpora and natural language processing algorithms, employing statistical models or neural network models to achieve automatic translation. This technology can quickly translate text from one language to another.
[0044] Optionally, the target language may include Chinese, English, Japanese, Korean, etc., and the source language and the target language may be different. For example, when the source language is Chinese, the target language may be English; when the source language is Japanese, the target language may be Chinese, etc. There are no restrictions here.
[0045] An embodiment of this application provides an audio translation method that obtains a first original language audio corresponding to an initial voiceprint, extracts the original language text corresponding to the first original language audio, synthesizes the original language text based on the target voiceprint to obtain a second original language audio corresponding to the target voiceprint, and performs speech-to-text translation on the second original language audio to obtain the target language text. By standardizing the voiceprint of the audio for speech translation scenarios, the method can avoid the problem that different people speaking the same speech will result in different translations, thereby improving the accuracy of speech translation.
[0046] Please see Figure 2, Figure 2 A flowchart illustrating an embodiment of the audio translation method provided in this application is shown. The following will focus on... Figure 2 The process shown will be explained in detail. The audio translation method may specifically include the following steps:
[0047] Step S210: Obtain the first original language audio corresponding to the initial voiceprint.
[0048] For a detailed description of step S210, please refer to step S110, which will not be repeated here.
[0049] Step S220: Input the first original language audio into the trained speech translation model, wherein the trained speech translation model is used to extract the original language text, and synthesize the original language text into the second original language audio based on the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text.
[0050] In this embodiment, upon obtaining the first original language audio, the first original language audio can be input into a trained audio translation model. This trained audio translation model is obtained through machine learning. Specifically, a training dataset is first collected, where one type of data in the training dataset has attributes or features that distinguish it from another type of data. Then, the collected training dataset is used to train and model a neural network according to a preset algorithm, thereby summarizing patterns based on the training dataset to obtain the trained audio translation model. In this embodiment, the training dataset may include, for example, the original language audio, the corresponding original language text for each original language audio, and the corresponding target speech text for each original language audio.
[0051] Understandably, the pre-trained audio translation model can be stored locally on the electronic device after training. Based on this, after acquiring the first original language audio, the electronic device can directly call the pre-trained audio translation model locally. For example, it can directly send instructions to the pre-trained audio translation model to instruct it to read the first original language audio from the target storage area. Alternatively, the electronic device can directly input the first original language audio into the locally stored pre-trained speech translation model. This effectively avoids the slowdown caused by network factors in inputting the first original language audio into the pre-trained speech translation model, thereby improving the speed at which the pre-trained speech translation model acquires the first original language audio and enhancing the user experience.
[0052] Furthermore, the trained speech translation model can be pre-trained and stored on a server connected to the electronic device. Based on this, after the electronic device acquires the first original language audio, it can send instructions via the network to the trained speech translation model stored on the server, instructing the model to read the first original language audio acquired by the electronic device through the network. Alternatively, the electronic device can send the first original language audio to the trained speech translation model stored on the server via the network. By storing the trained speech translation model on the server, the storage space occupied by the electronic device is reduced, minimizing the impact on its normal operation.
[0053] The trained speech translation model can be used to extract the original language text corresponding to the first original language audio, synthesize the original language text based on the target voiceprint to obtain the second original language audio, and perform speech-to-text translation on the second original language audio to finally obtain the target language text.
[0054] Please see Figure 3 , Figure 3 This application shows Figure 2 The flowchart shown illustrates step S220 of the audio translation method. In this embodiment, the speech translation model includes a voiceprint conversion module and a text translation module. The following will focus on... Figure 3 The process shown will be described in detail, and the method may specifically include the following steps:
[0055] Step S221: Input the first original language audio into the voiceprint conversion module, extract the content features of the first original language audio through the voiceprint conversion module to obtain the original language text, and synthesize the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0056] Please see Figure 4 , Figure 4 This illustration shows a schematic diagram of audio translation using a speech translation model provided in an embodiment of this application. Figure 4 As shown, the speech translation model may include a voiceprint conversion module and a text translation module (Transformer). The voiceprint conversion module may include a content encoder and a decoder.
[0057] In this embodiment, the speech translation model may include a voiceprint conversion module. This voiceprint conversion module can be used to change the timbre of the speech, making it sound as if another person is speaking the same content. The voiceprint conversion process may include: perceptual separation of speech, for example, the speech recognition module in the voiceprint voice changing system separates the linguistic information and the speaker's timbre information in the received speech; and synthesis of new speech, for example, the speech synthesis module resynthesizes the target speaker's voiceprint with the previously extracted linguistic content to create new speech, thereby achieving timbre transformation.
[0058] In some implementations, when a first original language audio is obtained, the first original language audio can be input into a voiceprint conversion model. Accordingly, when the voiceprint conversion model receives the first original language audio, it can extract content features from the first original language audio to obtain the original language text corresponding to the first original language audio, and synthesize audio from the original language text based on the target voiceprint to obtain a second original language audio corresponding to the target voiceprint.
[0059] Please see Figure 5 , Figure 5 This application shows Figure 3 The diagram illustrates step S221 of the audio translation method. In this embodiment, the voiceprint conversion module includes a content encoder and a decoder, which will be discussed below. Figure 5 The process shown will be described in detail, and the method may specifically include the following steps:
[0060] Step S2211: Input the first original language audio into the content encoder, and extract content features from the first original language audio through the content encoder to obtain the original language text.
[0061] In this embodiment, the voiceprint conversion module may include a content encoder. This content encoder is used to extract content information from the original input information. For example, the content encoder is a module used to convert the input sequence into a hidden representation. It takes the word embedding vector at each position in the input sequence as input and output, and then encodes the information at each position into a fixed-length hidden vector identifier through multiple layers of self-attention mechanisms and fully connected layers. This transforms the input sequence (such as the first original language audio) into a more compact and easier-to-process representation (i.e., a hidden vector) for subsequent processing.
[0062] In some implementations, upon obtaining a first original language audio, the first original language audio can be input into a content encoder. Accordingly, upon receiving the first language audio, the content encoder can extract content features from the first original language audio to obtain the original language text corresponding to the first original language audio.
[0063] Step S2212: Input the original language text into the decoder, and use the decoder to synthesize audio based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0064] In this embodiment, the voiceprint conversion module may further include a decoder, wherein the decoder can be connected to the content encoder, that is, the output of the content encoder is connected to the input of the decoder. The decoder can be used to decode the received encoded information or data, restoring it to the original signal or data.
[0065] In some implementations, when the original language text is obtained by processing the first original language audio through a content encoder, the original language text can be input into a decoder. Accordingly, when the decoder receives the original language text, it can perform audio synthesis on the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0066] Step S222: Input the second original language audio into the text translation module, and use the text translation module to perform speech-to-text translation on the second original language audio to obtain the target language text.
[0067] In this embodiment, the speech translation model may further include a text translation module, which can be connected to the voiceprint conversion module, i.e., the output of the voiceprint conversion module is connected to the input of the text translation module. The text translation module is used to translate audio or text in one language into audio or text in another language.
[0068] In some implementations, when the second original language audio is obtained by processing the first original language audio through the voiceprint conversion module, the second original language audio can be input into the text translation module. Accordingly, upon receiving the second original language audio, the text translation module can perform speech-to-text translation on the second original language audio to obtain the target language text corresponding to the second original language audio.
[0069] Step S230: Obtain the target language text output by the trained speech translation model.
[0070] In this embodiment, the trained speech translation model can output the target language text when it obtains the target language text, and the electronic device can obtain the target language text output by the trained speech translation model.
[0071] In some implementations, if the trained audio translation model is stored locally on the electronic device, the electronic device can directly obtain the target language text output by the trained audio translation model; if the trained audio translation model is stored on a server, the electronic device can obtain the target language text output by the trained audio translation model from the server via a network.
[0072] An embodiment of this application provides an audio translation method that obtains a first original language audio corresponding to an initial voiceprint, inputs the first original language audio into a trained speech translation model, wherein the trained speech translation model is used to extract the original language text, synthesize the original language text into a second original language audio based on the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text, thereby obtaining the target language text output by the trained speech translation model. Compared to Figure 1 The audio translation method shown in this embodiment also uses a trained speech translation model to translate the original language audio into text to obtain the target language text, which can improve the accuracy of speech translation.
[0073] Please see Figure 6 , Figure 6 A flowchart illustrating an embodiment of the audio translation method provided in this application is shown. The following will focus on... Figure 6 The process shown will be explained in detail. The audio translation method may specifically include the following steps:
[0074] Step S310: Obtain a first training dataset, wherein the first training dataset includes multiple third original language audios, original language texts corresponding to each of the multiple third original language audios, and target language texts corresponding to each of the multiple third original language audios.
[0075] In this embodiment, a first training dataset can be obtained, which may include multiple third original language audios, original language texts corresponding to each of the multiple original language audios, and target language texts corresponding to each of the multiple third original language audios. For example, the first training dataset D = {speech, x, y} can be defined, where speech represents the third original language audio, x represents the original language text corresponding to the third original language audio, and y represents the target language text corresponding to the third original language audio.
[0076] Step S320: Train the speech translation model to be trained based on the target voiceprint and the first training dataset to obtain a trained speech translation model, which includes a voiceprint conversion module and a text translation module.
[0077] Please see Figure 7 , Figure 7 A schematic diagram of the structure of the speech translation model provided in an embodiment of this application is shown. Figure 7 As shown, the speech translation model may include a voiceprint conversion module and a text translation module (Transformer). The voiceprint conversion module may include a content encoder and a decoder.
[0078] In this embodiment, given a first training dataset, the speech translation model to be trained can be trained based on the target voiceprint and the first training dataset to obtain a trained speech translation model. Alternatively, given a first training dataset, the voiceprint conversion module (content encoder and decoder) and text translation module in the speech training model can be trained using multiple third-language audio files, their corresponding original language texts, their corresponding target language texts, and the target voiceprint to obtain a trained speech translation model.
[0079] As an feasible approach, the loss function of a speech translation model can be adopted. in: Where s indicates speech features, y indicates the target text to be translated, N indicates the length of the output sentence, |V| indicates the size of the target speech vocabulary, and I(y t =k) The indicator function indicates whether the output value is equal to the actual target value.
[0080] In some implementations, such as Figure 7 As shown, the voiceprint conversion module may also include a speaker encoder, which is used to extract speaker feature information from the original input information. For example, the speaker encoder is a module used to convert the input sequence into a hidden representation. It takes the word embedding vector at each position in the input sequence as input and output, and then encodes the information at each position into a fixed-length hidden vector identifier through multiple layers of self-attention mechanism and fully connected layers. This transforms the input sequence (such as the first original language audio) into a more compact and easier-to-process representation (i.e., a hidden vector) for subsequent processing.
[0081] In some implementations, when training the language translation model (the speaker encoder in the speech translation model), a standard audio can be selected and used in conjunction with the aforementioned first training dataset to train the speech translation model. During training, the standard audio can be input into the speaker encoder, which extracts speaker features to obtain the corresponding voiceprint. Accordingly, after the speech translation model is trained, the voiceprint output by the speaker encoder can be used as the target voiceprint. Consequently, in subsequent speech translation processes using the speech translation model, it is no longer necessary for the speaker encoder to identify the target voiceprint, which can save computation time for the speech translation model and improve its processing speed.
[0082] In some implementations, the voiceprint conversion module can be obtained through pre-training. Based on this, the training efficiency of the speech translation model, which includes the voiceprint conversion module, can be improved. One approach is to obtain a second training dataset, which includes multiple fourth original language audio recordings and corresponding fifth original language audio recordings for each of the fourth original language audio recordings. The voiceprints corresponding to the fourth original language audio recordings and the corresponding fifth original language audio recordings are different, but their corresponding audio content is the same. With the second training dataset, the voiceprint conversion model can be trained based on it to obtain a pre-trained voiceprint conversion module for use in the subsequent speech translation model. It is understood that the pre-trained voiceprint conversion module can reconstruct the Mel spectrum features, regenerating the speaker's Mel spectrum features based on content features and speaker voiceprint features.
[0083] As an feasible approach, the loss function of the voiceprint conversion model can be adopted.
[0084]
[0085] Please see Figure 8 , Figure 8 A schematic diagram of the voiceprint conversion model provided in an embodiment of this application is shown. Figure 8 As shown, the voiceprint conversion model can include a content encoder, a speaker encoder, and a decoder. As one feasible approach, given a second training dataset, the content encoder, speaker encoder, and decoder in the voiceprint conversion model can be trained using multiple fourth original language audio recordings and their corresponding fifth original language audio recordings to obtain a pre-trained voiceprint conversion module. Optionally, during the deactivation of the pre-trained voiceprint conversion model, the inputs to the speaker encoder and content encoder can be different speech sounds, and the decoder outputs the Mel-spectral features of the audio containing the target voiceprint and target content.
[0086] In some implementations, the content encoder can be obtained through pre-training. Based on this, the training efficiency of the speaker conversion model and speech translation model, which include the content encoder, can be improved. One approach is to obtain a third training dataset, which may include multiple sixth-language audio recordings and their corresponding original language texts. With this third training dataset, the speech recognition model including the content encoder can be trained to obtain a pre-trained content encoder. It is understood that after the speech recognition model is trained, a speech recognition model can be obtained, where the pre-trained content encoder can serve as a speech content extractor.
[0087] Please see Figure 9 , Figure 9 A schematic diagram of the structure of the speech recognition model provided in an embodiment of this application is shown. Figure 9 As shown, the speech recognition model can employ the Transformer encoder and CTC. The input to the speech recognition model can be the sixth original speech audio, and the output can be the original language text corresponding to the sixth original speech audio. The third training dataset can be represented as D = {speech, text}, where speech represents the sixth original speech audio, and text represents the original language text corresponding to the sixth original speech audio. The input x = [x1, x2, x3...xn] of the content encoder of the speech recognition model is the fbank feature of the sixth original speech audio, and the output h = [h1, h2, h3...hn] of the content encoder of the speech recognition model is the feature representation of the sixth original speech audio.
[0088] As an feasible approach, the loss function of the speech recognition model can be L... CTC =-logP ctc (y s |h s ;θ ctc ).
[0089] In some implementations, the speaker encoder can be obtained through pre-training. Based on this, the training efficiency of the speaker recognition model and speech translation model, which include the speaker encoder, can be improved. As one approach, a fourth training dataset can be obtained, which may include multiple seventh-language audio recordings and their corresponding speaker identifiers. With the fourth training dataset, the speaker recognition model, including the speaker encoder, can be trained based on this dataset to obtain a pre-trained speaker encoder.
[0090] Please see Figure 10 , Figure 10 A schematic diagram of the speaker recognition model provided in an embodiment of this application is shown. Figure 10 As shown, the speaker recognition model can be a multi-classification model. It can include a speaker encoder followed by softmax and linear network layers. The fourth training dataset can be defined as D = {speech, spkid}, where speech represents the original seventh language audio, and spkid represents the speaker identifier (e.g., speaker ID) corresponding to the original seventh language audio. The feature extractor uses multiple convolutional neural networks and stacked pooling layers. The input to the speaker recognition model can be the fbank features x = [x1, x2, x3…xn] of the original seventh language audio, and the output of the speaker recognition model can be the speaker identifier corresponding to the original seventh language audio, which is a one-dimensional vector R.
[0091] Among them, Figure 10 In this context, ReLU stands for Rectified Linear Function; Convbank128-3-1 is a configuration in a convolutional neural network, where "Conv" represents a convolutional layer, "128" represents the number of convolutional kernels as 128, "3" represents the size of the convolutional kernel as 3×3, and "1" represents a fully connected layer after the convolution; Avgpool represents an average pooling layer.
[0092] As an feasible approach, the loss function of the speaker identification model can be adopted.
[0093]
[0094] Please see Figure 11 , Figure 11 A schematic diagram of the decoder provided in an embodiment of this application is shown. Figure 11 As shown, the voiceprint conversion model can include a decoder, and the decoder can use, for example... Figure 11 The convolutional neural network structure shown is shown.
[0095] Among them, Figure 11 In this context, ReLU stands for Rectified Linear Function; Convbank128-3-1 is a configuration in a convolutional neural network, where "Conv" represents a convolutional layer, "128" represents the number of convolutional kernels as 128, "3" represents the kernel size as 3×3, and "1" indicates a fully connected layer following the convolution; AdaIN represents an image transformation algorithm; Pixshuffed1D represents one-dimensional pixel shuffling or one-dimensional pixel scrambling; and Affine represents affine transformation.
[0096] Step S330: Obtain the first original language audio corresponding to the initial voiceprint.
[0097] Step S340: Input the first original language audio into the trained speech translation model, wherein the trained speech translation model is used to extract the original language text, and synthesize the original language text into the second original language audio based on the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text.
[0098] Step S350: Obtain the target language text output by the speech translation model.
[0099] For a detailed description of steps S330-S350, please refer to steps S110-S130, which will not be repeated here.
[0100] An embodiment of this application provides an audio translation method that obtains a first training dataset, wherein the first training dataset includes multiple third original language audios, original language texts corresponding to each of the multiple third original language audios, and target language texts corresponding to each of the multiple third original language audios. A speech translation model to be trained is trained based on the target voiceprint and the first training dataset to obtain a trained speech translation model. The speech translation model includes a voiceprint conversion module and a text translation module. The method obtains the first original language audio corresponding to the initial voiceprint, inputs the first original language audio into the trained speech translation model, and uses the trained speech translation model to extract the original language text. Based on the target voiceprint, the method performs audio synthesis on the original language text to obtain a second original language audio, and performs speech-to-text translation on the second original language audio to obtain the target language text, thus obtaining the target language text output by the trained speech translation model. Compared to... Figure 1 The audio translation method shown in this embodiment also trains the audio translation model to be trained to obtain a trained audio translation model, and performs speech-to-text translation of the original language audio based on the trained speech translation model to obtain the target language text, which can improve the accuracy of speech translation.
[0101] Please see Figure 12 , Figure 12 A block diagram of an audio translation device according to an embodiment of this application is shown. The following will focus on... Figure 12 The audio translation device 200, illustrated in the block diagram, includes: a source language audio acquisition module 210 and a target language text acquisition module 220, wherein:
[0102] The original language audio acquisition module 210 is used to acquire the first original language audio corresponding to the initial voiceprint.
[0103] The target language text acquisition module 220 is used to extract the original language text corresponding to the first original language audio, and to perform audio synthesis on the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint, and to perform speech-to-text translation on the second original language audio to obtain the target language text.
[0104] Furthermore, the target language text acquisition module 220 includes: a first original language audio input submodule and a target language text output submodule, wherein:
[0105] A first original language audio input submodule is used to input the first original language audio into a trained speech translation model, wherein the trained speech translation model is used to extract the original language text, synthesize the original language text into the second original language audio based on the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text.
[0106] Furthermore, the speech translation model includes a voiceprint conversion module and a text translation module. The first original language audio input submodule includes: a first original language audio input unit and a second original language audio input unit, wherein:
[0107] The first original language audio input unit is used to input the first original language audio into the voiceprint conversion module, extract the content features of the first original language audio through the voiceprint conversion module to obtain the original language text, and synthesize the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0108] Furthermore, the voiceprint conversion module includes a content encoder and a decoder, and the first original language audio input unit includes: a first original language audio input subunit and an original language text input subunit, wherein:
[0109] The first original language audio input subunit is used to input the first original language audio into the content encoder, and to extract the content features of the first original language audio through the content encoder to obtain the original language text.
[0110] The original language text input subunit is used to input the original language text into the decoder, and the decoder performs audio synthesis on the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
[0111] The second original language audio input unit is used to input the second original language audio into the text translation module, and the text translation module performs speech-to-text translation on the second original language audio to obtain the target language text.
[0112] The target language text output submodule is used to obtain the target language text output by the trained speech translation model.
[0113] Furthermore, the audio translation device 200 further includes: a first training dataset acquisition module and a speech translation model training module, wherein:
[0114] The first training dataset acquisition module is used to acquire a first training dataset, wherein the first training dataset includes multiple third original language audios, original language texts corresponding to each of the multiple third original language audios, and target language texts corresponding to each of the multiple third original language audios.
[0115] The speech translation model training module is used to train the speech translation model to be trained based on the target voiceprint and the first training dataset to obtain a trained speech translation model. The speech translation model includes a voiceprint conversion module and a text translation module.
[0116] Furthermore, the voiceprint conversion module is obtained through pre-training, and the audio translation device 200 further includes: a second training dataset acquisition module and a voiceprint conversion model training module, wherein:
[0117] The second training dataset acquisition module is used to acquire a second training dataset, wherein the second training dataset includes multiple fourth original language audios and fifth original language audios corresponding to each of the multiple fourth original language audios, and the voiceprints corresponding to the fourth original language audios and the corresponding fifth original language audios are different.
[0118] The voiceprint conversion model training module is used to train the voiceprint conversion model based on the second training dataset to obtain a pre-trained voiceprint conversion module.
[0119] Furthermore, the voiceprint conversion model includes a content encoder and a decoder, the content encoder being obtained through pre-training. The audio translation device 200 further includes: a third training dataset acquisition module and a speech recognition model training module, wherein:
[0120] The third training dataset acquisition module is used to acquire the third training dataset, wherein the third training dataset includes multiple sixth original language audios and the original language text corresponding to each of the multiple sixth original language audios.
[0121] The speech recognition model training module is used to train the speech recognition model including the content encoder based on the third training dataset to obtain a pre-trained content encoder.
[0122] Furthermore, the voiceprint conversion model also includes a speaker encoder, which is obtained through pre-training. The audio translation device 200 further includes: a fourth training dataset acquisition module and a speaker recognition model training module, wherein:
[0123] The fourth training dataset acquisition module is used to acquire the fourth training dataset, wherein the fourth training dataset includes multiple seventh original language audios and the speaker identifiers corresponding to each of the multiple seventh original language audios.
[0124] The speaker recognition model training module is used to train the speaker recognition model, including the speaker encoder, based on the fourth training dataset to obtain a pre-trained speaker encoder.
[0125] Furthermore, the audio translation device 200 also includes: a standard audio acquisition module and a target voiceprint acquisition module, wherein:
[0126] The standard audio acquisition module is used to acquire standard audio.
[0127] The target voiceprint acquisition module is used to input the standard audio into the speaker encoder, and obtain the target voiceprint by the speaker encoder extracting speaker features from the standard audio.
[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0129] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.
[0130] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0131] Please see Figure 13This document illustrates a structural block diagram of an electronic device 100 provided in an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.
[0132] The processor 110 may include one or more processing cores. The processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and by calling data stored in the memory 120. Optionally, the processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 110 and may be implemented separately using a communication chip.
[0133] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).
[0134] Please see Figure 14This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 300 stores program code that can be called by a processor to execute the methods described in the above method embodiments.
[0135] The computer-readable storage medium 300 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 300 has storage space for program code 310 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 310 may be compressed, for example, in a suitable form.
[0136] In summary, the audio translation method, apparatus, electronic device, and storage medium provided in this application obtain a first original language audio corresponding to an initial voiceprint, extract the original language text corresponding to the first original language audio, synthesize the original language text based on the target voiceprint to obtain a second original language audio corresponding to the target voiceprint, and perform speech-to-text translation on the second original language audio to obtain the target language text. By standardizing the voiceprint of the audio for speech translation scenarios, the problem of different translation results corresponding to different people speaking the same speech can be avoided, thereby improving the accuracy of speech translation.
[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. An audio translation method, characterized in that, The method includes: Obtain the first original language audio corresponding to the initial voiceprint; The first original language audio is input into the voiceprint conversion module of the trained speech translation model. The voiceprint conversion module extracts the content features of the first original language audio to obtain the original language text. Based on the target voiceprint, the original language text is synthesized into audio to obtain the second original language audio corresponding to the target voiceprint. The second original language audio is input into the text translation module of the trained speech translation model, and the text translation module performs speech-to-text translation on the second original language audio to obtain the target language text.
2. The method according to claim 1, characterized in that, The voiceprint conversion module includes a content encoder and a decoder. The process of inputting the first original language audio into the voiceprint conversion module, extracting content features from the first original language audio to obtain the original language text, and synthesizing audio from the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint includes: The first original language audio is input into the content encoder, and the content encoder extracts content features from the first original language audio to obtain the original language text; The original language text is input into the decoder, and the decoder performs audio synthesis on the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint.
3. The method according to claim 1 or 2, characterized in that, Before inputting the first original language audio into the trained speech translation model, the method further includes: Obtain a first training dataset, wherein the first training dataset includes multiple third original language audios, original language texts corresponding to each of the multiple third original language audios, and target language texts corresponding to each of the multiple third original language audios; The speech translation model to be trained is trained based on the target voiceprint and the first training dataset to obtain a trained speech translation model, which includes a voiceprint conversion module and a text translation module.
4. The method according to claim 3, characterized in that, The voiceprint conversion module is obtained through pre-training. Before training the speech translation model to be trained based on the target voiceprint and the first training dataset to obtain the trained speech translation model, the module further includes: Obtain a second training dataset, wherein the second training dataset includes multiple fourth original language audios and fifth original language audios corresponding to each of the multiple fourth original language audios, and the voiceprints corresponding to the fourth original language audios and the fifth original language audios that have a corresponding relationship are different; The voiceprint conversion model is trained based on the second training dataset to obtain a pre-trained voiceprint conversion module.
5. The method according to claim 4, characterized in that, The voiceprint conversion model includes a content encoder and a decoder. The content encoder is obtained through pre-training. Before training the voiceprint conversion model based on the second training dataset to obtain the pre-trained voiceprint conversion module, the model further includes: Obtain a third training dataset, wherein the third training dataset includes multiple sixth original language audios and the original language text corresponding to each of the multiple sixth original language audios; The speech recognition model, including the content encoder, is trained based on the third training dataset to obtain a pre-trained content encoder.
6. The method according to claim 4, characterized in that, The voiceprint conversion model further includes a speaker encoder, which is obtained through pre-training. Before training the voiceprint conversion model based on the second training dataset to obtain the pre-trained voiceprint conversion module, the model further includes: Obtain a fourth training dataset, wherein the fourth training dataset includes multiple seventh original language audios and speaker identifiers corresponding to each of the multiple seventh original language audios; The speaker recognition model, including the speaker encoder, is trained based on the fourth training dataset to obtain a pre-trained speaker encoder.
7. The method according to claim 6, characterized in that, The method further includes: Obtain standard audio; The standard audio is input into the speaker encoder, and the speaker encoder extracts speaker features from the standard audio to obtain the target voiceprint.
8. An audio translation device, characterized in that, The device includes: The original language audio acquisition module is used to acquire the first original language audio corresponding to the initial voiceprint; The target language text acquisition module is used to input the first original language audio into the voiceprint conversion module of the trained speech translation model, extract content features from the first original language audio through the voiceprint conversion module to obtain the original language text, and synthesize the original language text based on the target voiceprint to obtain the second original language audio corresponding to the target voiceprint; input the second original language audio into the text translation module of the trained speech translation model, and perform speech-to-text translation on the second original language audio through the text translation module to obtain the target language text.
9. An electronic device, characterized in that, The method includes a memory and a processor, the memory being coupled to the processor, the memory storing instructions, and the processor performing the method as described in any one of claims 1-7 when the instructions are executed by the processor.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-7.
Citation Information
Patent Citations
Translation method and device
CN107885736A
Acoustic feature sampling method based on multiple parallel convolutional neural networks
CN116631435A
