A method of multilingual translation and related apparatus

Abstract

The application discloses a multilingual translation method and related device, and relates to the technical field of software, which comprises the following steps: obtaining a target video file to be translated, the target video file containing first speech in a source language; calling a multilingual translation model trained in advance based on speech imitation learning, the multilingual translation model having translation capability from the source language to a target language; and translating the first speech into first subtitle text in the target language through the multilingual translation model. The application can directly translate the speech in the source language into the subtitle text in the target language with the help of the multilingual translation model, and through speech imitation learning, the multilingual translation model learns the speech intonation of different film and television characters, understands the meaning of the character tone, and thus more accurately realizes end-to-end multilingual translation, improves translation efficiency and reduces translation cost.

Description

Technical Field

[0001] This application relates to the field of software technology, and in particular to a multilingual translation method and related apparatus. Background Technology

[0002] In the film and television media industry, an increasing number of TV dramas are choosing to go global. The most significant challenge in this process is the translation of audio into subtitles. On one hand, film and television audio includes various accents, differences in speaking speed, background noise, and dialogue involving multiple characters. On the other hand, dialogue between characters contains various cultural allusions, slang, and memes. Therefore, compared to ordinary translation, translating film and television audio into subtitles is much more difficult. Traditional machine translation is ill-suited for this task, and professional translators are generally required, which is time-consuming and labor-intensive. Summary of the Invention

[0003] In view of the above problems, this application provides a multilingual translation method and related apparatus to achieve accurate end-to-end multilingual translation. The specific solution is as follows:

[0004] The first aspect of this application provides a multilingual translation method, the multilingual translation method comprising:

[0005] Obtain the target video file to be translated, wherein the target video file contains the first speech in the source language;

[0006] The multilingual translation model is retrieved, which is pre-trained using an end-to-end method for speech imitation learning;

[0007] The first speech is translated into the first subtitle text in the target language using the multilingual translation model.

[0008] In one possible implementation, the process of obtaining the multilingual translation model by pre-training speech imitation learning using an end-to-end method includes:

[0009] Obtain a translated sample video file, wherein the sample video file contains a second audio in the source language, a second subtitle text in the source language, and a third subtitle text in the target language;

[0010] Extract the speech features of the second speech, and simultaneously perform speech synthesis on the second subtitle text to obtain the third speech;

[0011] Using the second speech as a training sample and the translation result in the target language that approximates the third subtitle text as a training objective, the basic model is fine-tuned with all parameters.

[0012] The multilingual translation model is obtained by imitation learning training on the base model after full parameter fine-tuning, using the second speech, the speech features, and the third speech as training samples, and taking the translation result in the source language as close to the second subtitle text as the training objective.

[0013] In one possible implementation, extracting the speech features of the second speech includes:

[0014] According to the target language and the episode tag to which the sample video file belongs, the second speech is spliced ​​with contextual speech;

[0015] The speech features of the second speech are extracted from the splicing result of the second speech.

[0016] In one possible implementation, the step of fine-tuning the base model with all parameters using the second speech as training samples and the translation result in the target language that approximates the third subtitle text as the training objective includes:

[0017] Based on the mapping relationship between the source language and the target language, determine the training batches corresponding to the second speech and the third subtitle text;

[0018] In the training batch, prompt words are input into the base model, and the second speech is used as the training sample. The training objective is to make the translation result in the target language approximate the third subtitle text. The base model is then fine-tuned with all parameters.

[0019] In one possible implementation, the step of fine-tuning the base model with all parameters using the second speech as training samples and the translation result in the target language approximating the third subtitle text as the training objective further includes:

[0020] Obtain the fourth subtitle text in the target language translated from the second speech by the base model after full parameter fine-tuning;

[0021] The translation quality score of the fourth subtitle text is determined by evaluating the semantic similarity and lexical overlap between the fourth subtitle text and the third subtitle text.

[0022] The fourth subtitle text is divided into positive and negative samples for comparative learning based on the translation quality score.

[0023] The positive and negative samples are used to perform reinforcement feedback learning on the base model after full parameter fine-tuning.

[0024] In one possible implementation, the base model includes a speech encoder and a text encoder. The multilingual translation model is obtained by imitation learning training the base model after full parameter fine-tuning, using the second speech, the speech features, and the third speech as training samples, and aiming for the translation result in the source language to approximate the second subtitle text. This includes:

[0025] The second speech is encoded into a first speech representation by a speech encoder with full parameter fine-tuning, and the first speech representation is concatenated with the speech features to obtain the first speech concatenation result;

[0026] The third speech is encoded into a second speech representation using a speech encoder with fully fine-tuned parameters, and a dot product operation is performed on the first speech concatenation result and the second speech representation.

[0027] The first speech concatenation result and its corresponding dot product operation result are concatenated to obtain the second speech concatenation result;

[0028] The second speech splicing result is encoded into a first text representation using a fully parameter-fine-tuned text encoder, and the second subtitle text is encoded into a second text representation using a fully parameter-fine-tuned text encoder.

[0029] Using the goal of making the first text representation approximate the second text representation as the training objective, the parameters of the fully parameter-fine-tuned speech encoder and the fully parameter-fine-tuned text encoder are adjusted to obtain the multilingual translation model.

[0030] A second aspect of this application provides a multilingual translation apparatus, the multilingual translation apparatus comprising:

[0031] The model training module is used to pre-train a multilingual translation model by performing speech imitation learning using an end-to-end method.

[0032] A multilingual translation module is used to obtain a target video file to be translated, wherein the target video file contains a first speech in the source language; to retrieve the multilingual translation model; and to translate the first speech into a first subtitle text in the target language through the multilingual translation model.

[0033] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the multilingual translation method described in the first aspect or any implementation thereof.

[0034] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program so that the electronic device can implement the multilingual translation method of the first aspect or any implementation thereof.

[0037] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the multilingual translation method described in the first aspect or any implementation thereof.

[0038] By employing the above technical solution, this application provides a multilingual translation method and related apparatus, comprising: acquiring a target video file to be translated, the target video file containing first speech in the source language; retrieving a multilingual translation model, the multilingual translation model being pre-trained using end-to-end speech imitation learning; and translating the first speech into first subtitle text in the target language through the multilingual translation model. This application, by utilizing a multilingual translation model, can directly translate speech in the source language into subtitle text in the target language. Furthermore, through speech imitation learning, the multilingual translation model learns the intonation and tone of different film and television characters, understanding the meaning of their voices, thereby achieving more accurate end-to-end multilingual translation, improving translation efficiency, and reducing translation costs. Attached Figure Description

[0039] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0040] Figure 1 A flowchart illustrating a multilingual translation method provided in this application embodiment;

[0041] Figure 2 This is a partial flowchart illustrating a multilingual translation method provided in an embodiment of this application.

[0042] Figure 3 This is another schematic flowchart of a multilingual translation method provided in an embodiment of this application;

[0043] Figure 4 This is another schematic flowchart of a multilingual translation method provided in an embodiment of this application;

[0044] Figure 5 This is another schematic flowchart of a multilingual translation method provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the structure of a multilingual translation device provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0048] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0050] The export of films and television dramas faces the challenge of translating audio into subtitles. Due to the unique nature of film and television audio, human translation is generally the primary method, requiring people to listen to each segment of audio and translate it into subtitles segment by segment—a time-consuming, labor-intensive, and extremely costly undertaking. Machine translation is also used, typically employing a two-stage approach: first, the audio is recognized as text, and then this text is translated into the target language. However, machine translation for subtitles presents numerous difficulties. First, the process of converting audio to text and then translating it into subtitles results in precision loss, leading to a lower overall accuracy rate. Second, audio is continuous, and the semantic order differs across languages; traditional machine translation, which is generally literal, often makes it difficult to match the audio translation results exactly to the subtitles. Therefore, both human translation and traditional machine translation methods suffer from either low accuracy or slow speed, and are ill-suited for multilingual scenarios.

[0051] To address the aforementioned problems, embodiments of this application provide a multilingual translation method. The multilingual translation method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0052] See Figure 1 , Figure 1 This is a flowchart illustrating a multilingual translation method provided in an embodiment of this application. Figure 1 As shown in the embodiment of this application, a multilingual translation method may include steps S101 to S103, which are described in detail below.

[0053] S101, Obtain the target video file to be translated. The target video file contains the first speech in the source language.

[0054] In this embodiment of the application, the video file to be translated is obtained as the target video file. The target video file contains speech in the source language (i.e., the first speech). For example, the target video file can be an overseas film or television drama to be broadcast, in which the first speech is Chinese speech.

[0055] S102, retrieve the multilingual translation model, which was pre-trained using an end-to-end method for speech imitation learning.

[0056] In this embodiment, a multilingual translation model can be obtained through model training. This multilingual translation model can achieve end-to-end translation from the source language to the target language. Furthermore, by using speech imitation learning, the multilingual translation model can learn the intonation of different film and television characters and understand the context of the dialogue, thereby improving the overall accuracy of end-to-end translation. Only one multilingual translation model is needed to achieve mutual translation between multiple languages, especially Chinese speech to multiple languages.

[0057] See Figure 2 , Figure 2 This is a partial flowchart illustrating a multilingual translation method provided in an embodiment of this application. Figure 2 As shown in the embodiment of this application, a multilingual translation method is provided, wherein the process of obtaining the multilingual translation model by pre-training speech imitation learning using an end-to-end method may include steps S201 to S204, which are described in detail below.

[0058] S201, Obtain the translated sample video file. The sample video file contains the second audio in the source language, the second subtitle text in the source language, and the third subtitle text in the target language.

[0059] In this embodiment, translated video files can be collected and organized as sample video files. These sample video files contain audio in the source language (i.e., second audio), subtitle text in the source language (i.e., second subtitle text), and subtitle text in the target language (i.e., third subtitle text). For example, Chinese audio from overseas broadcast films and television dramas can be collected as the second audio, Chinese subtitle text as the second subtitle text, and foreign language subtitle text as the third subtitle text.

[0060] S202, extract the speech features of the second speech, and at the same time, perform speech synthesis on the second subtitle text to obtain the third speech.

[0061] In this embodiment of the application, the speech features of the second speech can be extracted by an open-source speech feature extraction model. The speech features may include features such as timbre and pitch, and the speech features are stored in vector form to construct the corresponding feature vector matrix.

[0062] In addition, the second subtitle text is processed into speech, and the synthesized speech becomes the third speech. It should be noted that the content of the third speech is the same as that of the second speech, the difference being that the second speech is a real human voice, while the third speech is a synthesized AI voice.

[0063] In one possible implementation, speech features can be accurately extracted by fully obtaining the context of the speech, ensuring the integrity of the speech features. See also Figure 3 , Figure 3 This is another schematic flowchart illustrating a multilingual translation method provided in an embodiment of this application. Figure 3 As shown in the embodiment of this application, a multilingual translation method is provided, wherein step S202, "extracting speech features of the second speech", may include steps S301 to S302, which are described in detail below.

[0064] S301, according to the target language and the series tag to which the sample video file belongs, perform contextual speech splicing on the second speech.

[0065] In this embodiment, the obtained sample video files can be classified according to their target language and the TV series tag they belong to, thereby grouping second audio files with the same target language and the same TV series tag into one set. For example, if the target language is English and the TV series tag is the number of episodes in a TV series, then second audio files with the same target language and the same number of episodes in a TV series can be grouped into one set.

[0066] For the second speech item in a set, recursive sampling can be performed according to the timestamp order. That is, the second speech items are concatenated in ascending order of timestamp. A 1-second silence signal (all zeros) is used to fill the gaps when concatenating two second speech items. Simultaneously, the corresponding second subtitle texts are also concatenated and separated by " / / ". For example, the concatenation result is "Jing Qing? / / Awak marah saya? / / Jangan cuba menghina saya". The concatenation stops if the total length of the concatenated second speech item exceeds 30 seconds, or if the time interval between concatenations exceeds 10 seconds. This allows for a complete understanding of the context of the speech, enabling subsequent models to better handle speech continuity issues.

[0067] S302, extract the speech features of the second speech from the splicing result of the second speech.

[0068] In this embodiment of the application, after splicing the upper and lower speech, speech features can be extracted from the splicing result of the second speech using an open-source speech feature extraction model.

[0069] S203 uses the second speech as the training sample and the translation result in the target language to approximate the third subtitle text as the training objective to fine-tune the full parameters of the basic model.

[0070] In this embodiment, Qwen2.5-Omni can be selected as the base model. The image and video processing units of Qwen2.5-Omni are reduced to only the speech and text units. Prompt words for translating the source language speech into target language subtitle text are constructed. These prompt words are fed into the base model, and the base model is fine-tuned with full parameters using a second speech sample and the goal of the translation result in the target language approximating the third subtitle text. After this training, the base model can be equipped with the ability to translate the source language speech into target language subtitle text.

[0071] In one possible implementation, since single-language translation tasks make the optimization objective more explicit for the model, to enable the multilingual translation model to translate multiple languages ​​(e.g., Chinese speech to foreign language subtitles, or foreign language speech to Chinese subtitles), this application embodiment uses batch learning to ensure the learning effect of different language translations. This time-for-space strategy guarantees the model's learning effect, and training can be done in batches according to the mapping relationship between the source language and the target language. See [link to implementation details]. Figure 4 , Figure 4 This is another schematic flowchart illustrating a multilingual translation method provided in an embodiment of this application. Figure 4As shown in the embodiment of this application, a multilingual translation method is provided, wherein step S203, "using the second speech as a training sample and the translation result in the target language that is close to the third subtitle text as a training target, and fine-tuning the full parameters of the basic model", may include steps S401 to S402, which are described in detail below.

[0072] S401, Based on the mapping relationship between the source language and the target language, determine the training batch corresponding to the second speech and the third subtitle text.

[0073] In this embodiment, the mapping relationship between the source language and target language in the sample video file can be determined according to the source language of the second audio and the target language of the third subtitle text. For example, if the source language of the second audio is Chinese and the target language of the third subtitle text is a foreign language, the mapping relationship is "Chinese → foreign language". Similarly, if the source language of the second audio is a foreign language and the target language of the third subtitle text is Chinese, the mapping relationship is "foreign language → Chinese". Therefore, the corresponding training batches of the second audio and third subtitle text can be determined based on the mapping relationship between the source and target languages. For example, the training batch of sample video files (containing the second audio and third subtitle text) with a mapping relationship of "Chinese → foreign language" is batch 1, and the training batch of sample video files (containing the second audio and third subtitle text) with a mapping relationship of "foreign language → Chinese" is batch 2.

[0074] S402 involves inputting prompt words into the basic model during the training batch, using the second speech as the training sample, and taking the translation result in the target language as close as possible to the third subtitle text as the training objective, to fine-tune all parameters of the basic model.

[0075] In this embodiment, the basic model is fine-tuned with full parameters by inputting corresponding prompt words into the training batch, using the second speech as the training sample, and aiming for the translation result in the target language to approximate the third subtitle text. For example, the basic model is first trained in batch 1 to construct prompt words for translating Chinese speech into foreign language subtitle text. For example, the prompt words could be "[{"messages": [{"role": "system", "content": "You are a speech translation model."}, {"role": "user", "content":" <audio>Listen to the provided Chinese speech and produce a translation in Malay text."}], "audios": ["Chinese_0_to_5.mp3"],}”, This prompt word is fed into the basic model, and the basic model is fine-tuned with all parameters using the second speech and third subtitle text from the sample video files in this batch. Through this training, the basic model can acquire the ability to translate Chinese speech into foreign language subtitle text. Then, the basic model is trained in batch 2 to construct prompt words for translating foreign language speech into Chinese subtitle text. For example, the prompt word could be "[{"messages": [{"role": "system", "content": "You are a speechtranslation model."}, {"role": "user", "content": " <audio>Listen to the provided Malay speech and produce a translation in Chinese text."}], "audios": ["Malay_0_to_5.mp3"],}”, This prompt word is fed into the basic model, and the basic model is fine-tuned with all parameters using the second speech and the third subtitle text from the sample video files in this batch. Through this training, the basic model can be equipped with the ability to translate foreign language speech into Chinese subtitle text.

[0076] In one possible implementation, to make the model output closer to human preferences, methods such as RLHF (Reinforcement Learning from Human Feedback) can be used to reinforce the base model through feedback learning. The multilingual translation method provided in this application embodiment, wherein step S203, "using the second speech as a training sample and the translation result in the target language approximating the third subtitle text as a training objective, to fine-tune the base model with all parameters," may further include the following steps:

[0077] The base model with fully fine-tuned parameters is used to translate the fourth subtitle text in the target language into the second speech. The translation quality score of the fourth subtitle text is determined by evaluating the semantic similarity and lexical overlap between the fourth and third subtitle texts. The fourth subtitle text is divided into positive and negative samples for contrastive learning according to the translation quality score. The positive and negative samples are used to perform reinforcement feedback learning on the base model with fully fine-tuned parameters.

[0078] In this embodiment of the application, after the base model under each training batch has completed full parameter fine-tuning, the base model with full parameter fine-tuning can be used to translate the second speech under that training batch to obtain the subtitle text in the target language (i.e., the fourth subtitle text).

[0079] Furthermore, the constructed reward function is used to evaluate the semantic similarity and lexical overlap between the fourth and third subtitle texts, thereby determining the translation quality score of the fourth subtitle text. The reward function is f = 0.7 × COMET + 0.3 × BLEU, where COMET (semantic similarity) is a metric for evaluating translation quality at the semantic level; higher semantic similarity indicates higher translation quality. BLEU (lexical overlap) is a metric for evaluating translation quality at the lexical level; higher lexical overlap also indicates higher translation quality. It should be noted that the evaluation of semantic similarity and lexical overlap can be implemented using existing open-source models.

[0080] Further, according to the translation quality scores of the fourth subtitle text, the fourth subtitle text can be divided into positive samples and negative samples. For example, the fourth subtitle text with a translation quality score greater than or equal to the corresponding score threshold is used as a positive sample, and conversely, the fourth subtitle text with a translation quality score less than the corresponding score threshold is used as a negative sample.

[0081] Finally, the DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) algorithm is used to perform reinforcement feedback learning on the base model fine-tuned with all parameters using positive samples and negative samples, so that the translation level of the base model is further improved compared to before, and the translation results are more in line with the translation style of real people.

[0082] S204. Using the second voice, voice features, and the third voice as training samples, and taking the translation result in the source language approaching the second subtitle text as the training target, perform imitation learning training on the base model fine-tuned with all parameters to obtain a multilingual translation model.

[0083] In the embodiments of the present application, in order to enable the model to imitate the speech intonation of movie and television characters and accurately translate the speech in the source language into subtitle text in the target language, the second voice, voice features, and the third voice are used as training samples, and taking the translation result in the source language approaching the second subtitle text as the training target, perform imitation learning training on the base model fine-tuned with all parameters to obtain a multilingual translation model. The multilingual translation model can imitate the pronunciation characteristics of different speech intonations and better adapt to the translation of different movie and television characters. For example, for the same Chinese speech "你去过北京。” and "你去过北京?”, the intonations of a declarative sentence and an interrogative sentence are different, and the translated content will naturally be different.

[0084] See Figure 5 , Figure 5 which is another part of the flowchart of a multilingual translation method provided by the embodiments of the present application. As Figure 5 shown, a multilingual translation method provided by the embodiments of the present application, where the base model includes a speech encoder and a text encoder. Step S204 "Using the second voice, voice features, and the third voice as training samples, and taking the translation result in the source language approaching the second subtitle text as the training target, perform imitation learning training on the base model fine-tuned with all parameters to obtain a multilingual translation model" can include steps S501 to S505, which will be described in detail below.

[0085] S501. Encode the second voice into a first voice representation through the speech encoder fine-tuned with all parameters, and splice the first voice representation with the voice features to obtain a first voice splicing result.

[0086] In this embodiment of the application, the basic model includes a speech encoder and a text encoder. After the full parameter fine-tuning in step S203, the second speech can be encoded into a corresponding speech representation (i.e., the first speech representation) by the speech encoder after full parameter fine-tuning, and the speech features and the first speech features are concatenated to obtain the corresponding speech concatenation result (i.e., the first speech concatenation result). The first speech concatenation result contains the speech content and speech features of real human speech.

[0087] S502 encodes the third speech into a second speech representation using a speech encoder with fully fine-tuned parameters, and performs a dot product operation on the first speech concatenation result and the second speech representation.

[0088] In this embodiment of the application, the third speech is encoded into a corresponding speech representation (i.e., the second speech representation) by a speech encoder with full parameter fine-tuning, and a dot product operation is performed on the first speech splicing result and the second speech representation. The result of the dot product operation can characterize the difference between the first speech splicing result and the second speech representation.

[0089] S503, the first speech concatenation result and its corresponding dot product operation result are concatenated to obtain the second speech concatenation result.

[0090] In this embodiment of the application, the first speech splicing result and its corresponding dot product operation result are spliced ​​together to obtain the corresponding speech splicing result (i.e., the second speech splicing result).

[0091] S504, the second speech concatenation result is encoded into a first text representation by a text encoder with full parameter fine-tuning, and at the same time, the second subtitle text is encoded into a second text representation by a text encoder with full parameter fine-tuning.

[0092] In this embodiment of the application, the second speech splicing result is encoded into a corresponding text representation (i.e., the first text representation) by a text encoder with full parameter fine-tuning, and at the same time, the second subtitle text is encoded into a corresponding text representation (i.e., the second text representation) by a text encoder with full parameter fine-tuning.

[0093] S505 aims to train the first text representation to approximate the second text representation. It adjusts the parameters of the fully parameterized speech encoder and the fully parameterized text encoder to obtain a multilingual translation model.

[0094] In this embodiment, the training objective is to make the first text representation approximate the second text representation. The parameters of the fully parameter-fine-tuned speech encoder and the fully parameter-fine-tuned text encoder are adjusted to complete the imitation learning and obtain the multilingual translation model.

[0095] S103 uses a multilingual translation model to translate the first speech into the first subtitle text in the target language.

[0096] In this embodiment, the first speech is input to a multilingual translation model, which translates the first speech into subtitle text (i.e., the first subtitle text) in the target language. Specifically, the speech encoder in the multilingual translation model encodes the first speech into a corresponding speech representation, and the text encoder encodes the speech representation of the first speech into a corresponding text representation. After decoding, the first subtitle text is obtained. Verification has shown that the multilingual translation model supports translating Chinese speech from various films and television dramas into more than ten foreign languages ​​based on their intonation, such as English, French, Thai, and Malay. It can translate the speech of characters with different intonations from period dramas and modern dramas very well, with results far exceeding those of traditional machine translation models.

[0097] Based on the above description, the multilingual translation method provided in this application allows the model to learn and imitate the tone, timbre, and context of film and television characters, thereby more accurately translating the speech of the source language into subtitle text in multiple target languages. It has strong generalization function, high translation quality, fast speed, and high reusability, and greatly improves the accuracy and speed of film and television speech translation.

[0098] The above describes a multilingual translation method provided by the embodiments of this application. The following will describe the apparatus for performing the above multilingual translation method.

[0099] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a multilingual translation device provided in an embodiment of this application. Figure 6 As shown in the figure, an embodiment of this application provides a multilingual translation device, comprising:

[0100] The model training module 601 is used to pre-train a multilingual translation model by performing speech imitation learning using an end-to-end method.

[0101] The multilingual translation module 602 is used to obtain the target video file to be translated, which contains the first speech in the source language; to call the multilingual translation model; and to translate the first speech into the first subtitle text in the target language through the multilingual translation model.

[0102] In one possible implementation, the model training module 601 is specifically used for:

[0103] Obtain translated sample video files, which contain second speech in the source language, second subtitle text in the source language, and third subtitle text in the target language. Extract speech features from the second speech and synthesize the third speech from the second subtitle text. Use the second speech as training samples and the goal of the translation result in the target language approximating the third subtitle text as training objective to fine-tune the base model with all parameters. Use the second speech, speech features, and third speech as training samples and the goal of the translation result in the source language approximating the second subtitle text as training objective to train the fine-tuned base model with imitation learning to obtain a multilingual translation model.

[0104] In one possible implementation, the model training module 601 for extracting speech features of the second speech is specifically used for:

[0105] Based on the target language and the episode tags to which the sample video file belongs, the second speech is spliced ​​together with contextual speech.

[0106] The speech features of the second speech are extracted from the splicing result of the second speech.

[0107] In one possible implementation, the model training module 601, which fine-tunes all parameters of the base model using the second speech as training samples and the translation result in the target language approximating the third subtitle text as the training objective, is specifically used for:

[0108] Based on the mapping relationship between the source language and the target language, the training batches corresponding to the second speech and the third subtitle text are determined. In the training batch, prompt words are input into the basic model, and the second speech is used as the training sample. The training objective is to make the translation result in the target language approach the third subtitle text. The basic model is then fine-tuned with all parameters.

[0109] In one possible implementation, the model training module 601, used to fine-tune the base model with full parameters using the second speech as training samples and the translation result in the target language approximating the third subtitle text as the training objective, is also used for:

[0110] The base model with fully fine-tuned parameters is used to translate the fourth subtitle text in the target language into the second speech. The translation quality score of the fourth subtitle text is determined by evaluating the semantic similarity and lexical overlap between the fourth and third subtitle texts. The fourth subtitle text is divided into positive and negative samples for contrastive learning according to the translation quality score. The positive and negative samples are used to perform reinforcement feedback learning on the base model with fully fine-tuned parameters.

[0111] In one possible implementation, the base model includes a speech encoder and a text encoder. A model training module 601, which uses a second speech, speech features, and a third speech as training samples, and aims to train the base model (after fine-tuning all parameters) to approximate the translation result in the source language to the second subtitle text, is used to obtain the multilingual translation model. Specifically, this module is used for:

[0112] The second speech is encoded into a first speech representation using a fully parameter-fine-tuned speech encoder, and the first speech representation is concatenated with speech features to obtain the first speech concatenation result. The third speech is encoded into a second speech representation using a fully parameter-fine-tuned speech encoder, and a dot product operation is performed on the first speech concatenation result and the second speech representation. The first speech concatenation result and its corresponding dot product operation result are concatenated to obtain the second speech concatenation result. The second speech concatenation result is encoded into a first text representation using a fully parameter-fine-tuned text encoder, and the second subtitle text is encoded into a second text representation using a fully parameter-fine-tuned text encoder. With the training objective of making the first text representation approximate the second text representation, the parameters of the fully parameter-fine-tuned speech encoder and the fully parameter-fine-tuned text encoder are adjusted to obtain a multilingual translation model.

[0113] It should be noted that the detailed functions of each module in the embodiments of this application can be found in the corresponding disclosures of the above-mentioned multilingual translation method embodiments, and will not be repeated here.

[0114] This application also provides an electronic device in its embodiments. See also... Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0115] like Figure 7 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0116] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, memory cards, hard drives, etc.; and communication devices 709. Communication device 709 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0117] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the multilingual translation methods provided in this application.

[0118] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the multilingual translation methods provided in this application.

[0119] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0121] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0122] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).< / audio> < / audio>

Claims

1. A multilingual translation method, characterized in that, The multilingual translation method includes: Obtain the target video file to be translated, wherein the target video file contains the first speech in the source language; The multilingual translation model is retrieved. The multilingual translation model is pre-trained using an end-to-end method for speech imitation learning. Based on the speech imitation learning, the multilingual translation model learns different real human speech intonations to determine different translation content corresponding to different real human speech intonations. The first speech is translated into the first subtitle text in the target language using the multilingual translation model. The process of obtaining the multilingual translation model through pre-training speech imitation learning using an end-to-end method includes: Obtain a translated sample video file, wherein the sample video file contains a second audio in the source language, a second subtitle text in the source language, and a third subtitle text in the target language; Extract the speech features of the second speech, and simultaneously synthesize the second subtitle text to obtain a third speech. The third speech has the same content as the second speech, the second speech is a real human voice, and the third speech is a synthesized AI speech. Using the second speech as a training sample and the translation result in the target language that approximates the third subtitle text as a training objective, the basic model is fine-tuned with all parameters. Using the second speech, the speech features, and the third speech as training samples, and taking the text representation in the source language as approximating the second subtitle text as the training objective, the multilingual translation model is obtained by imitation learning training on the base model after full parameter fine-tuning. The multilingual translation model learns the intonation of real human speech based on the difference between real human speech and AI synthesized speech.

2. The multilingual translation method according to claim 1, characterized in that, The extraction of speech features from the second speech includes: According to the target language and the episode tag to which the sample video file belongs, the second speech is spliced ​​with contextual speech; The speech features of the second speech are extracted from the splicing result of the second speech.

3. The multilingual translation method according to claim 1, characterized in that, The process of fine-tuning the basic model using the second speech as training samples and the translation result in the target language that approximates the third subtitle text as the training objective, includes: Based on the mapping relationship between the source language and the target language, determine the training batches corresponding to the second speech and the third subtitle text; In the training batch, prompt words are input into the base model, and the second speech is used as the training sample. The training objective is to make the translation result in the target language approximate the third subtitle text. The base model is then fine-tuned with all parameters.

4. The multilingual translation method according to claim 3, characterized in that, The step of using the second speech as a training sample and the translation result in the target language that approximates the third subtitle text as a training objective, and fine-tuning all parameters of the basic model, further includes: Obtain the fourth subtitle text in the target language translated from the second speech by the base model after full parameter fine-tuning; The translation quality score of the fourth subtitle text is determined by evaluating the semantic similarity and lexical overlap between the fourth subtitle text and the third subtitle text. The fourth subtitle text is divided into positive and negative samples for comparative learning based on the translation quality score. The positive and negative samples are used to perform reinforcement feedback learning on the base model after full parameter fine-tuning.

5. The multilingual translation method according to claim 1, characterized in that, The base model includes a speech encoder and a text encoder. The multilingual translation model is obtained by training the base model, after fine-tuning all parameters, using the second speech, the speech features, and the third speech as training samples, and aiming for the text representation in the source language to approximate the second subtitle text. This training includes: The second speech is encoded into a first speech representation by a speech encoder with full parameter fine-tuning, and the first speech representation is concatenated with the speech features to obtain the first speech concatenation result; The third speech is encoded into a second speech representation using a speech encoder with fully fine-tuned parameters, and a dot product operation is performed on the first speech concatenation result and the second speech representation. The first speech concatenation result and its corresponding dot product operation result are concatenated to obtain the second speech concatenation result; The second speech splicing result is encoded into a first text representation using a fully parameter-fine-tuned text encoder, and the second subtitle text is encoded into a second text representation using a fully parameter-fine-tuned text encoder. Using the goal of making the first text representation approximate the second text representation as the training objective, the parameters of the fully parameter-fine-tuned speech encoder and the fully parameter-fine-tuned text encoder are adjusted to obtain the multilingual translation model.

6. A multilingual translation device, characterized in that, The multilingual translation device includes: The model training module is used to pre-train a multilingual translation model by performing speech imitation learning using an end-to-end method. The multilingual translation model learns different speech intonations based on the speech imitation learning in order to determine different translation content corresponding to different speech intonations. A multilingual translation module is used to acquire a target video file to be translated, wherein the target video file contains a first audio in the source language; to retrieve the multilingual translation model; and to translate the first audio into a first subtitle text in the target language using the multilingual translation model. Specifically, the model training module is used to: obtain translated sample video files, wherein the sample video files contain second speech in the source language, second subtitle text in the source language, and third subtitle text in the target language; Extract the speech features of the second speech, and simultaneously synthesize the second subtitle text to obtain a third speech. The third speech has the same content as the second speech, the second speech is a real human voice, and the third speech is a synthesized AI speech. Using the second speech as a training sample and the translation result in the target language that approximates the third subtitle text as a training objective, the basic model is fine-tuned with all parameters. Using the second speech, the speech features, and the third speech as training samples, and taking the text representation in the source language as approximating the second subtitle text as the training objective, the multilingual translation model is obtained by imitation learning training on the base model after full parameter fine-tuning. The multilingual translation model learns the intonation of real human speech based on the difference between real human speech and AI synthesized speech.

7. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the multilingual translation method as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the multilingual translation method as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the multilingual translation method as described in any one of claims 1 to 5.

Images