Text translation method and device, computer device and storage medium

Abstract

The application discloses a text translation method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The application pre-trains an initial translation model through original text and annotated text, and continues training of the original text and the positive reference translation text obtained after knowledge distillation of a positive teacher model on the basis of the first translation model obtained through pre-training, so that the second translation model obtained finally can not only learn the original data distribution in the original text and the annotated text, such as the high mode distribution of a target language, the low-frequency word distribution and the like, but also learn the certainty knowledge after knowledge distillation of the positive teacher model, so that accurate prediction can be performed on high-frequency words and low-frequency words in parallel decoding, and the translation accuracy of the machine translation model supporting parallel translation is greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a text translation method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of computer technology, in the field of machine translation, machine translation methods based on deep learning have made good progress. Currently, the mainstream neural machine translation model is the AT (Autoregressive Machine Translation) model, that is, the generation of each translated character depends on the previous translation result. The AT model can only generate the translation word by word, and the decoding speed is slow.

[0003] On this basis, the NAT (Non-Autoregressive Machine Translation) model emerged. The NAT model can parallel-decode the entire sentence translation, significantly improving the decoding speed of the model. However, the high multimodality of language brings great difficulties to the translation accuracy of the NAT model. Taking Japanese to English translation as an example, multimodality means that the Japanese sentence "彼は日本語が上手です" can be translated into two equivalent English sentences "he is very good at Japanese" and "he speaks Japanese very well", but the following two seemingly similar sentences "he speaks very good at Japanese" and "he is very good at verywell" have no meaning. It is precisely because the parallel decoding method of the NAT model breaks the conditional dependence between output characters that it hinders the NAT model from correctly capturing the high-mode distribution of the target language, resulting in it being difficult to avoid the appearance of the latter, that is, the meaningless sentence in the above example, in the conditionally independent parallel decoding. Therefore, the translation accuracy of machine translation models that support parallel translation, such as the NAT model, is poor. Summary of the Invention

[0004] Embodiments of this application provide a text translation method, apparatus, computer device, and storage medium, which can improve the translation accuracy of machine translation models that support parallel translation, such as the NAT model. The technical solution is as follows: On the one hand, a text translation method is provided, and the method includes: Based on the original text and the annotated text corresponding to the semantics of the original text, adjust the parameters of the initial translation model to obtain a first translation model, where the initial translation model supports parallel translation of the input text; The original text is translated by calling the positive teacher model to obtain a positive reference translation. The positive reference translation and the labeled text belong to the same language. The positive teacher model supports serial translation of the input text. Based on the original text and the positive reference translation, the parameters of the first translation model are adjusted to obtain a second translation model, which is used to translate the input text into semantically corresponding output text.

[0005] On the one hand, a text translation method is provided, which includes: The original text is translated by calling a positive teacher model to obtain a positive reference translation. The positive teacher model supports serial translation of the input text. The reverse teacher model is invoked to translate the annotated text corresponding to the semantics of the original text, resulting in a reverse reference translation. The annotated text and the forward reference translation belong to the same language, and the reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text. Based on the original text, the forward reference translation, the reverse reference translation, and the labeled text, the parameters of the initial translation model are adjusted to obtain the target translation model, which is used to translate the input text into semantically corresponding output text.

[0006] On the one hand, a text translation device is provided, the device comprising: The first adjustment module is used to adjust the parameters of the initial translation model based on the original text and the labeled text corresponding to the semantics of the original text to obtain the first translation model. The initial translation model supports parallel translation of the input text. The translation module is used to call the positive teacher model to translate the original text and obtain a positive reference translation. The positive reference translation and the labeled text belong to the same language. The positive teacher model supports serial translation of the input text. The second adjustment module is used to adjust the parameters of the first translation model based on the original text and the positive reference translation to obtain a second translation model. The second translation model is used to translate the input text into semantically corresponding output text.

[0007] In one possible implementation, the first adjustment module includes: The first translation unit is used to call the initial translation model to translate the original text and obtain the first predicted text; A determining unit is configured to determine a first loss value based on the labeled text and the first predicted text; The first adjustment unit is used to adjust the parameters of the initial translation model based on the first loss value to obtain the first translation model.

[0008] In one possible implementation, the first adjustment unit is used to: Based on the labeled text and the first predicted text, determine the BLEU score for the bilingual assessment study; If neither the first loss value nor the BLEU value meets the first stopping condition, the parameters of the initial translation model are iteratively adjusted. If either the first loss value or the BLEU value meets the first stopping condition, the iteration stops, and the first translation model is obtained.

[0009] In one possible implementation, the first stopping condition is: the first loss value is less than or equal to a first loss threshold; or, the BLEU value is greater than the first BLEU threshold.

[0010] In one possible implementation, the second adjustment module is used to: The first translation model is invoked to translate the original text, resulting in the second predicted text; Based on the positive reference translation and the second predicted text, a second loss value is determined; Based on the second loss value, the parameters of the first translation model are adjusted to obtain the second translation model.

[0011] In one possible implementation, the second adjustment module includes: The second translation unit is used to call the reverse teacher model to translate the labeled text and obtain a reverse reference translation. The reverse reference translation is in the same language as the original text. The reverse teacher model supports serial translation of the input text. The second adjustment unit is used to adjust the parameters of the first translation model based on the original text, the forward reference translation, the reverse reference translation, and the labeled text, to obtain the second translation model.

[0012] In one possible implementation, the second adjustment unit is used to: The first translation model is invoked to translate the original text, resulting in the second predicted text; Based on the positive reference translation and the second predicted text, a second loss value is determined; The first translation model is invoked to translate the reverse reference translation, resulting in a third predicted text. Based on the labeled text and the third predicted text, a third loss value is determined; Based on the second loss value and the third loss value, the parameters of the first translation model are adjusted to obtain the second translation model.

[0013] In one possible implementation, the device further includes: The third adjustment module is used to adjust the parameters of the second translation model based on the original text and the positive reference translation to obtain the third translation model.

[0014] On the one hand, a text translation device is provided, the device comprising: The first translation module is used to call the positive teacher model to translate the original text and obtain a positive reference translation. The positive teacher model supports serial translation of the input text. The second translation module is used to call the reverse teacher model to translate the annotated text corresponding to the semantics of the original text to obtain a reverse reference translation. The annotated text and the forward reference translation belong to the same language, and the reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text. The adjustment module is used to adjust the parameters of the initial translation model based on the original text, the forward reference translation, the reverse reference translation, and the labeled text to obtain the target translation model, which is used to translate the input text into semantically corresponding output text.

[0015] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the at least one computer program being loaded and executed by the one or more processors to implement a text translation method as described in any of the possible implementations above.

[0016] On the one hand, a storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to implement a text translation method as described in any of the possible implementations above.

[0017] On one hand, a computer program product or computer program is provided, the computer program product or computer program comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, enabling the computer device to perform the text translation method of any of the above possible embodiments.

[0018] The beneficial effects of the technical solutions provided in this application include at least the following: The initial translation model is pre-trained using both raw and annotated text. Based on the pre-trained first translation model, further training is conducted using the original text and positive reference translations obtained after knowledge distillation from the positive teacher model. This results in a second translation model that not only learns the original data distribution in the raw and annotated text, such as the high-modulus distribution and low-frequency word distribution of the target language, but also learns the deterministic knowledge obtained after knowledge distillation from the positive teacher model. This enables accurate prediction of both high-frequency and low-frequency words during parallel decoding, significantly improving the translation accuracy of machine translation models supporting parallel translation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a comparison diagram of an AT model and a NAT model provided in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation environment of a text translation method provided in this application embodiment; Figure 3 This is a schematic diagram illustrating the principle of a knowledge distillation method provided in an embodiment of this application; Figure 4 This is a flowchart of a text translation method provided in an embodiment of this application; Figure 5 This is a schematic flowchart illustrating the principle of a text translation method provided in an embodiment of this application; Figure 6 This is a flowchart of a text translation method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the architecture of an initial translation model provided in an embodiment of this application; Figure 8 This is a schematic diagram of the architecture of a positive teacher model provided in an embodiment of this application; Figure 9 This is a flowchart of a text translation method provided in an embodiment of this application; Figure 10 This is a flowchart of a text translation method provided in an embodiment of this application; Figure 11 This is a flowchart illustrating the principle of a text translation method provided in an embodiment of this application; Figure 12This is a schematic diagram of the structure of a text translation device provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of a text translation device provided in an embodiment of this application; Figure 14 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application; Figure 15 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0022] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0023] In this application, the term "at least one" means one or more, and "multiple" means two or more, for example, multiple first positions means two or more first positions.

[0024] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0025] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include audio processing, computer vision, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation.

[0026] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0027] Enabling computers to hear, see, speak, and feel represents the future direction of human-computer interaction. Natural Language Processing (NLP) is a crucial area within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus has a close connection to linguistic research. NLP technologies typically include machine translation, text preprocessing, semantic understanding, question answering, and knowledge graphs.

[0028] In the field of Natural Language Processing (NLP), Machine Translation (MT), also known as Automatic Translation, refers to the process of using computers to convert one natural language (source language) into another natural language (target language). It is a branch of computational linguistics and one of the ultimate goals of artificial intelligence, possessing significant scientific research value. Simultaneously, machine translation also has important practical value. With economic globalization and the rapid development of the internet, machine translation technology plays an increasingly important role in promoting political, economic, and cultural exchanges. Deep learning-based translation models have achieved considerable success in machine translation. The technical solutions provided in this application involve artificial intelligence-based machine translation and other technologies, which will be described in detail below.

[0029] The following will explain the terms used in the embodiments of this application: Deep Learning (DL) is a branch of machine learning that attempts to perform high-level abstractions of data using multiple processing layers that contain complex structures or consist of multiple nonlinear transformations.

[0030] Neural Network (NN): A deep learning model in the fields of machine learning and cognitive science that mimics the structure and function of biological neural networks.

[0031] Machine translation (MT): The automatic translation of one natural language into another using devices such as computers.

[0032] Statistical Machine Translation (SMT): A traditional machine translation technique based on Bayesian probability statistics, which can use statistical feature mapping to achieve text conversion at the character level or phrase level.

[0033] Neural Machine Translation (NMT): A machine translation technique based on neural networks.

[0034] Self-Attention Network (SAN): A neural network structure model based on the self-attention mechanism.

[0035] Sequence to Sequence Model (Seq2Seq): A deep neural network model where both input and output are sequences.

[0036] Transformer: The Transformer adopts an encoder-decoder framework based on SAN network and belongs to the model structure for sequence-to-sequence Seq2Seq generation.

[0037] Tokens are the smallest semantic units in machine translation, also known as word segments. In English, a token refers to a word (vocabulary) or an English punctuation mark. In Chinese, a token refers to a Chinese word (which may include one or more Chinese characters) or a Chinese punctuation mark.

[0038] Bilingual Evaluation Understudy (BLEU) is an evaluation metric for machine translation models. A higher BLEU value indicates a better translation performance of the machine translation model.

[0039] In recent years, deep neural networks have become a core model in the field of artificial intelligence, widely used in speech recognition, computer vision, natural language processing, and other areas. Neural network models have demonstrated outstanding performance in various tasks, particularly machine translation. In machine translation, NMT models based on the encoder-decoder framework have seen rapid development, with performance far exceeding that of traditional SMT models. However, as NMT models become deeper and the data flow increases, decoding efficiency has gradually become one of the main problems in its application.

[0040] Mainstream NMT models employ an autoregressive machine translation (AT) mechanism, meaning that AT models generate translations word by word. The generation of each translated character depends on the previous translation results, thus supporting serial translation and resulting in slow decoding speed. Building on this, non-autoregressive machine translation (NAT) models have emerged. Because NAT models independently model the generation of each translated character, they can decode the entire sentence in parallel, supporting parallel translation and significantly improving decoding speed.

[0041] Figure 1 This is a comparison diagram of an AT model and a NAT model provided in an embodiment of this application, such as... Figure 1 As shown, the left part 101 illustrates the data processing flow of the AT model. It can be seen that the generation of the translation character in each step of the AT model depends on the previous translation results. The right part 102 illustrates the data processing flow of the NAT model. It can be seen that the NAT model independently models the generation of each translation character, thus enabling the generation of the entire sentence translation at the same time.

[0042] The standard AT model, given the source statement x = x1, …, x I Below, we directly optimize the target-side statement y = y1,…, y J Conditional probability:

[0043] in, These are the parameters of the AT model. yes The previous translation is above. The probability P(y|x) defines the encoder-decoder framework based on a neural network. The parameters of the AT model... The training process is to maximize the training samples. likelihood:

[0044] in, denoted by , where M represents the loss value and M represents the sample size.

[0045] Ideally, the conditional probability of the NAT model would be freed. Conditional dependency:

[0046] It should be noted that, in the embodiments of this application, "source end" refers to the untranslated characters, phrases or sentences on the source language side, while "target end" refers to the translated characters, phrases or sentences on the target language side. Therefore, the translation process is the process of converting the sentence to be translated from the source language into the target language.

[0047] Therefore, it can be seen that during training, the encoders of the AT and NAT models are the same, but the decoders are different. Regarding the length of the target statement, the AT model predicts the end-of-line character. <eos>The AT model implicitly predicts the statement length, while the NAT model explicitly predefines the output statement length. The decoder input of the AT model requires a shift mask (right shift) technique to mask the content after time t, while the NAT model does not.

[0048] The NAT model can effectively alleviate the slow decoding speed of the AT model. As shown in Table 1, two AT models are listed: the basic Transformer (Transformer-Base, Ro-En Romanian-English teacher model) and the enhanced Transformer (Transformer-Big, En-De English-German teacher model), as well as five NAT models: NAT model, Iterative NAT model, Disentangled ContextTransformer (DisCo) model, Mask-Predict NAT model, and Levenshtein NAT model, with differences in decoding speed and translation performance (BLEU).

[0049]

[0050] Among them, ALF (Accuracy Of Low-Frequency Words) is a metric used to evaluate whether low-frequency words are translated accurately; Speed ​​is a metric used to measure decoding speed, and here the decoding speed of the Transformer-Base model is used as the unit to calculate the decoding speedup of each other model; BLEU is a metric used to measure the overall translation performance of the model.

[0051] As can be seen from Table 1 above, the Speed column reflects that the decoding speeds of the 5 NAT models are all higher than those of the 2 NAT models, indicating that machine translation models that support parallel translation, such as the NAT model, have a better decoding speed than machine translation models that support serial translation, such as the AT model. However, the multimodality in text generation poses a fundamental challenge to the NAT model. Since language is highly multimodal, taking Japanese-to-English translation as an example, multimodality means that for the Japanese sentence "彼は日本語が上手です”, it can be translated into two equivalent English sentences "he is very good at Japanese” and "he speaks Japanese very well”, but the following two seemingly similar sentences "he speaks very good at Japanese” and "he is very good at very well” are meaningless. Exactly because the parallel decoding method of the NAT model breaks the conditional dependence between output characters, it hinders the model from correctly capturing the high-modal distribution of the target language, resulting in it being difficult to avoid the appearance of the latter meaningless sentences in the above examples in conditional independent parallel decoding. This is the main reason why there is still a certain gap in translation performance between machine translation models that support parallel translation, such as the NAT model, and machine translation models that support serial translation, such as the AT model. As shown in Table 1 above, by observing the BLEU column, it can be found that whether in the En-De (English-to-German) task or in the Ro-En (Romanian-to-English) task, the BLEU values of the 2 AT models are higher than those of the 5 NAT models, indicating that machine translation models that support serial translation, such as the AT model, have better translation performance than machine translation models that support parallel translation, such as the NAT model.

[0052] In view of this, an embodiment of the present application provides a text translation method, aiming to narrow the gap in translation performance between machine translation models that support parallel translation, such as the NAT model, and machine translation models that support serial translation, such as the AT model, while maintaining the decoding speed advantage of machine translation models that support parallel translation, such as the NAT model. As a result, machine translation models that support parallel translation, such as the NAT model, can deploy larger and deeper translation models under specific latency and budget in the industrial field, making machine translation models that support parallel translation, such as the NAT model, have higher application value.

[0053] Next, the system architecture of the embodiment of the present application will be described.

[0054] Figure 2 It is a schematic diagram of the implementation environment of a text translation method provided by an embodiment of the present application. Refer to Figure 2 The implementation environment includes a terminal 210 and a server 220, both of which are computer devices.

[0055] Terminal 210 can be used to provide text to be translated. After the user imports the text to be translated into terminal 210, terminal 210 generates a translation request carrying the text to be translated and sends the translation request to server 220. Server 220 responds to the translation request, translates the text to be translated, obtains the translated text, and returns the translated text to terminal 210.

[0056] Terminal 210 and server 220 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this.

[0057] Server 220 can be used to provide machine translation services. Using a teacher-student model training framework, taking the AT model as the teacher model and the NAT model as the student model as an example, server 220 can first use training sample pairs consisting of original text and labeled text to pre-train the initial NAT model (i.e., the initial translation model) to obtain the first NAT model (i.e., the first translation model). Then, the original text is translated using the positive teacher model to obtain a positive reference translation. Finally, the pre-trained first NAT model is fine-tuned using the training sample pairs consisting of the original text and the positive reference translation. The second NAT model (i.e., the second translation model) is obtained by directly exposing the original training sample pairs to the initial NAT model during the pre-training phase. This allows the initial NAT model to learn the low-frequency word distribution in the original training sample pairs. This preprocessing can transfer bilingual knowledge and data distribution from the original training sample pairs, especially for rare categories (such as low-frequency words). Because the low-frequency word distribution can be learned in advance during the pre-training phase, the low-frequency words will not be "unseen" during subsequent fine-tuning based on the positive teacher model, thus improving the robustness of the initial NAT model and consequently improving its translation performance. The NAT model is only one example of a machine translation model supporting parallel translation, and the AT model is only one example of a machine translation model supporting serial translation. The model training process refers to adjusting the model parameters (or simply "parameter tuning").

[0058] Optionally, server 220 maintains the second translation model only on its own server side, so that when server 220 receives a translation request from terminal 210, it can call the second translation model to translate the text to be translated carried in the translation request, obtain the translated text of the text to be translated, and return the translated text to terminal 210. By migrating the translation process to server 220, the local processing and computing resources of terminal 210 can be saved.

[0059] Optionally, after training the second translation model, the server 220 sends the second translation model to the terminal 210, so that the terminal 210 can call the second translation model locally to translate the text to be translated and obtain the translated text of the text to be translated, without having to send a translation request to the server 220 every time, which can reduce the communication overhead between the terminal 210 and the server 220.

[0060] Alternatively, the terminal 210 can train the second translation model locally and then call the second translation model locally to translate the text to be translated, thereby reducing the communication overhead between the terminal 210 and the server 220.

[0061] Server 220 may include at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, server 220 may undertake the primary computing task, and terminal 210 may undertake the secondary computing task; or, server 220 may undertake the secondary computing task, and terminal 210 may undertake the primary computing task; or, terminal 210 and server 220 may collaborate on computing using a distributed computing architecture. As disclosed in the text translation method of this application, multiple servers may form a blockchain, and the servers may be nodes on the blockchain.

[0062] Server 220 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0063] Terminal 210 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, in-vehicle terminal, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, e-book reader, etc., but is not limited to these.

[0064] Those skilled in the art will understand that terminal 210 can refer to one of a plurality of terminals, and the number of such terminals can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0065] The core concepts of the embodiments of this application will be described below: In an exemplary scenario, to balance decoding speed and translation performance, a teacher-student model training framework can be used. Typically, an AT model is introduced as the teacher model, and the complexity of the original training sample pairs is reduced by using sequence-level knowledge distillation (KD) technology, making it easier for the NAT model to learn deterministic translation knowledge, and ultimately achieving translation performance comparable to that of the AT model.

[0066] Figure 3 This is a schematic diagram illustrating the principle of a knowledge distillation method provided in this application embodiment. As shown in 300, the raw data refers to the original training sample pairs. Each training sample pair includes a pair of semantically corresponding training samples [src, tgt], where src refers to the original text (Source, source language) and tgt refers to the labeled text (Target, target language). The distilled data refers to the data obtained after knowledge distillation by the AT teacher model. In the knowledge distillation process, the original text src is input into the AT teacher model, and the positive reference translation tgt' is output. This is equivalent to using the AT teacher model to forge the actual labeled text tgt using the positive reference translation tgt', and using the original text src and the positive reference translation tgt' to construct new training data [src, tgt']. The initial NAT model is trained with the new training data [src, tgt'] to obtain the final NAT model. Here, the NAT model is only one example of a machine translation model that supports parallel translation, and the AT model is only one example of a machine translation model that supports serial translation.

[0067] Alternatively, assuming that the raw data is represented using Raw, then... To represent distillation data, the definitions of raw data and distillation data are as follows:

[0068] in, This represents the i-th original text. Let represent the i-th labeled text, where i is an integer greater than or equal to 1 and less than or equal to N, and N represents the total number of training sample pairs in the original data. This represents an AT-based translation model, trained on raw data, used to translate input text from the source language to the target language. This represents the set of original text from the source in the original data.

[0069] Knowledge distillation may lose some important information from the original data, leading to more errors in predicting low-frequency words in the trained NAT model. Therefore, a side effect of using knowledge distillation to train a NAT model is that it makes the distribution of the original data more unbalanced, resulting in serious word selection errors, especially for low-frequency words, where the error rate is significantly increased.

[0070] In view of this, the text translation method involved in the embodiments of this application can start from the data level, using preprocessing to directly expose the raw data to the NAT model, and also proposes a reverse knowledge distillation method to recall more alignment information of low-frequency target words. This makes this text translation method not limited to any NAT model framework, but can also complement most training methods, and can be extended to any machine translation model that supports parallel translation, with high scalability and usability. Therefore, by making full use of the existing raw data, low-frequency words can be activated as much as possible, mitigating the negative impact of knowledge distillation on the NAT model training process, ultimately leading to a significant increase in the translation performance of the NAT model, which will be described in detail below.

[0071] Figure 4 This is a flowchart illustrating a text translation method provided in an embodiment of this application. See also... Figure 4 This embodiment is applied to a computer device. The following description uses server 220 in the above-described implementation environment as an example. This embodiment includes the following steps: 401. Based on the original text and the labeled text corresponding to the semantics of the original text, the server adjusts the parameters of the initial translation model to obtain the first translation model, which supports parallel translation of the input text.

[0072] In this context, the original text refers to the text to be translated in the source language (Source), and the labeled text refers to the labeled translation in the target language (Target). The original text and the labeled text have the same semantics, but they belong to different languages. A pair of original text and labeled text can constitute a training sample pair.

[0073] The initial translation model is a machine translation model that supports parallel translation of input text, used to translate input text from a source language to a target language. For example, the initial translation model can be any NAT model or a variant of the NAT model, such as the NAT model, iterative NAT model, context-decoupled NAT model, mask prediction NAT model, or deletion-search-modification NAT model, etc. This application embodiment does not specifically limit the model architecture of the initial translation model.

[0074] In some embodiments, the server may obtain one or more training sample pairs from a training corpus. Each training sample pair includes a pair of semantically identical original text and labeled text. The training corpus may be stored locally on the server or in other distributed storage systems. This application embodiment does not specifically limit whether the training corpus is stored locally or in the cloud. Optionally, the labeled text may be a translation obtained by technicians through manual annotation of the original text to ensure the accuracy of the labeled text.

[0075] In some embodiments, the process of the server adjusting the parameters of the initial translation model, that is, the process of pre-training the initial translation model, may include: the server calling the initial translation model to translate the original text to obtain a first predicted text; determining a first loss value based on the labeled text and the first predicted text; and adjusting the parameters of the initial translation model based on the first loss value to obtain the first translation model.

[0076] Optionally, when obtaining the first predicted text, the server performs tokenization on the original text in each training sample pair to obtain the character sequence of the original text; performs embedding on the character sequence of the original text to obtain the embedding vector sequence of the original text; and encodes the position information of each character in the character sequence of the original text to obtain the position encoding vector of the original text, wherein the position encoding vector of the original text is used to represent the positional order information of each character in the original text.

[0077] Next, the server concatenates the embedding vector sequence of the original text and the positional encoding vector, then inputs them into the initial translation model. The initial translation model encodes the input vectors to obtain the semantic vector sequence (i.e., the latent vector sequence) of the original text. The semantic vector sequence of the original text is then subjected to exponential normalization (Softmax) to obtain the character count indication information of the original text. This character count indication information is used to indicate how many characters in the first predicted text correspond to each character in the original text. In other words, it can indicate how many characters in the first predicted text are needed to translate each character in the original text. Therefore, the character count indication information can be a character count sequence, the length of which is equal to the length of the character sequence of the input original text. Each element in the character count sequence is used to represent how many characters in the first predicted text are needed to translate the character at the corresponding position in the character sequence of the original text.

[0078] Next, the server transforms the character sequence of the original text based on the character count indication information to obtain the character sequence to be decoded. The transformation process mainly aims to match the number of each character in the character sequence to be decoded with the character count indication information. For example, if a certain character in the character sequence of the original text corresponds to the element 2 in the character count sequence, then this character will appear twice in the character sequence to be decoded.

[0079] Next, the character sequence to be decoded is embedded to obtain an embedding vector sequence; and the position information of each character in the character sequence is encoded to obtain a position encoding vector. Then, the embedding vector sequence and the position encoding vector are concatenated, and the concatenated vector is decoded to obtain the semantic vector sequence of the first predicted text. The semantic vector sequence of the first predicted text is then subjected to exponential normalization, and the character sequence of the first predicted text is output. The individual characters in this character sequence constitute the first predicted text.

[0080] It should be noted that this embodiment only provides a brief introduction to the translation process of the original text by the NAT model. In the next embodiment, a basic NAT model will be used as an example to describe the translation process of the NAT model in detail.

[0081] Optionally, when obtaining the first loss value, the server can obtain the cross-entropy loss between the labeled text and the first predicted text for each training sample pair as the first loss value.

[0082] In some embodiments, the server can directly determine whether the first loss value is less than or equal to a first loss threshold. If the first loss value is greater than the first loss threshold, the initial translation model is trained iteratively. If the first loss value is less than or equal to the first loss threshold, training is stopped, and a first translation model is obtained. The first loss threshold can be any value greater than or equal to 0 and less than or equal to 1. For example, the first loss threshold is 0.2. This application embodiment does not specifically limit the value of the first loss threshold.

[0083] In some embodiments, in addition to obtaining the first loss value, the server can also determine the BLEU score for each training sample pair based on the labeled text and the first predicted text. Based on the obtained first loss value and BLEU score, it can be determined whether each of the first loss value and BLEU score meets the first stopping condition. If neither the first loss value nor the BLEU score meets the first stopping condition, the parameters of the initial translation model are iteratively adjusted based on the backpropagation algorithm until either the first loss value or the BLEU score meets the first stopping condition, at which point the iteration stops, and the first translation model is obtained.

[0084] Optionally, the first stopping condition is: the BLEU value is greater than the first BLEU threshold; or, the first loss value is less than or equal to the first loss threshold. The first BLEU threshold can be any value greater than 0. For example, the first BLEU threshold can be 90% of the best BLEU of all training sample pairs (hereinafter referred to as the original data) used in the training. This application embodiment does not specifically limit the value of the first BLEU threshold.

[0085] In some embodiments, when performing the above judgment steps, it can first be determined whether the BLEU value is greater than a first BLEU threshold. If the BLEU value is already greater than the first BLEU threshold, then it is not necessary to determine whether the first loss value is less than or equal to the first loss threshold, and training can be stopped early. This is because the process from the initial translation model to the first translation model is only a rough pre-training process, and training can be stopped early when the BLEU value reaches a certain index, in order to save the computational resources consumed during training. Furthermore, if the BLEU value is less than or equal to the first BLEU threshold, then it is still necessary to determine whether the first loss value is less than or equal to the first loss threshold. If the first loss value is less than or equal to the first loss threshold, then training is stopped; if the first loss value is greater than the first loss threshold, then the next iteration of the training process continues.

[0086] In some embodiments, the first stopping condition may also be: the BLEU value is greater than a first BLEU threshold; or the first loss value is less than or equal to a first loss threshold; or the number of iterations is greater than a first number threshold. That is, besides triggering early stopping of training when the BLEU value is greater than the first BLEU threshold, another way to trigger early stopping of training is provided, namely, triggering early stopping of training when the number of iterations is greater than a first number threshold. This application embodiment does not specifically limit the content of the first stopping condition. The first number threshold can be any integer greater than or equal to 1, for example, the first number threshold is 50000. This application embodiment does not specifically limit the value of the first number threshold.

[0087] In some embodiments, when performing the above judgment steps, it can first be determined whether the number of iterations is greater than the first threshold. If the number of iterations is already greater than the first threshold, then it is not necessary to determine whether the subsequent BLEU value is greater than the first BLEU threshold or whether the first loss value is less than or equal to the first loss threshold, and training can be stopped early. This is because the process from the initial translation model to the first translation model is only a rough pre-training process. Training can be stopped early when the number of iterations exceeds a certain index to save computational resources consumed during training. This is especially true for cases where some loss functions are difficult to converge, which can greatly shorten the training time. In addition, if the number of iterations is less than or equal to the first threshold, it is still necessary to determine whether the BLEU value is greater than the first BLEU threshold. If the BLEU value is already greater than the first BLEU threshold, then it is not necessary to determine whether the first loss value is less than or equal to the first loss threshold, and training can still be stopped early. If the BLEU value is less than or equal to the first BLEU threshold, then it is still necessary to determine whether the first loss value is less than or equal to the first loss threshold. If the first loss value is less than or equal to the first loss threshold, training is stopped. If the first loss value is greater than the first loss threshold, the next iteration training process is continued.

[0088] 402. The server calls the positive teacher model to translate the original text and obtain a positive reference translation. The positive reference translation and the annotated text belong to the same language. The positive teacher model supports serial translation of the input text.

[0089] The positive teacher model is a machine translation model that supports serial translation of input text, used to translate input text from a source language to a target language. For example, the positive teacher model can be any AT model or a variant of the AT model, such as the Transformer model or a variant of the Transformer model. This application does not specifically limit the model architecture of the positive teacher model.

[0090] In some embodiments, when obtaining a positive reference translation, the server performs word segmentation on the original text in each training sample pair to obtain the character sequence of the original text; performs embedding processing on the character sequence of the original text to obtain the embedding vector sequence of the original text; and encodes the position information of each character in the character sequence of the original text to obtain the position encoding vector of the original text, wherein the position encoding vector of the original text is used to represent the positional order information of each character in the original text.

[0091] Next, the server concatenates the embedding vector sequence of the original text and the positional encoding vector and inputs them into the positive teacher model. The positive teacher model encodes the input vectors to obtain the semantic vector sequence (i.e., the latent vector sequence) of the original text.

[0092] Next, the server decodes the semantic vector sequence of the original text character by character based on the shifted right mechanism to obtain the final latent vector sequence. Then, it performs exponential normalization on the final latent vector sequence and outputs the character sequence of the positive reference translation. Each character in the character sequence of the positive reference translation can form the positive reference translation.

[0093] It should be noted that this embodiment only briefly introduces the translation process of the AT model for the original text. In the next embodiment, the translation process of the AT model will be described in detail using the Transformer model as an example.

[0094] In the above process, since the annotated text can be regarded as the correct translation of the original text, while the positive reference translation is equivalent to the predicted translation of the original text using the positive teacher model, the positive reference translation is equivalent to the annotated text "forged" by the positive teacher model. Therefore, the original text and the positive reference translation can constitute the distilled data obtained by knowledge distillation by the positive teacher model. This distilled data can be put into the retraining process of the first translation model that has been pre-trained.

[0095] 403. Based on the original text and the positive reference translation, the server adjusts the parameters of the first translation model to obtain a second translation model, which is used to translate the input text into semantically corresponding output text.

[0096] In some embodiments, the process of the server adjusting the parameters of the first translation model, that is, the process of fine-tuning the first translation model, may include: the server calling the first translation model to translate the original text to obtain a second predicted text; determining a second loss value based on the positive reference translation and the second predicted text; and adjusting the parameters of the first translation model based on the second loss value to obtain the second translation model.

[0097] In the above process, the server obtains the second predicted text in a similar way to obtain the first predicted text in step 401 above, and will not be described in detail here.

[0098] Optionally, when obtaining the second loss value, the server can obtain the cross-entropy loss between the positive reference translation and the second predicted text for each training sample pair as the second loss value. If the second stopping condition is not met, the parameters of the first translation model are iteratively adjusted based on the backpropagation algorithm until the second stopping condition is met, at which point the iteration stops and the second translation model is obtained.

[0099] Optionally, the second stopping condition includes: the BLEU value of the first translation model is greater than the second BLEU threshold, or the second loss value is less than or equal to the second loss threshold, or the number of iterations is greater than the second number threshold, etc. This embodiment does not specifically limit the content of the second stopping condition. The second BLEU threshold can be any value greater than 0, for example, the second BLEU threshold can be 95% of the best BLEU of all training sample pairs (hereinafter referred to as the original data) used in training; the second loss threshold can be any value greater than or equal to 0 and less than or equal to 1, for example, the second loss threshold is 0.1; the second number threshold can be any integer greater than or equal to 1, for example, the second number threshold is 100000. This embodiment does not specifically limit the values ​​of the second BLEU threshold, the second loss threshold, and the second number threshold.

[0100] Figure 5 This is a flowchart illustrating the principle of a text translation method provided in this application, as shown in Figure 500. The original data refers to the training sample pair [src, tgt] consisting of the original text src and the labeled text tgt. The distilled data refers to the synthetic sample pair [src, tgt'] consisting of the original text src and the positive reference translation tgt'. The positive reference translation tgt' is the translation obtained by the positive teacher model (AT model) through serial translation of the original text src. During preprocessing, the original data [src, tgt] is used to pre-train the initial translation model (NAT model) to obtain the first translation model. Then, the distilled data [src, tgt'] is used to further train the first translation model (also known as fine-tuning). Since the original data [src, tgt] maintains the original multimodal distribution of the training sample pairs, especially preserving more detailed information for low-frequency words, although the NAT model has difficulty learning high-modal data, preprocessing can obtain general knowledge from the original data [src, tgt], which will help the NAT model learn further tasks better and faster. Optionally, when the initial translation model achieves 90% of the best performance on the original data in terms of BLEU score, pre-training can be stopped early to obtain the first translation model. Furthermore, to preserve the advantages of low-modality performance, the first translation model is further trained on distilled data [src, tgt']. Since the NAT model easily learns deterministic knowledge, and the distilled data [src, tgt'] contains more deterministic knowledge, the first translation model can be fine-tuned. This training method provides a good balance between the original data (which has high modality and complete characteristics) and the distilled data (which has low modality and incomplete characteristics).

[0101] In the above process, pre-training can directly expose the original data to the initial translation model (i.e., the NAT model). Since pre-training can transfer knowledge and data distribution, especially for rare categories (such as low-frequency words), this training method that combines pre-training based on original data with knowledge distillation based on the positive teacher model can transfer the distribution of information lost in the distilled data (such as the distribution of low-frequency words) to the NAT model. This allows the NAT model to learn the distribution of low-frequency words in the original data, which can greatly improve the robustness and translation performance of the trained second translation model.

[0102] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0103] The method provided in this application pre-trains an initial translation model using original text and labeled text. Based on the pre-trained first translation model, it further trains the model using the original text and positive reference translations obtained after knowledge distillation by the positive teacher model. This results in a second translation model that can learn not only the original data distribution in the original and labeled text, such as the high-modulus distribution and low-frequency word distribution of the target language, but also the deterministic knowledge obtained after knowledge distillation by the positive teacher model. This enables accurate prediction of both high-frequency and low-frequency words in parallel decoding, greatly improving the translation accuracy of machine translation models that support parallel translation.

[0104] Figure 6 This is a flowchart illustrating a text translation method provided in an embodiment of this application. See also... Figure 6 This embodiment is applied to a computer device. The following description uses server 220 in the above-described implementation environment as an example. This embodiment includes the following steps: 601. The server obtains the original text and the labeled text, and the labeled text corresponds semantically to the original text.

[0105] In some embodiments, the server may obtain one or more training sample pairs from a training corpus. Each training sample pair includes a pair of semantically identical original text and labeled text. The training corpus may be stored locally on the server or in other distributed storage systems. This application embodiment does not specifically limit whether the training corpus is stored locally or in the cloud. Optionally, the labeled text may be a translation obtained by technicians through manual annotation of the original text to ensure the accuracy of the labeled text.

[0106] 602. The server calls the initial translation model to translate the original text and obtain the first predicted text. The initial translation model supports parallel translation of the input text.

[0107] In some embodiments, the server performs word segmentation on the original text in each training sample pair to obtain the character sequence of the original text; performs embedding processing on the character sequence of the original text to obtain the embedding vector sequence of the original text; and encodes the position information of each character in the character sequence of the original text to obtain the position encoding vector of the original text, wherein the position encoding vector of the original text is used to represent the positional order information of each character in the original text.

[0108] Next, the server concatenates the embedding vector sequence of the original text and the positional encoding vector, then inputs it into the initial translation model. The initial translation model encodes the input vectors to obtain the semantic vector sequence (i.e., the latent vector sequence) of the original text. The semantic vector sequence of the original text is then subjected to exponential normalization to obtain the character count indication information of the original text. This character count indication information is used to indicate how many characters in the first predicted text each character in the original text corresponds to. In other words, it can indicate how many characters in the first predicted text are needed to translate each character in the original text. Therefore, the character count indication information can be a character count sequence, the length of which is equal to the length of the character sequence of the input original text. Each element in the character count sequence is used to represent how many characters in the first predicted text are needed to translate the character at the corresponding position in the character sequence of the original text.

[0109] Next, the server transforms the character sequence of the original text based on the character count indication information to obtain the character sequence to be decoded. The transformation process mainly aims to match the number of each character in the character sequence to be decoded with the character count indication information. For example, if a certain character in the character sequence of the original text corresponds to the element 2 in the character count sequence, then this character will appear twice in the character sequence to be decoded.

[0110] Next, the character sequence to be decoded is embedded to obtain an embedding vector sequence; and the position information of each character in the character sequence is encoded to obtain a position encoding vector. Then, the embedding vector sequence and the position encoding vector are concatenated, and the concatenated vector is decoded to obtain the semantic vector sequence of the first predicted text. The semantic vector sequence of the first predicted text is then subjected to exponential normalization, and the character sequence of the first predicted text is output. The individual characters in this character sequence constitute the first predicted text.

[0111] Figure 7 This is a schematic diagram of the architecture of an initial translation model provided in an embodiment of this application. Please refer to it. Figure 7 The following explanation uses the NAT model as an example of an initial translation model. The NAT model includes an embedding module 701, an encoding module 702, a decoding module 703, and an output module 704. In acquiring the first predicted text, the original text is first segmented to obtain a character sequence. This character sequence is then input into the embedding module 701, which performs embedding processing to obtain an embedding vector sequence. Furthermore, the positional information of each character in the character sequence is encoded to obtain a positional encoding vector, which represents the order of positions of each character in the original text. The embedding vector sequence and the positional encoding vector are then concatenated and input into the encoding module 702.

[0112] The encoding module 702 can be composed of multiple cascaded encoders (e.g., N=6 cascaded encoders). Each encoder includes a multi-head attention layer and an MLP (Multi-Layer Perceptron) layer. The multi-head attention layer is used to comprehensively extract the relationships between characters in the original text from multiple representation subspaces. The MLP layer is used to weight the feature vectors output by the multi-head attention layer. The hidden layers of the MLP are also fully connected. After the multi-head attention layer and the MLP layer, a residual structure is set, which means that the input and output of the current layer are residually concatenated (i.e., spliced) and normalized before being input into the next layer. The input vectors are encoded by these multiple encoders, and the last encoder outputs the semantic vector sequence (i.e., the hidden vector sequence) of the original text.

[0113] Unlike the AT model, which translates word-by-word and stops translation with a terminator, the NAT model performs parallel translation and therefore requires a character count indicator. This character count indicator shows how many characters in the first predicted text each character in the original text corresponds to, meaning it indicates how many characters in the first predicted text are needed to translate each character in the original text. This character count indicator is extracted by setting an exponential normalization layer after the last encoder in the encoding module 702. In other words, the character count indicator is obtained by exponentially normalizing the semantic vector sequence of the original text through this exponential normalization layer.

[0114] Optionally, the character count indication can be a character count sequence, the length of which is equal to the length of the character sequence of the input original text. Each element in the character count sequence represents the number of characters needed for translation in the first predicted text for the corresponding character in the original text. For example, in the process of translating English into German, the original text is "We totally accept it.", its character sequence is [We, totally, accept, it, .], and the character count sequence is [1, 1, 2, 0, 1]. This means that the English word "We" needs to be represented by one German word, the English word "totally" needs to be represented by one German word, the English word "accept" needs to be represented by two German words, the English word "it" does not need to be represented by a separate German word after translation, and the punctuation mark "." needs to be represented by one punctuation mark. Here, N represents the number of cascaded encoders.

[0115] Next, based on the character count indication information output by the encoding module 702, the character sequence of the original text is transformed to obtain the character sequence to be decoded. The transformation process mainly aims to match the count of each character in the character sequence to be decoded with the character count indication information. For example, based on the character count sequence [1, 1, 2, 0, 1], the original text "We totally accept it ." is transformed to obtain the text to be decoded "We totally accept accept.", that is, the character sequence [We, totally, accept, it, .] of the original text is transformed to obtain the character sequence [We, totally, accept, accept, .].

[0116] Then, the character sequence to be decoded is input into the embedding module 701. The embedding module 701 performs embedding processing on the character sequence to be decoded to obtain the embedding vector sequence to be decoded. In addition, the position information of each character in the character sequence to be decoded is encoded to obtain the position encoding vector to be decoded. The embedding vector sequence to be decoded and the position encoding vector to be decoded are concatenated and then input into the decoding module 703.

[0117] The decoding module 703 can be composed of multiple decoders cascaded together (for example, N=6 decoders cascaded together). Each decoder includes a multi-head attention layer, a multi-head positional attention layer, a multi-head interactive attention layer, and an MLP layer. The multi-head attention layer in the decoder is similar to the multi-head attention layer in the encoder, and will not be elaborated here. The multi-head positional attention layer is used to introduce positional encoding vectors again based on the output of the multi-head attention layer to emphasize the positional order of each character in the character sequence to be decoded. The multi-head interactive attention layer takes the output of the multi-head positional attention layer of this decoder as input, as well as the output of the MLP layer of the encoder with the corresponding sequence number (referring to the result after residual connection and normalization) as input. This design is used to focus on the encoded information of the encoder. In other words, the decoder performs text translation by looking at the output of the encoder and its self-attention to its own output. The MLP layer of the decoder is similar to the MLP layer of the encoder, and will not be elaborated here. Similarly, residual structures are also set after the multi-head attention layer, multi-head positional attention layer, multi-head interactive attention layer, and MLP layer of the decoder. This means that the input and output of the current layer are residually concatenated (i.e., spliced) and normalized before being input into the next layer. Here, N represents the number of cascaded decoders. It can be seen that the number of cascaded encoders and the number of cascaded decoders need to be consistent. These multiple decoders can decode the spliced ​​vector, with the last decoder outputting the semantic vector sequence of the first predicted text.

[0118] The output module 704 includes an exponential normalization layer. This layer maps the semantic vector sequence of the first predicted text output by the last decoder in the decoding module 703 to the character sequence of the final first predicted text. Each character in this character sequence constitutes the first predicted text. In other words, the exponential normalization layer maps the semantic vector of each character in the semantic vector sequence of the first predicted text to multiple probabilities. Each probability represents the likelihood that the semantic vector of the current character will be mapped to each character in the target language (e.g., a dictionary). Optionally, the character with the highest probability is selected as the output at the corresponding time in the character sequence, or a character is randomly sampled from the top K (K≥1) characters with the highest probability as the output at the corresponding time in the character sequence.

[0119] 603. The server determines the first loss value based on the labeled text and the first predicted text.

[0120] In some embodiments, when obtaining the first loss value, the server can obtain the cross-entropy loss between the labeled text and the first predicted text for each training sample pair as the first loss value. Optionally, it can also be combined with a contrastive learning method to determine the contrastive learning loss as the first loss value. This application embodiment does not specifically limit the type of loss function.

[0121] 604. Based on the first loss value, the server adjusts the parameters of the initial translation model to obtain the first translation model.

[0122] In some embodiments, the server can directly determine whether the first loss value is less than or equal to a first loss threshold. If the first loss value is greater than the first loss threshold, the initial translation model is trained iteratively. If the first loss value is less than or equal to the first loss threshold, training is stopped, and a first translation model is obtained. The first loss threshold can be any value greater than or equal to 0 and less than or equal to 1. For example, the first loss threshold is 0.2. This application embodiment does not specifically limit the value of the first loss threshold.

[0123] In some embodiments, in addition to obtaining the first loss value, the server may also determine the bilingual evaluation study BLEU value for each training sample pair based on the labeled text and the first predicted text.

[0124] Based on the obtained first loss value and BLEU value, it can be determined whether each of the first loss value and BLEU value meets the first stopping condition. If neither the first loss value nor the BLEU value meets the first stopping condition, the parameters of the initial translation model are iteratively adjusted based on the backpropagation algorithm until either the first loss value or the BLEU value meets the first stopping condition, at which point the iteration stops and the first translation model is obtained.

[0125] Optionally, the first stopping condition is: the BLEU value is greater than the first BLEU threshold; or, the first loss value is less than or equal to the first loss threshold. The first BLEU threshold can be any value greater than 0. For example, the first BLEU threshold can be 90% of the best BLEU of all training sample pairs (hereinafter referred to as the original data) used in the training. This application embodiment does not specifically limit the value of the first BLEU threshold.

[0126] In some embodiments, when performing the above judgment steps, it can first be determined whether the BLEU value is greater than a first BLEU threshold. If the BLEU value is already greater than the first BLEU threshold, then it is not necessary to determine whether the first loss value is less than or equal to the first loss threshold, and training can be stopped early. This is because the process from the initial translation model to the first translation model is only a rough pre-training process, and training can be stopped early when the BLEU value reaches a certain index, in order to save the computational resources consumed during training. Furthermore, if the BLEU value is less than or equal to the first BLEU threshold, then it is still necessary to determine whether the first loss value is less than or equal to the first loss threshold. If the first loss value is less than or equal to the first loss threshold, then training is stopped; if the first loss value is greater than the first loss threshold, then the next iteration of the training process continues.

[0127] In some embodiments, the first stopping condition may also be: the BLEU value is greater than a first BLEU threshold; or the first loss value is less than or equal to a first loss threshold; or the number of iterations is greater than a first number threshold. That is, besides triggering early stopping of training when the BLEU value is greater than the first BLEU threshold, another way to trigger early stopping of training is provided, namely, triggering early stopping of training when the number of iterations is greater than a first number threshold. This application embodiment does not specifically limit the content of the first stopping condition. The first number threshold can be any integer greater than or equal to 1, for example, the first number threshold is 50000. This application embodiment does not specifically limit the value of the first number threshold.

[0128] In some embodiments, when performing the above judgment steps, it can first be determined whether the number of iterations is greater than the first threshold. If the number of iterations is already greater than the first threshold, then it is not necessary to determine whether the subsequent BLEU value is greater than the first BLEU threshold or whether the first loss value is less than or equal to the first loss threshold, and training can be stopped early. This is because the process from the initial translation model to the first translation model is only a rough pre-training process. Training can be stopped early when the number of iterations exceeds a certain index to save computational resources consumed during training. This is especially true for cases where some loss functions are difficult to converge, which can greatly shorten the training time. In addition, if the number of iterations is less than or equal to the first threshold, it is still necessary to determine whether the BLEU value is greater than the first BLEU threshold. If the BLEU value is already greater than the first BLEU threshold, then it is not necessary to determine whether the first loss value is less than or equal to the first loss threshold, and training can still be stopped early. If the BLEU value is less than or equal to the first BLEU threshold, then it is still necessary to determine whether the first loss value is less than or equal to the first loss threshold. If the first loss value is less than or equal to the first loss threshold, training is stopped. If the first loss value is greater than the first loss threshold, the next iteration training process is continued.

[0129] In steps 602-604 above, a possible implementation of the server adjusting the parameters of the initial translation model based on the original text and the labeled text corresponding to the semantics of the original text to obtain the first translation model is shown. This is equivalent to pre-training the initial translation model using the original text and the labeled text. Through pre-training, the original data can be directly exposed to the initial translation model (i.e., the NAT model). Since pre-training can transfer knowledge and data distribution, especially for rare categories (such as low-frequency words), the NAT model can learn the distribution of low-frequency words in the original data.

[0130] 605. The server calls the positive teacher model to translate the original text and obtain a positive reference translation. The positive reference translation and the annotated text belong to the same language. The positive teacher model supports serial translation of the input text.

[0131] In some embodiments, when obtaining a positive reference translation, the server performs word segmentation on the original text in each training sample pair to obtain the character sequence of the original text; performs embedding processing on the character sequence of the original text to obtain the embedding vector sequence of the original text; and encodes the position information of each character in the character sequence of the original text to obtain the position encoding vector of the original text, wherein the position encoding vector of the original text is used to represent the positional order information of each character in the original text.

[0132] Next, the server concatenates the embedding vector sequence of the original text and the positional encoding vector and inputs them into the positive teacher model. The positive teacher model encodes the input vectors to obtain the semantic vector sequence (i.e., the latent vector sequence) of the original text.

[0133] Next, the server decodes the semantic vector sequence of the original text character by character based on the shifted right mechanism to obtain the final latent vector sequence. Then, it performs exponential normalization on the final latent vector sequence and outputs the character sequence of the positive reference translation. Each character in the character sequence of the positive reference translation can form the positive reference translation.

[0134] Figure 8 This is a schematic diagram of the architecture of a positive teacher model provided in an embodiment of this application. Please refer to it. Figure 8 The Transformer model, using the positive teacher model as an example, will be used for illustration. The Transformer translation model includes an embedding module 801, an encoding module 802, a decoding module 803, and an output module 804. It adopts an encoder-decoder framework based on a SAN network, which is a mainstream sequence-to-sequence generation model structure.

[0135] In the process of obtaining the positive reference translation, the original text is first segmented to obtain a character sequence. This character sequence is then input into the embedding module 801, which performs embedding processing to obtain an embedding vector sequence. Furthermore, the positional information of each character in the character sequence is encoded to obtain a positional encoding vector, which represents the order of each character within the original text. The embedding vector sequence and the positional encoding vector are then concatenated and input into the encoding module 802.

[0136] The encoding module 802 can be composed of multiple cascaded encoders (e.g., N=6 cascaded encoders). Each encoder includes a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer is used to comprehensively extract the relationships between characters in the original text from multiple representation subspaces. The feedforward neural network layer is used to fully connect the feature vectors output by the multi-head attention layer. After the multi-head attention layer and the feedforward neural network layer, a residual structure is set, which means that the input and output of the current layer are residually concatenated (i.e., spliced) and normalized before being input into the next layer. Here, N represents the number of cascaded encoders. The input vector is encoded by these multiple encoders, and the last encoder outputs the semantic vector sequence (i.e., the latent vector sequence) of the original text.

[0137] The decoding module 803 can be composed of multiple decoders cascaded together (e.g., N=6 decoders cascaded together). Each decoder internally includes a masked multi-head attention layer, a fusion multi-head attention layer, and a feedforward neural network layer. The masked multi-head attention layer is similar to the multi-head attention layer, but it only focuses on the translation results before the current time step, so it needs to mask the translation results after the current time step. The fusion multi-head attention layer is also similar to the multi-head attention layer, but in addition to taking the output of the masked multi-head attention layer of this decoder as input, the fusion multi-head attention layer also takes the feedforward of the encoder with the corresponding sequence number as input. The output of the neural network layer (referring to the result after residual connections and normalization) serves as input. This design focuses on the encoded information of the encoder. In other words, the decoder predicts the next character in the forward reference translation by examining the encoder's output and its self-attention to its own output. The feedforward neural network layer of the decoder is similar to that of the encoder's feedforward neural network layer, and will not be elaborated here. Similarly, residual structures are also set after the masked multi-head attention layer, fused multi-head attention layer, and feedforward neural network layer of the decoder. That is, the input and output of the current layer are residually connected (i.e., concatenated) and normalized before being input into the next layer. Here, N represents the number of cascaded decoders, that is, the number of cascaded encoders must be consistent with the number of cascaded decoders. Through these multiple decoders, the semantic vector sequence of the original text can be decoded, and the final latent vector sequence is output by the last decoder.

[0138] The output module 804 includes a linear layer and an exponential normalization (Softmax) layer. It maps each latent vector in the final latent vector sequence output by the decoding module 803 to a character in the final positive reference translation, thus forming the positive reference translation. The linear layer is a fully connected layer with a dimension equal to the total number of characters in the target language corpus. The exponential normalization layer maps the semantic vector of each character in the final latent vector sequence to multiple probabilities. Each probability represents the likelihood that the semantic vector of the current character will map to a character in the target language (e.g., a dictionary). Optionally, the character with the highest probability is selected as the output at the corresponding time step in the character sequence; alternatively, a character is randomly sampled from the top K (K≥1) characters with the highest probabilities as the output at the corresponding time step in the character sequence.

[0139] In addition, the input of the decoding module 803 involves a Shifted Right mechanism. Since the learning objective during the model training phase is to predict what the character (Token) at the next position is, Masking (masking) means are used here to cover the content on the right side at the current moment, ensuring that the learning process predicts character by character from left to right. The Shifted Right mechanism ensures that the masked multi-head attention layer can only focus on the translation results before the current moment. For example, assume the output sequence is "I love China.", which includes 4 characters in total. When predicting "love", it is necessary to cover "China.", that is, only predict "love" based on "I". Similarly, when predicting "China", it is necessary to cover ".", that is, only predict "China" based on "I love".

[0140] In the above process, since the annotated text can be regarded as the correct translation of the original text after translation, and the forward reference translation is equivalent to the predicted translation of the original text by the forward teacher model. The forward reference translation is equivalent to the annotated text "forged" by the forward teacher model. Therefore, the original text and the forward reference translation can form the distilled data obtained by knowledge distillation through the forward teacher model, and this distilled data can be input into the retraining process of the pre-trained first translation model.

[0141] 606. The server invokes the first translation model to translate the original text and obtains a second predicted text.

[0142] The above step 606 is similar to the above step 602 and will not be elaborated here.

[0143] 607. The server determines a second loss value based on the forward reference translation and the second predicted text.

[0144] In some embodiments, when obtaining the second loss value, the server can obtain the cross-entropy loss between the forward reference translation and the second predicted text for each training sample pair as the second loss value. Optionally, a contrastive learning loss can also be combined and determined as the second loss value. The embodiments of the present application do not specifically limit the type of the loss function.

[0145] 608. The server adjusts the parameters of the first translation model based on the second loss value to obtain the second translation model, and the second translation model is used to translate the input text into a semantically corresponding output text.

[0146] In some embodiments, if the second stop condition is not satisfied, the parameters of the first translation model are iteratively adjusted based on the backpropagation algorithm until the second stop condition is satisfied, and then the iteration is stopped to obtain the second translation model.

[0147] Optionally, the second stopping condition includes: the BLEU value of the first translation model is greater than the second BLEU threshold, or the second loss value is less than or equal to the second loss threshold, or the number of iterations is greater than the second number threshold, etc. This embodiment does not specifically limit the content of the second stopping condition. The second BLEU threshold can be any value greater than 0, for example, the second BLEU threshold can be 95% of the best BLEU of all training sample pairs (hereinafter referred to as the original data) used in training; the second loss threshold can be any value greater than or equal to 0 and less than or equal to 1, for example, the second loss threshold is 0.1; the second number threshold can be any integer greater than or equal to 1, for example, the second number threshold is 100000. This embodiment does not specifically limit the values ​​of the second BLEU threshold, the second loss threshold, and the second number threshold.

[0148] In steps 606-608 above, a possible implementation of the server adjusting the parameters of the first translation model based on the original text and the positive reference translation is shown to obtain the second translation model. That is, based on the pre-training of the initial translation model on the original data, the original data is distilled by combining the positive teacher model to obtain distilled data, and the distilled data is used to continue training on the first translation model to obtain the final second translation model, which can greatly improve the robustness and translation performance of the trained second translation model.

[0149] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0150] The method provided in this application pre-trains an initial translation model using original text and labeled text. Based on the pre-trained first translation model, it further trains the model using the original text and positive reference translations obtained after knowledge distillation by the positive teacher model. This results in a second translation model that can learn not only the original data distribution in the original and labeled text, such as the high-modulus distribution and low-frequency word distribution of the target language, but also the deterministic knowledge obtained after knowledge distillation by the positive teacher model. This enables accurate prediction of both high-frequency and low-frequency words in parallel decoding, greatly improving the translation accuracy of machine translation models that support parallel translation.

[0151] In the above embodiments, a training method based on pre-training on raw data to reconstruct the distribution of low-frequency words, combined with traditional knowledge distillation, is provided. However, since the aforementioned forward teacher model only provides distilled data from the source to the target, this distilled data can be regarded as forward distilled data from the source to the target, which can "fake" the labeled text. Correspondingly, an additional reverse teacher model can also be used to provide reverse distilled data from the target to the source. In other words, the reverse teacher model is used to translate the labeled text from the target language to the source language, and therefore can "fake" the original text. Forward distillation data can make low-frequency words more definitively aligned from the source to the target, but due to the loss of details in the original data, low-frequency words cannot be well aligned from the target to the source. In view of this, a reverse knowledge distillation method based on the reverse teacher model is proposed to recall more low-frequency word alignment information from the target end to the source end. The extracted reverse distillation data is combined with traditional forward distillation data, which can simultaneously maintain the advantages of both deterministic knowledge and low-frequency information. This allows for the maximum utilization of both original and distilled data, thereby further improving the translation accuracy of machine translation models that support parallel translation, such as the NAT model.

[0152] Figure 9 This is a flowchart illustrating a text translation method provided in an embodiment of this application. See also... Figure 9 This embodiment is applied to a computer device. The following description uses server 220 in the above-described implementation environment as an example. This embodiment includes the following steps: 901. Based on the original text and the labeled text corresponding to the semantics of the original text, the server adjusts the parameters of the initial translation model to obtain the first translation model, which supports parallel translation of the input text.

[0153] Step 901 above is similar to step 401 above, and will not be described in detail here.

[0154] 902. The server calls the positive teacher model to translate the original text and obtain a positive reference translation. The positive reference translation and the annotated text belong to the same language. The positive teacher model supports serial translation of the input text.

[0155] Step 902 is similar to step 402 above, and will not be described in detail here.

[0156] In the above process, since the annotated text can be regarded as the correct translation of the original text, while the positive reference translation is equivalent to the predicted translation of the original text using the positive teacher model, the positive reference translation is equivalent to the annotated text "forged" by the positive teacher model. Therefore, the original text and the positive reference translation can constitute the positive distillation data obtained by knowledge distillation by the positive teacher model. This positive distillation data can be put into the retraining process of the pre-trained first translation model.

[0157] 903. The server calls the reverse teacher model to translate the annotated text and obtains a reverse reference translation. The reverse reference translation is in the same language as the original text. The reverse teacher model supports serial translation of the input text.

[0158] The reverse teacher model is a machine translation model that supports serial translation of input text, used to translate input text from a target language to a source language. For example, the reverse teacher model can be any AT model or a variant of the AT model, such as the Transformer model or a variant of the Transformer model. This application does not specifically limit the model architecture of the reverse teacher model.

[0159] It should be noted that both the reverse teacher model and the forward teacher model are machine translation models that support sequential translation (such as the AT model). However, the difference lies in the fact that the forward teacher model is a translation model from the source language to the target language, extracting forward distilled data (i.e., forward reference translation), while the reverse teacher model is a translation model from the target language to the source language, extracting back distilled data (i.e., reverse reference translation).

[0160] In some embodiments, when obtaining the reverse reference translation, the server performs word segmentation on the labeled text in each training sample pair to obtain the character sequence of the labeled text; performs embedding processing on the character sequence of the labeled text to obtain the embedding vector sequence of the labeled text; and encodes the position information of each character in the character sequence of the labeled text to obtain the position encoding vector of the labeled text, wherein the position encoding vector of the labeled text is used to represent the positional order information of each character in the labeled text.

[0161] Next, the server concatenates the embedding vector sequence of the labeled text and the position encoding vector and inputs them into the reverse teacher model. The reverse teacher model encodes the input vector to obtain the semantic vector sequence (i.e., the latent vector sequence) of the labeled text.

[0162] Next, the server decodes the semantic vector sequence of the annotated text character by character based on the shifted right mechanism to obtain the final latent vector sequence. Then, it performs exponential normalization on the final latent vector sequence and outputs the character sequence of the reverse reference translation. Each character in the character sequence of the reverse reference translation can form the reverse reference translation.

[0163] In the above process, since the original text can be regarded as the correct translation after the annotation text is translated, while the reverse reference translation is equivalent to the predicted translation after the annotation text is translated using the reverse teacher model, the reverse reference translation is equivalent to the original text "forged" by the reverse teacher model. Therefore, the annotation text and the reverse reference translation can constitute the backdistilled data obtained by knowledge distillation through the reverse teacher model. This backdistilled data can be fed into the retraining process of the first translation model that has been pre-trained, together with the forward distilled data.

[0164] 904. The server calls the first translation model to translate the original text and obtain the second predicted text.

[0165] Step 904 above is similar to step 606 above, and will not be described in detail here.

[0166] 905. The server determines the second loss value based on the positive reference translation and the second predicted text.

[0167] Step 905 is similar to step 607 above, and will not be described in detail here.

[0168] 906. The server calls the first translation model to translate the reverse reference translation and obtains the third predicted text.

[0169] In some embodiments, the server performs word segmentation on each reverse reference translation to obtain a character sequence of the reverse reference translation; performs embedding processing on the character sequence of the reverse reference translation to obtain an embedding vector sequence of the reverse reference translation; and encodes the position information of each character in the character sequence of the reverse reference translation to obtain a position encoding vector of the reverse reference translation, wherein the position encoding vector of the reverse reference translation is used to represent the positional order information of each character in the reverse reference translation.

[0170] Next, the server concatenates the embedding vector sequence of the reverse reference translation and the positional encoding vector, then inputs it into the first translation model. The first translation model encodes the input vectors to obtain the semantic vector sequence (i.e., the latent vector sequence) of the reverse reference translation. The semantic vector sequence of the reverse reference translation is then subjected to exponential normalization to obtain the character count indication information of the reverse reference translation. This character count indication information is used to indicate how many characters in the third predicted text each character in the reverse reference translation corresponds to. In other words, it can indicate how many characters in the third predicted text are needed to translate each character in the reverse reference translation. Therefore, the character count indication information can be a character count sequence, the length of which is equal to the length of the character sequence of the input reverse reference translation. Each element in the character count sequence is used to represent how many characters in the third predicted text are needed to translate the character at the corresponding position in the character sequence of the reverse reference translation.

[0171] Next, the server transforms the character sequence of the reverse reference translation based on the character count indication information to obtain the character sequence to be decoded. The transformation process mainly makes the count of each character in the character sequence to be decoded match the character count indication information. For example, if a certain character in the character sequence of the reverse reference translation corresponds to the element 2 in the character count sequence, then this character will appear twice in the character sequence to be decoded.

[0172] Next, the character sequence to be decoded is embedded to obtain an embedding vector sequence; and the position information of each character in the character sequence is encoded to obtain a position encoding vector. Then, the embedding vector sequence and the position encoding vector are concatenated, and the concatenated vector is decoded to obtain the semantic vector sequence of the third predicted text. The semantic vector sequence of the third predicted text is then subjected to exponential normalization, and the character sequence of the third predicted text is output. The individual characters in this character sequence constitute the third predicted text.

[0173] 907. The server determines the third loss value based on the labeled text and the third predicted text.

[0174] In some embodiments, when obtaining the third loss value, the server may obtain the cross-entropy loss between the labeled text and the third predicted text as the third loss value. Optionally, it may also be combined with a contrastive learning method to determine the contrastive learning loss as the third loss value. This application embodiment does not specifically limit the type of loss function.

[0175] 908. Based on the second loss value and the third loss value, the server adjusts the parameters of the first translation model to obtain the second translation model.

[0176] In some embodiments, if the third stopping condition is not met, the parameters of the first translation model are iteratively adjusted based on the backpropagation algorithm until the third stopping condition is met, at which point the iteration stops and the second translation model is obtained.

[0177] Optionally, the third stopping condition includes: the BLEU value of the first translation model is greater than the third BLEU threshold; or, the sum of the second loss value and the third loss value is less than or equal to the third loss threshold; or, the number of iterations is greater than the third number threshold, etc. This application embodiment does not specifically limit the content of the third stopping condition. The third BLEU threshold can be any value greater than 0, for example, the third BLEU threshold can be 90% of the best BLEU of the original data; the third loss threshold can be any value greater than or equal to 0 and less than or equal to 1, for example, the third loss threshold is 0.2; the third number threshold can be any integer greater than or equal to 1, for example, the third number threshold is 20000. This application embodiment does not specifically limit the values ​​of the third BLEU threshold, the third loss threshold, and the third number threshold.

[0178] In steps 904-908 above, a possible implementation of the server adjusting the parameters of the first translation model based on the original text, the forward reference translation, the reverse reference translation, and the labeled text is shown. The original text and the forward reference translation can form a set of synthesized training sample pairs (forward distillation data), and the reverse reference translation and the labeled text can form another set of synthesized training sample pairs (backdistillation data). This doubles the amount of original data used for the continued training of the first translation model, providing complementary alignment information for low-frequency words, resulting in better alignment quality of the second translation model and improving the overall translation performance of the second translation model.

[0179] In some embodiments, the second translation model trained in step 908 can be directly applied to various text translation tasks without performing the subsequent step 909, which simplifies the model training process and reduces the complexity of model training. In some embodiments, the second translation model trained in step 908 may experience fluctuations in its translation performance from the source to the target due to the fusion of source-to-target and target-to-source alignment information. Therefore, step 909 can be performed to fine-tune the second translation model again using forward distillation data to optimize the performance of the final third translation model, resulting in a third translation model with better translation performance.

[0180] 909. Based on the original text and the positive reference translation, the server adjusts the parameters of the second translation model to obtain the third translation model.

[0181] In some embodiments, the process of the server adjusting the parameters of the second translation model, that is, the process of fine-tuning the second translation model, may include: the server calling the second translation model to translate the original text to obtain a fourth predicted text; determining a fourth loss value based on the positive reference translation and the fourth predicted text; and adjusting the parameters of the second translation model based on the fourth loss value to obtain the third translation model.

[0182] Step 909 is similar to step 403 above, and will not be described in detail here.

[0183] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0184] The method provided in this application pre-trains an initial translation model using original text and labeled text. Based on the pre-trained first translation model, a forward distillation data is extracted using a forward teacher model. The forward distillation data provides alignment information of low-frequency words from the source to the target. A reverse teacher model is used to extract back distillation data, which provides alignment information of low-frequency words from the target to the source. This bidirectional knowledge distillation method can extract a pair of complementary bidirectional distillation data. The first translation model is then trained on the bidirectional distillation data to obtain a second translation model. The second translation model is then trained on the forward distillation data to obtain a third translation model. This results in the third translation model being able to learn not only the original data distribution in the original and labeled text, such as the high-modulus distribution and low-frequency word distribution of the target language, but also the deterministic knowledge of more low-frequency word alignment recalled in both directions by the bidirectional distillation data. This enables accurate prediction of both high-frequency and low-frequency words in parallel decoding, greatly improving the translation accuracy of machine translation models supporting parallel translation.

[0185] In the above embodiments, training strategies based on pre-training of raw data are shown, which can be represented as follows: And a combined pipeline training strategy that combines pre-training on raw data with bidirectional knowledge distillation, which can be expressed as: In this application embodiment, another possible bidirectional distillation training strategy is introduced, which can be represented as follows: The bidirectional distillation training strategy only requires training with bidirectional distillation data, without the need for pre-training with raw data. This simplifies the training process of machine translation models that support parallel translation and reduces training complexity, which will be detailed below.

[0186] Figure 10 This is a flowchart illustrating a text translation method provided in an embodiment of this application. See also... Figure 10 This embodiment is applied to a computer device. The following description uses server 220 in the above-described implementation environment as an example. This embodiment includes the following steps: 1001. The server calls the positive teacher model to translate the original text and obtain a positive reference translation. This positive teacher model supports serial translation of the input text.

[0187] Step 1001 above is similar to step 902 above, and will not be repeated here.

[0188] 1002. The server calls the reverse teacher model to translate the annotated text corresponding to the semantics of the original text, and obtains the reverse reference translation. The annotated text and the forward reference translation belong to the same language, and the reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text.

[0189] Step 1002 above is similar to step 903 above, and will not be described in detail here.

[0190] 1003. Based on the original text, the forward reference translation, the reverse reference translation, and the labeled text, the server adjusts the parameters of the initial translation model to obtain the target translation model, which is used to translate the input text into semantically corresponding output text.

[0191] Step 1003 above is similar to steps 904-908 above, and will not be repeated here.

[0192] Figure 11 This is a flowchart illustrating the principle of a text translation method provided in this application, as shown in Figure 1100. The original data refers to the training sample pair [src, tgt] consisting of the original text src and the labeled text tgt. The forward distillation data refers to the synthetic sample pair [src, tgt'] consisting of the original text src and the forward reference translation tgt', where the forward reference translation tgt' is the translation obtained by the forward teacher model (AT model) through serial translation of the original text src. The backdistillation data refers to the synthetic sample pair [src', tgt] consisting of the back reference translation src' and the labeled text tgt, where the back reference translation src' is the translation obtained by the back teacher model (AT model) through serial translation of the labeled text tgt. During training, the forward distillation data [src, tgt'] and the backdistillation data [src', tgt] are merged into a new training set. The initial translation model (NAT model) is trained on this new training set to obtain the final target translation model.

[0193] In one exemplary embodiment, with This represents forward distillation data. This involves replacing the labeled text tgt in the original data with the positive reference translation tgt' synthesized by the positive AT model (positive teacher model), in order to... This represents back distillation data. It replaces the original text src in the original data with the reverse reference translation src' synthesized by the reverse AT model (reverse teacher model). Under the bidirectional distillation training strategy, it can connect the two different distillation data mentioned above to jointly train the NAT model.

[0194] Reverse distillation data It can be expressed using the following formula:

[0195] in, This represents the set of labeled text at the target end in the original data. Then, it represents the i-th labeled text, where i is an integer greater than or equal to 1 and less than or equal to N, and N represents the total number of training sample pairs in the original data. This represents an AT-based translation model, which is trained on raw data and used to translate input text from the target language to the source language.

[0196] Traditional knowledge distillation training methods (referring to using forward distillation data alone) (Training) simplifies the training data by replacing the target words with high-frequency target words. This helps to more easily align the source words with the target words to achieve higher bilingual coverage. However, because bilingual knowledge loses detailed information from the original data after distillation through the forward AT model, therefore... Forward knowledge distillation reduces the matching opportunities between low-frequency target words and source words. However, the bidirectional distillation training strategy provided in this application uses two... and AT models, respectively generate and Considering the complementarity of the two sets of data, the two sets of data are merged into a new training set and fed into the training of the NAT model. The new training set can not only maintain the advantages of low-frequency patterns, but also recall more low-frequency word connections in both directions through bidirectional distillation. It can recover low-frequency knowledge in the original data, improve alignment quality, and improve the overall translation performance of the NAT model.

[0197] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0198] The method provided in this application extracts forward distillation data using a forward teacher model, which provides alignment information for low-frequency words from the source to the target. It also extracts back distillation data using a reverse teacher model, which provides alignment information for low-frequency words from the target to the source. This bidirectional knowledge distillation method can extract a pair of complementary bidirectional distillation data. The initial translation model is trained on the bidirectional distillation data, so that the final target translation model can learn more deterministic knowledge of low-frequency word alignment recalled in both directions by the bidirectional distillation data. This enables accurate prediction of both high-frequency and low-frequency words in parallel decoding, greatly improving the translation accuracy of machine translation models that support parallel translation.

[0199] Based on the method provided in this application, effectiveness tests were conducted on five widely used translation datasets, including: WMT14 (English-German, En-De), WMT16 (Romanian-English, Ro-En), WMT17 (Chinese-English, Zh-En), WAT17 (Japanese-English, Ja-En), and WMT19 (En-De). The test results show that the NAT model trained using the method provided in this application significantly improves translation performance on multiple translation datasets. Analysis confirms that the improved translation performance stems from higher accuracy in word translation, particularly for low-frequency words, which will be detailed below.

[0200] Table 2 lists the decoding speed and translation performance of the AT model, the existing NAT model, and the NAT model trained based on the training strategy provided in this application, on the WMT14 dataset (En-De) and the WMT16 dataset (Ro-En), respectively. Here, Iteration represents the number of iterations for refinement, Speed ​​represents the decoding acceleration rate (a metric for decoding speed, expressed in units of the Transformer-Based model's decoding speed), ALF represents the translation accuracy for low-frequency words, and BLEU is a metric for translation performance.

[0201]

[0202] As shown in Table 2 above, the training strategy provided in this application was used to train the two high-performance NAT models, Mask-Predict and Levenshtein, to recall low-frequency words. It can be seen that, compared to the standard NAT model, the NAT model obtained based on the training strategy of this application significantly and consistently improves the translation performance (BLEU) of the NAT model across different language pairs while maintaining the same decoding speed. Furthermore, its ALF value also shows a significant and continuous improvement, indicating a substantial increase in translation accuracy for low-frequency words. On the WMT14 dataset (En-De), compared to the standard Mask-Predict model, the Mask-Predict model trained using this scheme shows a 0.8 increase in BLEU score and a significant 3.9 increase in ALF score. Furthermore, the training strategy of this scheme drives existing NAT models to achieve new state-of-the-art (SOTA) performance, namely, a maximum BLEU score of 28.2 on the WMT14 dataset (En-De) and a maximum BLEU score of 33.9 on the WMT16 dataset (Ro-En).

[0203] Furthermore, refined testing was conducted on the Mask-Predict model to examine whether the overall improvement in translation performance was due to increased accuracy and precision in translating low-frequency words. Therefore, a fine-grained analysis of lexical selection was performed, categorizing all words in the dictionary into "High," "Med," and "Low" classes based on selection frequency. The accuracy of word selection was then tested on words of different frequencies, as shown in Table 3. It can be seen that the improvement in translation performance primarily stems from increased accuracy in translating low-frequency words.

[0204]

[0205] It should be noted that the three different training strategies provided in this application's embodiments are: raw data pre-training strategy. Two-way distillation training strategy and combined pipeline training strategies This training strategy can be applied to any NAT model or any machine translation model that supports parallel translation. Here, we only use the Mask-Predict model and the Levenshtein model as examples for testing to verify the effectiveness and versatility of this training strategy. Furthermore, due to its strong complementarity with traditional training methods, it can greatly improve the performance of machine translation models that support parallel translation, such as the NAT model. Moreover, machine translation models that support parallel translation, such as the NAT model, can not only be applied to translation scenarios, but also to multiple NLP tasks. In speech tasks, machine translation models that support parallel translation, such as the NAT model, have also received widespread attention and application.

[0206] Due to their unique acceleration effects, machine translation models that support parallel translation, such as NAT models, have the potential to become the next generation of neural network translation systems. Currently, the training strategy provided in this application can be applied to various NAT model structures, multiple NLP tasks, and even tasks such as speech recognition. Its effectiveness has been verified in machine translation systems through the aforementioned test results. This training strategy helps in understanding and improving translation system models, enhancing user experience, and can be applied to various artificial intelligence translation systems to support translation products such as intelligent translation. Furthermore, it can provide external APIs (Application Programming Interfaces) to other products with translation needs, assisting overseas businesses, saving translation costs, and improving translation efficiency. In addition, it can open up the translation platform and API for external To C (To Consumer, individual users) and To B (To Business, institutional users) users.

[0207] Figure 12 This is a schematic diagram of the structure of a text translation device provided in an embodiment of this application. Please refer to it. Figure 12 The device includes: The first adjustment module 1201 is used to adjust the parameters of the initial translation model based on the original text and the labeled text corresponding to the semantics of the original text to obtain the first translation model, which supports parallel translation of the input text; Translation module 1202 is used to call the positive teacher model to translate the original text and obtain a positive reference translation. The positive reference translation and the labeled text belong to the same language. The positive teacher model supports serial translation of the input text. The second adjustment module 1203 is used to adjust the parameters of the first translation model based on the original text and the positive reference translation to obtain a second translation model, which is used to translate the input text into semantically corresponding output text.

[0208] The apparatus provided in this application pre-trains an initial translation model using original text and labeled text. Based on the first translation model obtained from the pre-training, it continues to train using the original text and the positive reference translation obtained after knowledge distillation by the positive teacher model. This allows the final second translation model to not only learn the original data distribution in the original text and labeled text, such as the high-modulus distribution and low-frequency word distribution of the target language, but also to learn the deterministic knowledge obtained after knowledge distillation by the positive teacher model. This enables accurate prediction of both high-frequency and low-frequency words in parallel decoding, greatly improving the translation accuracy of machine translation models that support parallel translation.

[0209] In one possible implementation, the first adjustment module 1201 includes: The first translation unit is used to call the initial translation model to translate the original text and obtain the first predicted text. A determining unit is used to determine a first loss value based on the labeled text and the first predicted text; The first adjustment unit is used to adjust the parameters of the initial translation model based on the first loss value to obtain the first translation model.

[0210] In one possible implementation, the first adjustment unit is used for: Based on the annotated text and the first predicted text, the BLEU score for the bilingual assessment study was determined; If neither the first loss value nor the BLEU value meets the first stopping condition, the parameters of the initial translation model are iteratively adjusted. If either the first loss value or the BLEU value meets the first stopping condition, the iteration stops, and the first translation model is obtained.

[0211] In one possible implementation, the first stopping condition is: the first loss value is less than or equal to a first loss threshold; or, the BLEU value is greater than the first BLEU threshold.

[0212] In one possible implementation, the second adjustment module 1203 is used for: The first translation model is invoked to translate the original text, resulting in the second predicted text. Based on the positive reference translation and the second predicted text, the second loss value is determined; Based on the second loss value, the parameters of the first translation model are adjusted to obtain the second translation model.

[0213] In one possible implementation, based on Figure 12 The device comprises, wherein the second adjustment module 1203 includes: The second translation unit is used to call the reverse teacher model to translate the annotated text and obtain a reverse reference translation. The reverse reference translation belongs to the same language as the original text. The reverse teacher model supports serial translation of the input text. The second adjustment unit is used to adjust the parameters of the first translation model based on the original text, the positive reference translation, the negative reference translation, and the annotated text, to obtain the second translation model.

[0214] In one possible implementation, the second adjustment unit is used for: The first translation model is invoked to translate the original text, resulting in the second predicted text. Based on the positive reference translation and the second predicted text, the second loss value is determined; The first translation model is invoked to translate the reverse reference translation, resulting in the third predicted text. Based on the labeled text and the third predicted text, determine the third loss value; Based on the second and third loss values, the parameters of the first translation model are adjusted to obtain the second translation model.

[0215] In one possible implementation, based on Figure 12 The device comprises: The third adjustment module is used to adjust the parameters of the second translation model based on the original text and the positive reference translation to obtain the third translation model.

[0216] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0217] It should be noted that the text translation device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the text translation device and the text translation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the text translation method embodiments, which will not be repeated here.

[0218] Figure 13 This is a schematic diagram of the structure of a text translation device provided in an embodiment of this application. Please refer to it. Figure 13 The device includes: The first translation module 1301 is used to call the positive teacher model to translate the original text and obtain a positive reference translation. The positive teacher model supports serial translation of the input text. The second translation module 1302 is used to call the reverse teacher model to translate the annotated text corresponding to the semantics of the original text to obtain a reverse reference translation. The annotated text and the forward reference translation belong to the same language, and the reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text. The adjustment module 1303 is used to adjust the parameters of the initial translation model based on the original text, the forward reference translation, the reverse reference translation, and the labeled text to obtain the target translation model, which is used to translate the input text into semantically corresponding output text.

[0219] The apparatus provided in this application extracts forward distillation data using a forward teacher model, which provides alignment information for low-frequency words from the source to the target. It also extracts back distillation data using a reverse teacher model, which provides alignment information for low-frequency words from the target to the source. This bidirectional knowledge distillation method can extract a pair of complementary bidirectional distillation data. The initial translation model is trained on the bidirectional distillation data, so that the final target translation model can learn more deterministic knowledge of low-frequency word alignment recalled in both directions by the bidirectional distillation data. This enables accurate prediction of both high-frequency and low-frequency words in parallel decoding, greatly improving the translation accuracy of machine translation models that support parallel translation.

[0220] It should be noted that the text translation device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the text translation device and the text translation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the text translation method embodiments, which will not be repeated here.

[0221] Figure 14 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application, such as... Figure 14 As shown, terminal 1400 is used as an example of a computer device for explanation. Optionally, the device type of terminal 1400 includes: smartphones, tablets, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, or desktop computers. Terminal 1400 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0222] Typically, terminal 1400 includes a processor 1401 and a memory 1402.

[0223] Optionally, the processor 1401 includes one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Optionally, the processor 1401 is implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). In some embodiments, the processor 1401 includes a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 1401 integrates a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, the processor 1401 also includes an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0224] In some embodiments, memory 1402 includes one or more computer-readable storage media, optionally non-transitory. Optionally, memory 1402 also includes high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1402 is used to store at least one program code, which is executed by processor 1401 to implement the text translation methods provided in the various embodiments of this application.

[0225] In some embodiments, the terminal 1400 may also optionally include a peripheral device interface 1403 and at least one peripheral device. The processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407, and a power supply 1409.

[0226] Peripheral interface 1403 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1401 and memory 1402. In some embodiments, processor 1401, memory 1402 and peripheral interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1401, memory 1402 and peripheral interface 1403 are implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0227] The radio frequency (RF) circuit 1404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1404 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1404 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. Optionally, the RF circuit 1404 communicates with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1404 also includes circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0228] Display screen 1405 is used to display a UI (User Interface). Optionally, the UI includes graphics, text, icons, videos, and any combination thereof. When display screen 1405 is a touch display screen, display screen 1405 also has the ability to collect touch signals on or above the surface of display screen 1405. The touch signals can be input to processor 1401 for processing as control signals. Optionally, display screen 1405 is also used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there is one display screen 1405, which is set on the front panel of terminal 1400; in other embodiments, there are at least two display screens 1405, which are respectively set on different surfaces of terminal 1400 or have a folded design; in still other embodiments, display screen 1405 is a flexible display screen, which is set on the curved surface or folded surface of terminal 1400. Furthermore, optionally, display screen 1405 is set as a non-rectangular irregular shape, that is, an irregularly shaped screen. Optionally, the display screen 1405 is made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0229] The camera assembly 1406 is used to acquire images or videos. Optionally, the camera assembly 1406 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1406 also includes a flash. Optionally, the flash is a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, used for light compensation at different color temperatures.

[0230] In some embodiments, the audio circuit 1407 includes a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1401 for processing, or input to the radio frequency circuit 1404 to realize voice communication. For stereo acquisition or noise reduction purposes, multiple microphones are used, each located at a different part of the terminal 1400. Optionally, the microphone is an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 1401 or the radio frequency circuit 1404 into sound waves. Optionally, the speaker is a conventional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as distance measurement. In some embodiments, the audio circuit 1407 also includes a headphone jack.

[0231] Power supply 1409 is used to power the various components in terminal 1400. Optionally, power supply 1409 is AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 1409 includes a rechargeable battery, the rechargeable battery supports wired or wireless charging. The rechargeable battery also supports fast charging technology.

[0232] In some embodiments, the terminal 1400 further includes one or more sensors 1410. The one or more sensors 1410 include, but are not limited to: an accelerometer 1411, a gyroscope 1412, a pressure sensor 1413, an optical sensor 1415, and a proximity sensor 1416.

[0233] In some embodiments, the accelerometer 1411 detects the magnitude of acceleration along the three coordinate axes of a coordinate system established with the terminal 1400. For example, the accelerometer 1411 is used to detect the components of gravitational acceleration along the three coordinate axes. Optionally, the processor 1401 controls the display screen 1405 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by the accelerometer 1411. The accelerometer 1411 is also used for acquiring motion data from games or the user.

[0234] In some embodiments, the gyroscope sensor 1412 detects the orientation and rotation angle of the terminal 1400. The gyroscope sensor 1412 and the accelerometer sensor 1411 work together to acquire the user's 3D movements on the terminal 1400. Based on the data acquired by the gyroscope sensor 1412, the processor 1401 implements the following functions: motion sensing (e.g., changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0235] Optionally, the pressure sensor 1413 is disposed on the side bezel of the terminal 1400 and / or on the lower layer of the display screen 1405. When the pressure sensor 1413 is disposed on the side bezel of the terminal 1400, it can detect the user's grip signal on the terminal 1400, and the processor 1401 performs left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1413. When the pressure sensor 1413 is disposed on the lower layer of the display screen 1405, the processor 1401 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1405. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0236] An optical sensor 1415 is used to collect ambient light intensity. In one embodiment, the processor 1401 controls the display brightness of the display screen 1405 based on the ambient light intensity collected by the optical sensor 1415. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1405 is increased; when the ambient light intensity is low, the display brightness of the display screen 1405 is decreased. In another embodiment, the processor 1401 also dynamically adjusts the shooting parameters of the camera assembly 1406 based on the ambient light intensity collected by the optical sensor 1415.

[0237] The proximity sensor 1416, also known as a distance sensor, is typically located on the front panel of the terminal 1400. The proximity sensor 1416 is used to detect the distance between the user and the front of the terminal 1400. In one embodiment, when the proximity sensor 1416 detects that the distance between the user and the front of the terminal 1400 is gradually decreasing, the processor 1401 controls the display screen 1405 to switch from a screen-on state to a screen-off state; when the proximity sensor 1416 detects that the distance between the user and the front of the terminal 1400 is gradually increasing, the processor 1401 controls the display screen 1405 to switch from a screen-off state to a screen-on state.

[0238] Those skilled in the art will understand that Figure 14 The structure shown does not constitute a limitation on terminal 1400, which may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0239] Figure 15 This is a schematic diagram of a computer device 1500 provided in an embodiment of this application. The computer device 1500 can vary significantly due to differences in configuration or performance. The computer device 1500 includes one or more Central Processing Units (CPUs) 1501 and one or more memories 1502. The memories 1502 store at least one computer program, which is loaded and executed by the one or more processors 1501 to implement the text translation methods provided in the various embodiments described above. Optionally, the computer device 1500 also includes wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The computer device 1500 also includes other components for implementing device functions, which will not be elaborated here.

[0240] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, which can be executed by a processor in a terminal to perform the text translation methods in the various embodiments described above. For example, the computer-readable storage medium includes ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0241] In an exemplary embodiment, a computer program product or computer program is also provided, including one or more lines of program code stored in a computer-readable storage medium. One or more processors of a computer device are capable of reading the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, enabling the computer device to perform the text translation method described in the above embodiments.

[0242] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. Optionally, the program is stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0243] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.< / eos>

Claims

1. A text translation method, characterized in that, The method includes: Based on the original text and the labeled text corresponding to the semantics of the original text, the parameters of the initial translation model are adjusted to obtain the first translation model, which supports parallel translation of the input text; The original text is translated by calling the positive teacher model to obtain a positive reference translation. The positive reference translation and the labeled text belong to the same language. The positive teacher model supports serial translation of the input text. The reverse teacher model is invoked to translate the annotated text to obtain a reverse reference translation. The reverse reference translation is in the same language as the original text. The reverse teacher model supports serial translation of the input text. Based on the original text, the forward reference translation, the reverse reference translation, and the labeled text, the parameters of the first translation model are adjusted to obtain a second translation model, which is used to translate the input text into semantically corresponding output text.

2. The method according to claim 1, characterized in that, The process of adjusting the parameters of the initial translation model based on the original text and the labeled text corresponding to the semantics of the original text to obtain the first translation model includes: The initial translation model is invoked to translate the original text, resulting in the first predicted text; Based on the labeled text and the first predicted text, a first loss value is determined; Based on the first loss value, the parameters of the initial translation model are adjusted to obtain the first translation model.

3. The method according to claim 2, characterized in that, The step of adjusting the parameters of the initial translation model based on the first loss value to obtain the first translation model includes: Based on the labeled text and the first predicted text, determine the BLEU score for the bilingual assessment study; If neither the first loss value nor the BLEU value meets the first stopping condition, the parameters of the initial translation model are iteratively adjusted. If either the first loss value or the BLEU value meets the first stopping condition, the iteration stops, and the first translation model is obtained.

4. The method according to claim 3, characterized in that, The first stopping condition is: the first loss value is less than or equal to the first loss threshold; or, the BLEU value is greater than the first BLEU threshold.

5. The method according to claim 1, characterized in that, The step of adjusting the parameters of the first translation model based on the original text and the positive reference translation to obtain the second translation model includes: The first translation model is invoked to translate the original text, resulting in the second predicted text; Based on the positive reference translation and the second predicted text, a second loss value is determined; Based on the second loss value, the parameters of the first translation model are adjusted to obtain the second translation model.

6. The method according to claim 1, characterized in that, The step of adjusting the parameters of the first translation model based on the original text, the forward reference translation, the reverse reference translation, and the annotated text to obtain the second translation model includes: The first translation model is invoked to translate the original text, resulting in the second predicted text; Based on the positive reference translation and the second predicted text, a second loss value is determined; The first translation model is invoked to translate the reverse reference translation, resulting in a third predicted text. Based on the labeled text and the third predicted text, a third loss value is determined; Based on the second loss value and the third loss value, the parameters of the first translation model are adjusted to obtain the second translation model.

7. The method according to claim 1, characterized in that, After adjusting the parameters of the first translation model based on the original text, the forward reference translation, the reverse reference translation, and the annotated text to obtain the second translation model, the method further includes: Based on the original text and the positive reference translation, the parameters of the second translation model are adjusted to obtain the third translation model.

8. A text translation method, characterized in that, The method includes: The original text is translated by calling a positive teacher model to obtain a positive reference translation. The positive teacher model supports serial translation of the input text. The reverse teacher model is invoked to translate the annotated text corresponding to the semantics of the original text, resulting in a reverse reference translation. The annotated text and the forward reference translation belong to the same language, and the reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text. Based on the original text, the forward reference translation, the reverse reference translation, and the labeled text, the parameters of the initial translation model are adjusted to obtain the target translation model. The initial translation model supports parallel translation of the input text, and the target translation model is used to translate the input text into semantically corresponding output text.

9. A text translation device, characterized in that, The device includes: The first adjustment module is used to adjust the parameters of the initial translation model based on the original text and the labeled text corresponding to the semantics of the original text to obtain the first translation model. The initial translation model supports parallel translation of the input text. The translation module is used to call the positive teacher model to translate the original text and obtain a positive reference translation. The positive reference translation and the labeled text belong to the same language. The positive teacher model supports serial translation of the input text. The second adjustment module is used to call the reverse teacher model to translate the labeled text to obtain a reverse reference translation. The reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text. Based on the original text, the forward reference translation, the reverse reference translation, and the labeled text, the parameters of the first translation model are adjusted to obtain a second translation model. The second translation model is used to translate the input text into semantically corresponding output text.

10. The apparatus according to claim 9, characterized in that, The first adjustment module includes: The first translation unit is used to call the initial translation model to translate the original text and obtain the first predicted text; A determining unit is configured to determine a first loss value based on the labeled text and the first predicted text; The first adjustment unit is used to adjust the parameters of the initial translation model based on the first loss value to obtain the first translation model.

11. The apparatus according to claim 10, characterized in that, The first adjustment unit is used for: Based on the labeled text and the first predicted text, determine the BLEU score for the bilingual assessment study; If neither the first loss value nor the BLEU value meets the first stopping condition, the parameters of the initial translation model are iteratively adjusted. If either the first loss value or the BLEU value meets the first stopping condition, the iteration stops, and the first translation model is obtained.

12. A text translation device, characterized in that, The device includes: The first translation module is used to call the positive teacher model to translate the original text and obtain a positive reference translation. The positive teacher model supports serial translation of the input text. The second translation module is used to call the reverse teacher model to translate the annotated text corresponding to the semantics of the original text to obtain a reverse reference translation. The annotated text and the forward reference translation belong to the same language, and the reverse reference translation and the original text belong to the same language. The reverse teacher model supports serial translation of the input text. The adjustment module is used to adjust the parameters of the initial translation model based on the original text, the forward reference translation, the reverse reference translation, and the labeled text to obtain the target translation model. The initial translation model supports parallel translation of the input text, and the target translation model is used to translate the input text into semantically corresponding output text.

13. A computer device, characterized in that, The computer device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the text translation method as described in any one of claims 1 to 7 or claim 8.

14. A storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the text translation method as described in any one of claims 1 to 7 or claim 8.

Images