Machine translation model generation method and apparatus

Abstract

The application provides a machine translation model generation method and device, comprising: obtaining training data; training a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model is used to translate a text to be translated into a text translation; the machine translation model comprises a memory enhancement adapter layer, a first memory and a second memory; the memory enhancement adapter layer is used to retrieve information in the first memory and / or the second memory and utilize the information; the first memory is used to save source language data obtained by performing reverse translation and forward calculation processing on the training data; and the second memory is used to save target language data obtained by performing reverse translation and forward calculation processing on the training data. The application saves the training data in a form acceptable to the model, reads information from the training data to assist the model when performing a translation task, and achieves the effect of adapting to various translation scenarios while taking into account lower algorithm time complexity and better model performance.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and apparatus for generating machine translation models. Background Technology

[0002] Existing neural machine translation models primarily employ a 1-for-N service model, representing general-purpose translation models suitable for most people's translation needs in common domains. However, for specialized fields such as finance, law, healthcare, IT, education, and manufacturing, general-purpose models often fail to perform well. To achieve accurate translation in specialized domains—that is, to provide diverse services such as style customization and domain customization—it is usually necessary to train the entire model specifically, but this approach is overly complex.

[0003] In existing technologies, the adapter can initially achieve this goal by adding a module with a few parameters. However, due to the simple design of the added module, the model performance is poor. kNN-MT (k-nearest neighbor machine translation) is a parameterless method that can achieve this goal. It can significantly improve the performance of any trained neural machine translation model in intra-domain, inter-domain, and multilingual scenarios by building a retrieval table (Datastore) for k-nearest neighbor retrieval. However, the algorithm has high time complexity, slow decoding speed, and high time cost.

[0004] In summary, existing machine translation models cannot simultaneously achieve both low algorithmic time complexity and good model performance when adapting to various translation scenarios. Summary of the Invention

[0005] This invention provides a machine translation model generation method and apparatus to address the shortcomings of existing technologies that cannot simultaneously achieve low algorithm time complexity and good model performance when adapting to various translation scenarios, thereby achieving a better balance between low algorithm time complexity and good model performance when adapting to various translation scenarios.

[0006] This invention provides a method for generating machine translation models, comprising:

[0007] Obtain training data;

[0008] A machine translation model is obtained by training a pre-stored deep neural network using training data; the machine translation model is used to translate the text to be translated into a translated text.

[0009] The machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory. The memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory. The first memory is used to store the source language data obtained after the training data has undergone back-translation and forward computation processing. The second memory is used to store the target language data obtained after the training data has undergone back-translation and forward computation processing.

[0010] According to the machine translation model generation method provided by the present invention, a machine translation model is obtained by training training data based on a pre-stored deep neural network, specifically including:

[0011] A predetermined number of semantic phrases are extracted from the training data using a syntactic structure tree, wherein the semantic phrases are semantically coherent phrases;

[0012] The semantic phrases are input into a pre-built reverse translation model to obtain source language semantic phrases; the semantic phrases and the corresponding source language semantic phrases together form pseudo-parallel training data;

[0013] The pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a teacher-forced manner to obtain a first memory and a second memory; wherein, the deep neural network includes an encoder and a decoder.

[0014] The first memory includes the feedforward layer output of the encoder of the deep neural network;

[0015] The second memory includes the self-attention layer output of the decoder of the deep neural network.

[0016] According to a machine translation model generation method provided by the present invention, the pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a tutor-forced manner to obtain a first memory and a second memory, and then the method further includes:

[0017] Based on preset rules, the first memory, the second memory, and the memory enhancement adapter layer are connected to the deep neural network to obtain an untrained machine translation model.

[0018] The training data is input into the machine translation model;

[0019] Based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, and obtains a first translation result based on the first retrieval result.

[0020] According to a machine translation model generation method provided by the present invention, based on the training data, a memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, and obtains a first translation result based on the first retrieval result, and then further includes:

[0021] Information stored in the first memory and the second memory is discarded according to a preset probability to obtain the discarded first memory and the discarded second memory;

[0022] The memory enhancement adapter layer retrieves information in the discarded first memory and / or discarded second memory to obtain a second retrieval result, and obtains a second translation result based on the second retrieval result;

[0023] The machine translation model is trained by a loss function so that the first translation result and the second translation result are the same under a preset error, so as to obtain the trained machine translation model.

[0024] According to a machine translation model generation method provided by the present invention, the loss function includes:

[0025]

[0026] in, This indicates the machine translation model's ability to fit the training data when using the first memory before discarding and the second memory before discarding. This indicates the machine translation model's ability to fit the training data when using the discarded first memory and the discarded second memory. Let P(y|x,θ,M) represent the KL divergence, and let P(y|x,θ,M) represent the probability of generating the target sentence y given the source sentence x and the machine translation model θ, using the non-dropout memory M. Given a source sentence x and a model θ, using discarded memory The probability of generating the target sentence y, where y represents the target sentence, x represents the source sentence, θ represents the model, and M represents the first memory and the second memory before discarding. Let α represent the first memory and the second memory after discarding, and let β represent the first regularization coefficient and β represent the second regularization coefficient.

[0027] According to a machine translation model generation method provided by the present invention, the machine translation model is used to translate a text to be translated into a translated text, specifically including:

[0028] Based on the text to be translated, the memory enhancement adapter layer uses a soft attention mechanism to retrieve information in the first memory and / or the second memory to obtain auxiliary translation information;

[0029] The translated text is obtained based on the aforementioned translation assistance information.

[0030] The present invention also provides a machine translation model generation apparatus, comprising:

[0031] The acquisition unit is used to acquire training data;

[0032] A generation unit is used to train a machine translation model based on a pre-stored deep neural network using training data; the machine translation model is used to translate the text to be translated into a translated text.

[0033] The machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory. The memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory. The first memory is used to store the source language data obtained after the training data has undergone back-translation and forward computation processing. The second memory is used to store the target language data obtained after the training data has undergone back-translation and forward computation processing.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the machine translation model generation method as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the machine translation model generation method as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the machine translation model generation method as described above.

[0037] This invention provides a machine translation model generation method and apparatus, which acquires training data; trains a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model is used to translate text to be translated into a target text; the machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory; the first memory is used to store source language data obtained after back-translation and forward computation processing of the training data; the second memory is used to store target language data obtained after back-translation and forward computation processing of the training data. This invention stores the training data in a form acceptable to the model, and retrieves necessary information from it to assist the machine translation model during translation tasks, achieving a better balance between lower algorithm time complexity and better model performance, while adapting to various translation scenarios. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is one of the flowcharts illustrating the machine translation model generation method provided by the present invention;

[0040] Figure 2 This is a schematic diagram of the network architecture of the machine translation model in the machine translation model generation method provided by the present invention;

[0041] Figure 3 This is the second flowchart of the machine translation model generation method provided by the present invention;

[0042] Figure 4 This is the third flowchart of the machine translation model generation method provided by the present invention;

[0043] Figure 5 This is a schematic diagram of the machine translation model generation device provided by the present invention;

[0044] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0045] Figure label:

[0046] 510: Acquisition unit; 520: Generation unit;

[0047] 610: Processor; 620: Communication interface; 630: Memory; 640: Communication bus. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] Addressing the issues of poor model performance and / or high translation time costs in existing technologies, this invention provides a machine translation model generation method. The aim is to combine the advantages of parametric and non-parametric methods, enabling it to:

[0050] 1) More fully retain information in customized data, especially when the amount of data is small;

[0051] 2) Perform translation tasks more quickly, especially when translating in batches;

[0052] 3) Adaptable to various customized translation service scenarios, such as style customization and domain customization;

[0053] 4) It still exhibits excellent performance when there is only single-language data on the target side.

[0054] The following is combined Figures 1-4 The machine translation model generation method of the present invention is described below. Figure 1 This is a flowchart illustrating the machine translation model generation method provided by the present invention, as shown below. Figure 1 As shown, it includes the following steps:

[0055] Step 110: Obtain training data.

[0056] The data acquired is training data, corresponding to the machine translation model generation task, i.e., the machine translation model customization task. The training data can be customized data from various fields, styles, and languages. For example, fields such as finance, law, health, IT, education, and manufacturing; styles such as Lu Xun's style, Lin Daiyu's style, and Zhen Huan's style; languages ​​such as English, Chinese, German, and French. This invention does not impose any limitations on these aspects.

[0057] In other words, the essence of this invention is to customize a personalized translation model based on a pre-existing pre-trained translation model and combine it with newly arrived customized data (training data) to complete personalized translation services (such as stylized translation, legal translation, IT translation, etc.).

[0058] Step 120: A machine translation model is trained using training data based on a pre-stored deep neural network; the machine translation model is used to translate the text to be translated into a translated text; the machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory; the first memory is used to store the source language data obtained after the training data has undergone back-translation and forward computation processing; the second memory is used to store the target language data obtained after the training data has undergone back-translation and forward computation processing.

[0059] The training data is target-side monolingual data, customized data selected to enable the machine translation model to adapt to various translation service scenarios. The training data is stored in memory in a format acceptable to the machine translation model, comprising a first memory and a second memory.

[0060] When performing a translation task, the text to be translated is first acquired. Then, the acquired text is input into a pre-trained machine translation model. The memory enhancement adapter layer reads necessary information from memory to assist the machine translation model. Specifically, the memory enhancement adapter layer uses a soft attention mechanism to retrieve and utilize similar information in memory. Based on the auxiliary translation information, the machine translation model outputs the translated text. Corresponding to the machine translation model, the text to be translated and the output translated text can be from various fields, styles, and languages. For example, fields such as finance, law, health, IT, education, and manufacturing; styles such as Lu Xun's style, Lin Daiyu's style, and Zhen Huan's style; languages ​​such as English, Chinese, German, and French. This invention does not impose any limitations.

[0061] like Figure 2 , Figure 2 The network architecture of the machine translation model is illustrated. The machine translation model includes a memory-augmented adapter layer, a first memory, and a second memory, as well as a feed-forward layer (FFN layer), a self-attention layer, and a cross-attention layer. In some embodiments, the feed-forward layer, self-attention layer, and cross-attention layer are inherent structures of the deep neural network itself. The first memory, second memory, and memory-augmented adapter layer are connected to the deep neural network. The memory-augmented adapter layer works in conjunction with the first and second memories connected to the deep neural network to achieve style customization and domain customization.

[0062] Considering the limited learning capacity of additional neural network parameters on training data, the training data is stored in a form acceptable to the model in a first memory and a second memory. The first memory stores the source language data obtained after back-translation and forward computation of the training data, and the second memory stores the target language data obtained after back-translation and forward computation of the training data. It is important to understand that the numbers "first" and "second" are only used to distinguish the memory locations and do not represent the execution order. The data stored in the first and second memories differs; the first memory stores the source language data obtained from the processed training data, and the second memory stores the target language data obtained from the processed training data. For example, for a machine translation model trained to perform an English-to-Chinese translation task, the first memory stores the English data obtained from the processed training data, and the second memory stores the Chinese data obtained from the processed training data.

[0063] During translation tasks, necessary information is retrieved from memory to assist the machine translation model. To extract data from the first and second memories, a memory enhancement adapter layer is added. This layer uses a soft attention mechanism to retrieve and utilize similar information from the memory. Specifically, in practice, the memory enhancement adapter layer retrieves information from the first and / or second memories to obtain auxiliary translation information, and then uses this information to obtain the translated text. Figure 2 In some embodiments, the first memory and the second memory each have a memory enhancement adapter layer matched with them, which may be referred to as the first memory enhancement adapter layer and the second memory enhancement adapter layer.

[0064] The memory enhancement adapter layer, trained using a dropout training method, can extract data from memory while improving translation quality. The trained first and second memories, along with the memory enhancement adapter layer, are added to the pre-stored deep neural network to obtain the trained machine translation model. In other words, the first memory, second memory, and memory enhancement adapter layer are additional components added to the pre-stored deep neural network; only they are trainable, hence this method is inexpensive.

[0065] Machine translation models also include self-attention layers, feedforward layers, and cross-attention layers. These are pre-existing structures in deep neural networks. The self-attention layer learns the relationships between words in the sentence's context. The feedforward layer further extracts features from the output of the self-attention layer. The cross-attention layer uses alternating attention to capture local information.

[0066] Based on the above embodiments, in this method, a machine translation model is trained using training data based on a pre-stored deep neural network, specifically including:

[0067] A predetermined number of semantic phrases are extracted from the training data using a syntactic structure tree, wherein the semantic phrases are semantically coherent phrases;

[0068] The semantic phrases are input into a pre-built reverse translation model to obtain source language semantic phrases; the semantic phrases and the corresponding source language semantic phrases together form pseudo-parallel training data;

[0069] The pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a teacher-forced manner to obtain a first memory and a second memory; wherein, the deep neural network includes an encoder and a decoder.

[0070] The first memory includes the feedforward layer output of the encoder of the deep neural network;

[0071] The second memory includes the self-attention layer output of the decoder of the deep neural network.

[0072] Specifically, the first step is to acquire training data. This training data consists of target-side monolingual data, selected to enable the machine translation model to adapt to various customized translation service scenarios. To achieve diverse services such as style customization and domain customization, it's crucial to retain as much information as possible from the customized data at a low cost. To reasonably extract information at different granularities from the customized data to meet the varying needs of retrieval, a syntactic structure tree is used as an auxiliary tool during training. Figure 3 (a) By performing lateral cues on the leaf nodes of each subtree in the syntactic structure tree, a predetermined number of semantic phrases of various granularities are obtained. A semantic phrase is a semantically coherent phrase. In some embodiments, the predetermined number can be 500k, or 500,000.

[0073] The obtained semantic phrases are input into a pre-built back-translation model to obtain the source language semantic phrases. The pre-built back-translation model is a pre-trained back-translation model corresponding to the machine translation model, used to train the machine translation model. The semantic phrases and their corresponding source language semantic phrases together form pseudo-parallel training data, such as... Figure 3 (b)

[0074] The obtained pseudo-parallel data is then used for forward computation in a pre-stored deep neural network using a teacher-forcing method. The pre-stored deep neural network includes an encoder and a decoder. In some embodiments, the pre-stored deep neural network can be an existing, complete, pre-trained translation model that is not fully adapted to the customized translation service scenario. The outputs of the feedforward layer (FFN layer) of the encoder and the self-attention layer of the decoder are saved. The output of the encoder's feedforward layer (FFN layer) is saved as a first memory, and the output of the decoder's self-attention layer is saved as a second memory, such as... Figure 4 .

[0075] Based on the above embodiments, in this method, the pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a tutor-forced manner to obtain a first memory and a second memory, and then the method further includes:

[0076] Based on preset rules, the first memory, the second memory, and the memory enhancement adapter layer are connected to the deep neural network to obtain an untrained machine translation model.

[0077] The training data is input into the machine translation model;

[0078] Based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, and obtains a first translation result based on the first retrieval result.

[0079] Based on the above embodiments, in this method, based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, obtains a first translation result based on the first retrieval result, and then further includes:

[0080] Information stored in the first memory and the second memory is discarded according to a preset probability to obtain the discarded first memory and the discarded second memory;

[0081] The memory enhancement adapter layer retrieves information in the discarded first memory and / or discarded second memory to obtain a second retrieval result, and obtains a second translation result based on the second retrieval result;

[0082] The machine translation model is trained by a loss function so that the first translation result and the second translation result are the same under a preset error, so as to obtain the trained machine translation model.

[0083] Specifically, after the first and second memories are constructed, to avoid over-reliance on the information in the memories by the machine translation model, which could lead to overfitting, this invention designs a dropout training method. That is, when training the memory enhancement adapter layer, information in the memories is discarded according to a preset probability, and a loss function is used to force the machine translation model to obtain similar outputs when "dropping" and "not dropping". This training method can significantly improve translation quality. In other words, assuming a preset probability of a%, the memory enhancement adapter layer is specified to retain only 1-a% of the retrieved information when retrieving information from the first and / or second memories. The memory enhancement adapter layer is trained using a loss function so that it obtains similar outputs when retaining 1-a% of the retrieved information as when retaining all the retrieved information. In some embodiments, the preset probability can be 0.1.

[0084] Specifically, such as Figure 2 A first memory, a second memory, and a memory enhancement adapter layer are connected to the deep neural network to obtain an untrained machine translation model. In some embodiments, the first memory, the second memory, and the memory enhancement adapter layer are connected to a pre-stored pre-trained translation model, which may be a Transformer translation model. The memory enhancement adapter layer, in conjunction with the first and second memories connected to the pre-trained translation model, achieves style customization and domain customization.

[0085] Next, the memory enhancement adapter layer is trained. The goal is to force the machine translation model to produce similar outputs regardless of whether information is "discarded" or "not discarded" during training. In other words, the machine translation model's output falls into two categories during training: "discarded" and "not discarded." The "not discarded" case involves inputting training data into an untrained machine translation model. The memory enhancement adapter layer retrieves and utilizes information from the first and / or second memories based on the training data to obtain a first translation result. The "discarded" case involves discarding information stored in the first and second memories according to a preset probability, resulting in discarded first and second memories. The memory enhancement adapter layer then retrieves and utilizes information from these discarded memories to obtain a second translation result.

[0086] After obtaining two translation results, a machine translation model is trained using a loss function to make the first and second translation results the same under a preset error, i.e., the two output results are basically similar, thus obtaining the trained machine translation model. The similarity is quantified using KL divergence.

[0087] Based on the above embodiments, in this method, the loss function includes:

[0088]

[0089] in, This indicates the machine translation model's ability to fit the training data when using the first memory before discarding and the second memory before discarding. This indicates the machine translation model's ability to fit the training data when using the discarded first memory and the discarded second memory. Let P(y|x,θ,M) represent the KL divergence, and let P(y|x,θ,M) represent the probability of generating the target sentence y given the source sentence x and the machine translation model θ, using the non-dropout memory M. Given a source sentence x and a model θ, using discarded memory The probability of generating the target sentence y, where y represents the target sentence, x represents the source sentence, θ represents the model, and M represents the first memory and the second memory before discarding. Let α represent the first memory and the second memory after discarding, and let β represent the first regularization coefficient and β represent the second regularization coefficient.

[0090] Specifically, the first term of the formula This is a necessary loss term for training a machine translation model, called the negative log-likelihood, which represents the model's ability to fit externally customized data when using full memory. Using full memory means that the memory-enhanced adapter layer does not discard data when retrieving information from the first and / or second memory. The second term in the formula... It is also the negative log-likelihood, representing the model's ability to fit external customized data when using a memory that randomly discards data. The third term of the formula... KL divergence is a constraint that forces the model to produce similar outputs when using full memory and when using memory with randomly dropped data. It is a key constraint to avoid over-reliance on memory (overfitting). P(y|x,θ,M) represents the probability of generating the target sentence y when using memory without dropping data, given the source sentence x and the machine translation model θ. Given a source sentence x and a model θ, using discarded memory The probability of generating the target sentence y, where y represents the target sentence, x represents the source sentence, θ represents the model, and M represents the first memory and the second memory before discarding. Let α represent the first memory and the second memory after discarding, and let β represent the first regularization coefficient and β represent the second regularization coefficient.

[0091] Based on the above embodiments, in this method, the machine translation model is used to translate the text to be translated into a translated text, specifically including:

[0092] Based on the text to be translated, the memory enhancement adapter layer uses a soft attention mechanism to retrieve information in the first memory and / or the second memory to obtain auxiliary translation information;

[0093] The translated text is obtained based on the aforementioned translation assistance information.

[0094] Specifically, when performing a translation task, the text to be translated is first acquired. Then, the acquired text is input into a pre-trained machine translation model. The memory enhancement adapter layer retrieves necessary information from memory to assist the machine translation model. The memory enhancement adapter layer uses a soft attention mechanism to retrieve and utilize similar information in memory. Specifically, in practice, the memory enhancement adapter layer retrieves information from the first and / or second memory to obtain auxiliary translation information, obtains the translated text based on the auxiliary translation information, and the machine translation model outputs the translated text.

[0095] In some embodiments, the present invention compares the provided machine translation model generation method with existing translation methods, and the machine translation model generation method provided by the present invention can achieve the following effects:

[0096] 1) In terms of style customization, the translation quality is higher than that of existing main solutions, averaging as high as 2.1 BLEU;

[0097] 2) In terms of domain-specific customization, the translation quality is higher than that of existing major solutions, averaging as high as 1.2 BLEU;

[0098] 3) The translation speed is 3-4 times faster than the existing mainstream solution (kNN-MT);

[0099] 4) It still outperforms existing mainstream solutions when there is only a small amount of monolingual data.

[0100] This invention provides a machine translation model generation method, which involves acquiring training data; training a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model is used to translate text to be translated into a target text; the machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer is used to retrieve and utilize information in the first and / or second memory; the first memory is used to store source language data obtained after back-translation and forward computation processing of the training data; the second memory is used to store target language data obtained after back-translation and forward computation processing of the training data. This invention stores the training data in a form acceptable to the model, and retrieves necessary information from it to assist the machine translation model during translation tasks, achieving a better balance between lower algorithm time complexity and better model performance, while adapting to various translation scenarios.

[0101] The machine translation model generation apparatus provided by the present invention is described below. The machine translation model generation apparatus described below can be referred to in correspondence with the machine translation model generation method described above.

[0102] Figure 5 This is a schematic diagram of the machine translation model generation device provided by the present invention, as shown below. Figure 5 As shown, it includes an acquisition unit 510 and a generation unit 520, wherein,

[0103] The acquisition unit 510 is used to acquire training data; the generation unit 520 is used to train a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model is used to translate the text to be translated into a translated text; the machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory; the first memory is used to store the source language data obtained after the training data has undergone back-translation and forward computation processing; the second memory is used to store the target language data obtained after the training data has undergone back-translation and forward computation processing.

[0104] Based on the above embodiments, in this device, a machine translation model is trained using training data based on a pre-stored deep neural network, specifically including:

[0105] A predetermined number of semantic phrases are extracted from the training data using a syntactic structure tree, wherein the semantic phrases are semantically coherent phrases;

[0106] The semantic phrases are input into a pre-built reverse translation model to obtain source language semantic phrases; the semantic phrases and the corresponding source language semantic phrases together form pseudo-parallel training data;

[0107] The pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a teacher-forced manner to obtain a first memory and a second memory; wherein, the deep neural network includes an encoder and a decoder.

[0108] The first memory includes the feedforward layer output of the encoder of the deep neural network;

[0109] The second memory includes the self-attention layer output of the decoder of the deep neural network.

[0110] Based on the above embodiments, in this device, the pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a tutor-forced manner to obtain a first memory and a second memory, and then the device further includes:

[0111] Based on preset rules, the first memory, the second memory, and the memory enhancement adapter layer are connected to the deep neural network to obtain an untrained machine translation model.

[0112] The training data is input into the machine translation model;

[0113] Based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, and obtains a first translation result based on the first retrieval result.

[0114] Based on the above embodiments, in this device, based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, obtains a first translation result based on the first retrieval result, and then further includes:

[0115] Information stored in the first memory and the second memory is discarded according to a preset probability to obtain the discarded first memory and the discarded second memory;

[0116] The memory enhancement adapter layer retrieves information in the discarded first memory and / or discarded second memory to obtain a second retrieval result, and obtains a second translation result based on the second retrieval result;

[0117] The machine translation model is trained by a loss function so that the first translation result and the second translation result are the same under a preset error, so as to obtain the trained machine translation model.

[0118] Based on the above embodiments, in this device, the loss function includes:

[0119]

[0120] in, This indicates the machine translation model's ability to fit the training data when using the first memory before discarding and the second memory before discarding. This indicates the machine translation model's ability to fit the training data when using the discarded first memory and the discarded second memory. Let P(y|x,θ,M) represent the KL divergence, and let P(y|x,θ,M) represent the probability of generating the target sentence y given the source sentence x and the machine translation model θ, using the non-dropout memory M. Given a source sentence x and a model θ, using discarded memory The probability of generating the target sentence y, where y represents the target sentence, x represents the source sentence, θ represents the model, and M represents the first memory and the second memory before discarding. Let α represent the first memory and the second memory after discarding, and let β represent the first regularization coefficient and β represent the second regularization coefficient.

[0121] Based on the above embodiments, in this device, the machine translation model is used to translate the text to be translated into a translated text, specifically including:

[0122] Based on the text to be translated, the memory enhancement adapter layer uses a soft attention mechanism to retrieve information in the first memory and / or the second memory to obtain auxiliary translation information;

[0123] The translated text is obtained based on the aforementioned translation assistance information.

[0124] This invention provides a machine translation model generation method and apparatus, which acquires training data; trains a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model is used to translate text to be translated into a target text; the machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory; the first memory is used to store source language data obtained after back-translation and forward computation processing of the training data; the second memory is used to store target language data obtained after back-translation and forward computation processing of the training data. This invention stores the training data in a form acceptable to the model, and retrieves necessary information from it to assist the machine translation model during translation tasks, achieving a better balance between lower algorithm time complexity and better model performance, while adapting to various translation scenarios.

[0125] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a machine translation model generation method, which includes: acquiring training data; training a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model being used to translate the text to be translated into a translated text; the machine translation model including a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer being used to retrieve and utilize information in the first memory and / or the second memory; the first memory being used to store source language data obtained after back-translation and forward computation processing of the training data; and the second memory being used to store target language data obtained after back-translation and forward computation processing of the training data.

[0126] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the machine translation model generation method provided by the above methods. The method includes: acquiring training data; training a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model being used to translate the text to be translated into a text translation; the machine translation model including a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer being used to retrieve and utilize information in the first memory and / or the second memory; the first memory being used to store source language data obtained after the training data has undergone back-translation and forward computation processing; and the second memory being used to store target language data obtained after the training data has undergone back-translation and forward computation processing.

[0128] On another front, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the machine translation model generation method provided by the above-described methods. This method includes: acquiring training data; training a machine translation model based on a pre-stored deep neural network using the training data; the machine translation model being used to translate a text to be translated into a text translation; the machine translation model including a memory enhancement adapter layer, a first memory, and a second memory; the memory enhancement adapter layer being used to retrieve and utilize information in the first memory and / or the second memory; the first memory being used to store source language data obtained after the training data has undergone back-translation and forward computation processing; and the second memory being used to store target language data obtained after the training data has undergone back-translation and forward computation processing. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating machine translation models, characterized in that, include: Obtain training data; A machine translation model is obtained by training a pre-stored deep neural network using training data; the machine translation model is used to translate the text to be translated into a translated text. The machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; The memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory; The first memory is used to store the source language data obtained after the training data has undergone back translation and forward computation processing; The second memory is used to store the target language data obtained after the training data has undergone back translation and forward computation processing.

2. The machine translation model generation method according to claim 1, characterized in that, A machine translation model is trained using pre-stored deep neural networks and training data, specifically including: A predetermined number of semantic phrases are extracted from the training data using a syntactic structure tree, wherein the semantic phrases are semantically coherent phrases; The semantic phrases are input into a pre-built reverse translation model to obtain source language semantic phrases; the semantic phrases and the corresponding source language semantic phrases together form pseudo-parallel training data; The pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a teacher-forced manner to obtain a first memory and a second memory; wherein, the deep neural network includes an encoder and a decoder. The first memory includes the feedforward layer output of the encoder of the deep neural network; The second memory includes the self-attention layer output of the decoder of the deep neural network.

3. The machine translation model generation method according to claim 2, characterized in that, The pseudo-parallel training data is subjected to forward computation in a pre-stored deep neural network in a tutor-forced manner to obtain a first memory and a second memory. The process then includes: Based on preset rules, the first memory, the second memory, and the memory enhancement adapter layer are connected to the deep neural network to obtain an untrained machine translation model. The training data is input into the machine translation model; Based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, and obtains a first translation result based on the first retrieval result.

4. The machine translation model generation method according to claim 3, characterized in that, Based on the training data, the memory enhancement adapter layer retrieves information in the first memory and / or the second memory to obtain a first retrieval result, and obtains a first translation result based on the first retrieval result, and then further includes: Information stored in the first memory and the second memory is discarded according to a preset probability to obtain the discarded first memory and the discarded second memory; The memory enhancement adapter layer retrieves information in the discarded first memory and / or discarded second memory to obtain a second retrieval result, and obtains a second translation result based on the second retrieval result; The machine translation model is trained by a loss function so that the first translation result and the second translation result are the same under a preset error, so as to obtain the trained machine translation model.

5. The machine translation model generation method according to claim 4, characterized in that, The loss function includes: in, This indicates the machine translation model's ability to fit the training data when using the first memory before discarding and the second memory before discarding. This indicates the machine translation model's ability to fit the training data when using the discarded first memory and the discarded second memory. Let P(y|x,θ,M) represent the KL divergence, and let P(y|x,θ,M) represent the probability of generating the target sentence y given the source sentence x and the machine translation model θ, using the non-dropout memory M. Given a source sentence x and a model θ, using discarded memory The probability of generating the target sentence y, where y represents the target sentence, x represents the source sentence, θ represents the model, and M represents the first memory and the second memory before discarding. Let α represent the first memory and the second memory after discarding, and let β represent the first regularization coefficient and β represent the second regularization coefficient.

6. The machine translation model generation method according to claim 1, characterized in that, The machine translation model is used to translate the text to be translated into a translated text, specifically including: Based on the text to be translated, the memory enhancement adapter layer uses a soft attention mechanism to retrieve information in the first memory and / or the second memory to obtain auxiliary translation information; The translated text is obtained based on the aforementioned translation assistance information.

7. A machine translation model generation device, characterized in that, The acquisition unit is used to acquire training data; A generation unit is used to train a machine translation model based on a pre-stored deep neural network using training data; the machine translation model is used to translate the text to be translated into a translated text. The machine translation model includes a memory enhancement adapter layer, a first memory, and a second memory; The memory enhancement adapter layer is used to retrieve and utilize information in the first memory and / or the second memory; The first memory is used to store the source language data obtained after the training data has undergone back translation and forward computation processing; The second memory is used to store the target language data obtained after the training data has undergone back translation and forward computation processing.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the machine translation model generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the machine translation model generation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the machine translation model generation method as described in any one of claims 1 to 6.

Images