A non-autoregressive machine translation system, method and electronic device

By splitting the non-autoregressive neural machine translation model into a non-autoregressive translation decoding module and a masked language model decoding module, the shortcomings of existing models in translation efficiency and effectiveness are addressed, achieving high-efficiency and high-quality translation results.

CN116502654BActive Publication Date: 2024-11-01SHISHI TECHNOLOGY (PINGHU) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310321437.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-29
Publication Date
2024-11-01
Estimated Expiration
2043-03-29

AI Technical Summary

Technical Problem

Existing non-autoregressive neural machine translation models struggle to improve both translation efficiency and accuracy simultaneously. While fully non-autoregressive models enhance translation efficiency, they fail to effectively capture target-side dependencies. Furthermore, iterative non-autoregressive models exhibit decreased translation efficiency when improving translation accuracy.

Method used

The decoder of a traditional non-autoregressive neural machine translation model is split into two modules: a non-autoregressive translation decoding module and a masked language model decoding module. The non-autoregressive translation decoding module generates potential translation results, and the masked language model decoding module is used to mask and predict errors in the potential translation results to optimize translation quality.

Benefits of technology

It achieves high-efficiency and high-quality machine translation, improves translation accuracy, reduces multimodal problems, and generates more fluent and meaningful translation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116502654B_ABST
    Figure CN116502654B_ABST
Patent Text Reader

Abstract

The present invention relates to a non-autoregressive machine translation system, method and electronic device. The system comprises: sequentially serializing and encoding a source sequence X to obtain a coded sequence; taking the serialized source sequence X as a reference, decoding the coded sequence to obtain a sequence Y with noise pot ; For the noisy sequence Y pot The error information in the mask sequence is masked to obtain a mask sequence; the encoded sequence is used as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result; the prediction result is compared with the noisy sequence Y pot The unmasked information in the text is combined to obtain the translation result. The present invention can effectively translate the text to be translated, and the translation efficiency is high.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of non-autoregressive machine translation, and in particular to a non-autoregressive machine translation system, method and electronic device. Background Art

[0002] Machine translation is the process of using computers to convert one natural language (source language) into another natural language (target language). It is a major branch of natural language processing and one of the ultimate goals of artificial intelligence, with important scientific research value. In recent years, with the rapid development of deep learning, neural machine translation models with different neural networks as the main structure, such as convolutional neural networks, recurrent neural networks, etc., have made breakthrough progress in machine translation tasks, especially the "Transformer" model that uses the self-attention mechanism. Today, mainstream neural machine translation models are improved on the basis of the "Transformer". Its basic model is an "encoder-decoder" structure. By adding self-attention mechanisms, masking mechanisms, word vector representations, position encoding and other strategies, it can effectively capture global context information and establish long-range dependence on the target, thereby extracting more powerful features and obtaining better translation results. However, existing neural machine translation models all use an "autoregressive" decoding scheme, which means that the translation results are generated in a left-to-right sequential manner. The translation results at each position depend not only on the original input sentence, but also on the translation results at the previous position. Although this autoregressive translation mode effectively improves the translation effect, it also greatly increases the required translation time. Especially for longer sentences, the huge translation delay is unacceptable. In order to solve this problem and accelerate the decoding process of neural machine translation, a non-autoregressive neural machine translation method was proposed. The non-autoregressive neural machine translation model aims to output the translation results of all positions at the same time through parallel decoding, thereby improving the translation speed. Non-autoregressive neural machine translation has received widespread attention since it was proposed. However, despite the huge improvement in translation speed, the parallel decoding method makes it difficult for the model to capture sequence information well, resulting in its effect being far inferior to the autoregressive method. After that, a large number of non-autoregressive neural machine translation works are committed to making up for the difference in effect between it and autoregressive neural machine translation, pursuing the balance of "efficiency-effect". They can be roughly divided into "completely non-autoregressive neural machine translation" and "iteration-based non-autoregressive neural machine translation" according to different methods. Completely non-autoregressive neural machine translation is a strict non-autoregressive method that obtains all translation results in parallel through only one round of decoding. Iterative non-autoregressive neural machine translation obtains partial translation results through parallel decoding in each round of iteration, and then optimizes and updates the translation results through iteration. Compared with completely non-autoregressive neural machine translation, it achieves better results, but sacrifices a certain translation speed. Although a large number of studies on non-autoregressive neural machine translation have enabled it to significantly improve translation speed while achieving results close to or even exceeding those of autoregressive methods, there is still much room for improvement.

[0003] The non-autoregressive neural machine translation model was first proposed by Gu et al., who used a potential fertility predictor to predict the number of words in the target translation corresponding to each word in the original input, and then decoded and translated in parallel, which improved the efficiency of decoding, but the effect was much worse than that of the transformer using the autoregressive method. Later, Guo et al. proposed a method based on phrase table lookup and embedding mapping to enable the input of the decoding end to obtain part of the target side information. Sun et al. used conditional random fields to model the target position context. Chan et al. tried to model the potential alignment to help the model predict. Marjan et al. and Du, Tu, and Jiang introduced a loss function that is more suitable for the non-autoregressive neural machine translation model, so that it can better learn information during the training stage. In addition, using hidden variables as part of the model is also a popular method, such as alignment information, position information, syntactic tags, etc. between the source sentence and the target sentence. At the same time, in order to better achieve a balance between translation speed and translation quality, non-autoregressive neural machine translation based on iteration has also made progress. Unlike fully non-autoregressive neural machine translation, iterative non-autoregressive neural machine translation learns conditional distributions on partially observed generation results in each iteration, and uses multiple iterations to optimize the translation results. There are many optimization methods, such as heuristic denoising, insertion and deletion, mask re-prediction, etc. However, these iterative methods all improve the quality of translation by sacrificing translation speed. Recently, inspired by course learning, Qian et al. adopted a "glance sampling strategy" to dynamically select positions for mask re-prediction based on the performance of the model, which significantly improved the effect of fully non-autoregressive neural machine translation. Based on its simple method and excellent performance, many works have been carried out on its basis to further improve the performance. Gu and Kong attributed the key to the fully non-autoregressive neural machine translation model to dependency reduction in the target language space at the decoding end. They combined several useful techniques during the training process and achieved the best results at the time. Huang et al. introduced a deep supervision mechanism and additional layer-by-layer prediction for each decoder layer. Zhan et al. proposed a general method to enhance the target dependency from the decoder input and the decoder self-attention inside the non-autoregressive neural translation decoder, achieving the best results of the fully non-autoregressive neural translation model.

[0004] It can be seen that it is still difficult for existing non-autoregressive neural machine translation models to achieve a two-way improvement in efficiency and effect. Although fully non-autoregressive neural machine translation has greatly improved translation efficiency, it cannot achieve the expected effect because it cannot capture target-side dependencies well. Although the iterative non-autoregressive neural machine translation model uses an iterative optimization method to improve the accuracy of translation on the basis of parallel decoding, its advantage in translation efficiency is greatly reduced due to the multiple rounds of reasoning required for iteration. Although many methods have been proposed to improve the effect of fully non-autoregressive neural translation models, most of them only use some effective techniques in the training process. In the reasoning stage, they still do not capture the information on the target side to guide the translation results. Therefore, each word in the target translation is translated independently, which still leads to the multimodal problem common in non-autoregressive neural translation models and reduces the quality of translation results. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that the non-autoregressive neural machine translation model in the prior art is difficult to improve both the translation efficiency and the translation effect.

[0006] In order to solve the above technical problems, the present invention provides a non-autoregressive machine translation system, comprising:

[0007] Encoder module: used to serialize and encode the source sequence X in sequence to obtain the encoded sequence;

[0008] Decoding module: used to decode the coded sequence based on the serialized source sequence X to obtain the sequence Y with noise pot ;

[0009] Mask module: used to mask the noisy sequence Y pot Mask the error information in to obtain a mask sequence;

[0010] Prediction module: used for using the encoding sequence as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result;

[0011] Combination module: used to combine the prediction result with the noisy sequence Y pot The unmasked information in is combined to obtain the translation result.

[0012] In one embodiment of the present invention, the decoding module comprises a non-autoregressive translation decoding module, and the non-autoregressive translation decoding module is used to decode the encoding sequence;

[0013] The formula of the non-autoregressive translation decoding module is:

[0014]

[0015] in, represents the hidden state of the last layer of the non-autoregressive translation decoding module, Dec 1:N-k represents the non-autoregressive translation decoding module, represents the 1st to NKth decoder layers of the non-autoregressive translation decoding module, Y pot represents the noisy sequence output by the non-autoregressive translation decoding module, H represents a soft copy or uniform copy of the word embedding representation from the source sequence X, and E X represents a coding sequence;

[0016] The loss formula of the non-autoregressive translation decoding module during training is:

[0017]

[0018] Among them, T represents the total length of the correct target sequence during training, and y t1 is a noisy sequence Y pot The corresponding correct result, X is the source sequence, and θ is the model parameter.

[0019] In one embodiment of the present invention, the prediction module includes a masked language model decoding module, and the masked language model decoding module is used to predict the masked error information in the mask sequence;

[0020] The formula of the masked language model decoding module is:

[0021]

[0022] Among them, E Y Represents the hidden state of the last layer of the masked language model decoding module, Dec N-k+1:N represents the masked language model decoding module, represents the N-K+1th layer to the Nth layer of the masked language model decoding module, Y' represents the mask sequence, and emb(·) is the word embedding function that maps discrete words to vector representations;

[0023] The loss formula of the masked language model decoding module during training is:

[0024]

[0025] Among them, y t2 represents the correct result corresponding to the masked part in the correct target sequence during training, Y obs represents the unmasked part of the correct target sequence during training, X represents the source sequence, and θ represents the model parameters.

[0026] In one embodiment of the present invention, the total loss formula generated by the non-autoregressive translation decoding module in the decoding module and the masked language model decoding module in the prediction module during the training process is: L = L pot +L mlm +L len , where L len Represents the length loss of predicted translation results during training.

[0027] In one embodiment of the present invention, an optimization module is further included: used to optimize the model parameter θ through the total loss L, and the formula is: Among them, γ represents the learning rate, and ← represents replacing the original model parameter symbol with the new model parameter.

[0028] In one embodiment of the present invention, the mask module further includes a search module, which is used to search for the noisy sequence Y during inference. pot Find the location of the error message when masking the error message in .

[0029] In one embodiment of the present invention, the method used by the search module to search for the position of the error information includes a threshold method and a formula method;

[0030] The threshold method includes:

[0031] Get the noisy sequence Y pot The confidence level of each mark in

[0032] Compare the confidence of each identification with a first preset threshold α1, if the identification is higher than the first preset threshold α1, then the identification is regarded as correct information; if the identification is lower than the first preset threshold α1, then the identification is regarded as wrong information;

[0033] The formula method satisfies:

[0034]

[0035] Wherein, mean(confidence) represents the average confidence of all identifiers in the current generated sentence, α2 is the second preset threshold, when mean(confidence)>α2, scale is set to α2, and the identifier with confidence less than α2 in the current generated sentence is used as error information; when mean(confidence)≤α2, scale is set to the average confidence value mean(confidence), and the identifier with confidence less than mean(confidence) in the current generated sentence is used as error information.

[0036] In order to solve the above technical problems, the present invention provides a non-autoregressive machine translation method, comprising:

[0037] Step S1: Serialize and encode the source sequence X in sequence to obtain an encoded sequence;

[0038] Step S2: Taking the serialized source sequence X as the reference, the coded sequence is decoded to obtain the sequence Y with noise pot ;

[0039] Step S3: For the noisy sequence Y pot Mask the error information in to obtain a mask sequence;

[0040] Step S4: using the encoding sequence as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result;

[0041] Step S5: Compare the prediction result with the noisy sequence Y pot The unmasked information in is combined to obtain the translation result.

[0042] To solve the above technical problems, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above non-autoregressive machine translation method when executing the computer program.

[0043] In order to solve the above technical problem, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above non-autoregressive machine translation method are implemented.

[0044] The above technical solution of the present invention has the following advantages compared with the prior art:

[0045] The non-autoregressive machine translation system of the present invention can perform machine translation tasks with high efficiency and high quality. It mainly splits the decoder of the traditional non-autoregressive neural machine translation model into two modules. The first module (NAT module) is used to generate potential translation results, and the second module (MLM module) is modified based on the potential translation results to further improve the translation quality. The framework has good generalization and can be applied to most existing non-autoregressive machine translation models.

[0046] The present invention innovatively proposes an efficient decoding strategy, specifically: performing the first decoding through a non-autoregressive translation decoding module (NAT module) to obtain a possible translation result, then masking possible error positions in the translation result, inputting the sentence after the masking strategy into a masked language model decoding module (MLM module), re-predicting and updating the masked positions, and experiments have shown that the translation quality of the present invention is good. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0048] Figure 1 is a flow chart of the present invention;

[0049] Figure 2 Schematic diagram of the overall framework of the non-autoregressive machine translation model in an embodiment of the present invention;

[0050] Figure 3 is a comparison chart of experimental results of machine translation tasks on four WMT datasets in an embodiment of the present invention;

[0051] Figure 4 1. It is a comparison diagram of the prediction effects of the non-autoregressive translation decoding module and the masked language model decoding module in the embodiment of the present invention, each taking different numbers of layers;

[0052] Figure 5 is a comparison diagram of experimental results of different training strategies for the masked language model decoding module in an embodiment of the present invention;

[0053] Figure 6 It is a comparison chart of experimental results conducted on the IWSLT14 DE-EN dataset without data distillation in an embodiment of the present invention;

[0054] Figure 7 is a schematic diagram of the effect of input sentence length on different models in an embodiment of the present invention;

[0055] Figure 8 It is a comparison chart of the results of two specific machine translation examples using different models in the embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0057] Embodiment 1

[0058] Reference Figure 1 As shown, the present invention provides a non-autoregressive machine translation system, comprising:

[0059] Encoder module: used to serialize and encode the source sequence X in sequence to obtain the encoded sequence;

[0060] Decoding module: used to decode the coded sequence based on the serialized source sequence X to obtain the sequence Y with noise pot ;

[0061] The decoding module includes a non-autoregressive translation decoding module (NAT module), and the non-autoregressive translation decoding module is used to decode the encoding sequence;

[0062] Mask module: used to mask the noisy sequence Y pot Mask the error information in to obtain a mask sequence;

[0063] Prediction module: used for using the encoding sequence as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result;

[0064] The prediction module includes a masked language model decoding module (MLM module), and the masked language model decoding module is used to predict the masked error information in the mask sequence;

[0065] Combination module: used to combine the prediction result with the noisy sequence Y pot The unmasked information in is combined to obtain the translation result.

[0066] The non-autoregressive machine translation system of the present invention can perform machine translation tasks with high efficiency and high quality. It mainly splits the decoder of the traditional non-autoregressive neural machine translation model into two modules. The first module (NAT module) is used to generate potential translation results, and the second module (MLM module) is modified based on the potential translation results to further improve the translation quality. The framework has good generalization and can be applied to most existing non-autoregressive machine translation models.

[0067] The following is a detailed introduction to this embodiment:

[0068] This example is implemented through a non-autoregressive machine translation model. The overall framework of the model is detailed in Figure 2, mainly divides the decoder of the traditional non-autoregressive neural machine translation model into two sub-modules, namely the non-autoregressive translation decoding module (NAT Decoder Block) in the decoding module and the masked language model decoding module (MLMDecoder Block) in the prediction module. Specifically, the first non-autoregressive translation decoding module (NAT module) receives information from the encoder and the original input sentence to generate a potential possible translation, and then the second masked language model decoding module (MLM module) tries to obtain some valuable contextual information from the potential translation to capture the target side dependency. However, since the output of the first decoding module (NAT module) is the result of parallel translation, it can be regarded as a noisy result that inevitably carries erroneous information. These errors may be further amplified in the second decoding module (MLM module), resulting in the inability to obtain the correct target dependency information, resulting in poor translation results. In order to better obtain information from the potential translation output for the first time, it is crucial to discard the erroneous information. Inspired by the masked language model, this embodiment masks the error positions in the potential translation output by the first decoding module (NAT module) when training the second decoding module (MLM module) to ensure the correctness of the acquired information.

[0069] The following is a detailed introduction to the structure and function of each module in the non-autoregressive machine translation model, as well as how the entire model is trained and inferred. It should be noted that (1) during the masking process of training, the model masks the correct target sequence; (2) during the masking process of inference, the model masks the noisy sequence Y output by the NAT module. pot Masks potential error messages in .

[0070] (I) Encoder module

[0071] See also Figure 2 The structure of the encoder module is roughly the same as the original "Transformer". The encoder module is responsible for mapping the input language sequence through a series of layers to the word vector output of the hidden layer. Given the original input sequence (i.e., source sequence X), the hidden layer information E containing the context information in the original input is obtained through N encoder layers. X , recorded as:

[0072]

[0073] Among them, Enc 1:N (X) indicates encoder module, represents the 1st to Nth layers of the encoder module, emb(·) is the word embedding function that maps discrete words to vector representations, and EX is the hidden layer output (i.e., the encoded sequence) of the last layer of the encoder module.

[0074] (II) Non-autoregressive translation decoding module (NAT module)

[0075] The NAT module receives as input a uniform or soft copy of the word embedding representation of emb(X).

[0076] H=f(emb(X))

[0077] Where f represents the copy function. The non-autoregressive translation decoding module receives information from the encoder module, performs a decoding process internally, and passes the hidden state of the last decoder layer through a softmax layer (the softmax layer is used to obtain the probability distribution of the predicted position corresponding to the entire vocabulary, and then selects the result with the highest probability through argmax. For example, assuming that the entire vocabulary is 5 words, the probability obtained after softmax is the probability of the current predicted position corresponding to these 5 words, which may be [0.1, 0.2, 0.2, 0.1, 0.4], and then take the highest probability 0.4 as the result of the current predicted position) to obtain the possible output sequence.

[0078]

[0079] in, represents the hidden state of the last layer of the non-autoregressive translation decoding module, Dec 1:N-k represents the non-autoregressive translation decoding module, represents the 1st to NKth decoder layers of the non-autoregressive translation decoding module, H represents a soft copy or uniform copy of the word embedding representation from the source sequence X, W represents the first parameter matrix to be trained, and Y pot represents the possible translation results output by the non-autoregressive translation decoding module, express After softmax, we get the probability distribution of each word in the possible translation results. pot It can be regarded as a noisy version of the target output result, so it can be trained directly with the real value. The loss generated here is called L pot , can be obtained by the following formula:

[0080]

[0081] Among them, T represents the total length of the correct target sequence during training, and y t1 is a noisy sequence Y pot The corresponding correct result, X is the source sequence, and θ is the model parameter.

[0082] (III) Masked Language Model Decoding Module (MLM Module)

[0083] The possible translation result Y is predicted in the NAT submodule pot After that, the masked language model decoding module is committed to obtaining some valuable contextual information from it and updating it to a more reliable translation result in the inference stage. This embodiment does not change the structure of the masked language model layer inside the masked language model decoding module, but only changes the training scheme. Specifically, given the target output sentence sequence Y, the masked language model decoding module adopts a uniform sampling strategy to select a subset of the target sequence and mask them. Therefore, during training, the target sequence Y is divided into Y mask and Y obs The two parts are used as the input of the masked language model decoding module. The masked position can be regarded as the first possible potential translation error, and the remaining unmasked position can be regarded as valuable context information. At the same time, the masked language model decoding module also receives the output information E of the encoding module in the encoder-decoder attention layer. X To obtain useful original input information. In this embodiment, the modeling process can be described by the following formula:

[0084]

[0085] P(Y mlm |E Y )=softmax(W'·E Y )

[0086] Among them, E Y Represents the hidden state of the last layer of the masked language model decoding module, Dec N-k+1:N represents the masked language model decoding module, represents the N-K+1th layer to the Nth layer of the masked language model decoding module, and Y' represents the mask sequence. W' is the second parameter matrix to be trained, Y mlm is the output of the masked language model decoding module, where only the masked position is of interest. In this way, during training, only the loss value at the masked position needs to be calculated, using L mlm It is expressed as follows:

[0087]

[0088] Among them, y t2 represents the correct result corresponding to the masked part in the correct target sequence during training, Y obs represents the unmasked part of the correct target sequence during training, X represents the source sequence, and θ represents the model parameters.

[0089] (IV) Length prediction module

[0090] Since the non-autoregressive neural machine translation model cannot automatically end the inference like the autoregressive method, it is necessary to train an additional length prediction module to obtain the length of the target translation. Specifically, in the Encoder module, a special character "length" is added to the front of the original input sequence (i.e., the length prediction module), and the Encoder module performs the prediction, and the output result obtained by passing it through the encoder module is used as the length. In this embodiment, the loss generated in the length prediction module is recorded as L len Therefore, the total loss of the non-autoregressive translation decoding module in the decoding module and the masked language model decoding module in the prediction module during the training process consists of three parts:

[0091] L=L pot +L mlm +L len

[0092] The above content has discussed the overall model structure and functions of the updateable non-autoregressive machine translation model. The following introduces the specific process of model training and inference.

[0093] First, given the training data (X, Y) and the learning rate γ, the expected output of this embodiment is the parameter θ learned by the model.

[0094] Initialize the model parameters θ and repeat the following steps until the model converges: For each pair of data (X, Y) in the training data, where X is the input sentence (i.e., source sequence) and Y is the target translation (i.e., correct target series), perform the following steps: Generate a noisy sequence Y through a non-autoregressive translation decoding module (NAT module) pot . Sample and mask the target translation, and transform the noisy sequence Y pot Divided into Y obs and Y mask Two parts (this process is a training process, so the correct target sequence should be divided into Y based on a uniform probability distribution obs and Y mask ), and then predict Y through the masked language model decoding module (MLM module) mask The total loss L generated in the above process is calculated as L = L pot +L mlm +L len , calculate the gradient value through the optimization module back propagation and update the parameter value

[0095] During the inference process, the updateable non-autoregressive machine translation model obtains the translation result through a single round of generation. First, given an original output, its hidden state is obtained through the encoder module, and the encoder module predicts the length of the target translation. After that, the first decoding is performed through the non-autoregressive translation decoding module to obtain a possible translation result. Then, the mask module is used to mask the possible error positions in the translation result. For the specific masking method, please refer to the "Mask Strategy" below. The sentence after the masking strategy is input into the masked language model decoding module, and the masked position is re-predicted and updated.

[0096] The masking strategy is as follows:

[0097] During the reasoning process, it is necessary to mask the possible erroneous parts of the first translation result. How to find the location of the error is a difficult problem. This embodiment uses a search module to find the location of the error information. Unlike the random sampling method used in previous work, this embodiment uses a threshold setting method and a formula method. Specifically, (1) the threshold method outputs the confidence of the result at each position while the non-autoregressive translation decoding module outputs the possible potential translation results. This embodiment sets a first preset threshold α1. For the position in the translation result whose confidence is less than the first preset threshold α1, this embodiment regards it as an erroneous translation and masks it (MASK). Through a large number of experiments, this embodiment found that the optimal first preset threshold α1 setting should be changed according to different models. (2) The expression of the formula method is as follows:

[0098]

[0099] Among them, mean(confidence) represents the average confidence of all identifiers in the current generated sentence, α2 is the second preset threshold, when mean(confidence)>α2, scale is set to α2, and the identifier with confidence less than α2 in the current generated sentence is regarded as error information and masked (MASK); when mean(confidence)≤α2, scale is set to the average confidence value mean(confidence), and the identifier with confidence less than mean(confidence) in the current generated sentence is regarded as error information and masked (MASK).

[0100] The following experimental comparison is based on the mask of the threshold method as an example:

[0101] for Figure 4-Figure 8: RenewNAT is the model of this embodiment, GLAT is the existing model, DSLP is the existing model, GLAT w / ours means: the decoder of GLAT is used as the NAT decoding module of the model of this embodiment, but the MLM decoding module still uses the method of this embodiment. GLAT+DSLP w / ours means: the decoder of GLAT+DSLP is used as the NAT decoding module of the model of this embodiment, but the MLM decoding module still uses the method of this embodiment.

[0102] This embodiment compares the updateable non-autoregressive machine translation model with many different basic models. Compared with the original non-autoregressive neural machine translation model, the model of this embodiment has achieved significant improvement on each translation data set, especially on WMT14 DEEN and WMT16 ROEN data sets, achieving an improvement of 2.2, 2.6 Bleu scores. In addition, this embodiment is also compared with more powerful complete non-autoregressive neural machine translation models such as GLAT and DSLP, and has also achieved a Bleu improvement of about 0.5 while keeping the reasoning speed and model parameter size unchanged. At the same time, there are also a large number of experiments to verify that this embodiment is effective, indicating that the updateable non-autoregressive machine translation model of this embodiment can be applied to other complete non-autoregressive neural machine translation models. Although there may be a certain gap in effect with the non-autoregressive neural machine translation model based on iteration, its advantage in decoding speed is very significant.

[0103] Figure 3 This is the experimental result of the machine translation task on four WMT datasets in this embodiment, where RenewNAT is the model of this embodiment, and + indicates the result achieved by this embodiment. (A large number of studies have shown that knowledge distillation is of great help to non-autoregressive neural machine translation models. This embodiment also performs sentence-level knowledge distillation on all datasets. First, an original "transformer" model is used to train the original data, and then the trained model is used to generate the training data to obtain the translation result. The obtained translation result replaces the original real result, and then the model of this embodiment is trained.)

[0104] At the same time, this embodiment also conducts a large number of ablation experiments to verify the effectiveness of each module of the model of this embodiment.

[0105] Number of layers of masked language model decoding module

[0106] In the updateable non-autoregressive machine translation model, this embodiment divides the decoder into two sub-modules, wherein the non-autoregressive translation decoding module and the masked language model decoding module have NK layers and K layers respectively. This embodiment selects different values ​​of K and conducts experiments under the conditions of N=6 and N=12 respectively. The experimental results show that according to the given N, the selection of different K values ​​has a great influence on the final effect. The experimental results are shown in Figure 4 shown.

[0107] Masked language model decoding module training strategy

[0108] This embodiment experiments with different training strategies for the masked language model decoding module. First, this embodiment uses the CMLMC model to replace the basic masked language model for the masked language model decoding module, but the experimental results show that the effect improvement is not obvious. At the same time, in terms of the input of the masked language model decoding module, this embodiment compares three different input options, the real target words, the output words of the non-autoregressive translation decoding module, and a mixture of the two. Finally, this embodiment selects a mixture of the two as input in the model, because this embodiment believes that it can provide useful information in the potential translation, which is more in line with the reasoning process. The experimental results are as follows. Figure 5 shown.

[0109] Finding Error Location Strategies

[0110] After the first non-autoregressive translation decoding module generates possible translation results, this embodiment adopts different strategies to mask the erroneous positions. First, this embodiment sets a fixed ratio and uses this ratio to mask the position with the lowest confidence value. This embodiment finds that the performance is best when the ratio is between 0.2 and 0.5, but it is obviously not the best method because the proportion of errors in each potential translation is different. Therefore, this embodiment adopts another simple and efficient method to set a confidence threshold α1 to select the wrong position. The experimental results of this embodiment show that the best α value is related to the model. On the WMT14 dataset, the original non-autoregressive neural machine translation model and the GLAT model both achieve the best results when α1 is between 0.6 and 0.7.

[0111] Original dataset effect

[0112] Previous experiments were conducted on data that had undergone knowledge distillation. This example wants to verify whether our method is effective on the original data. This example conducts experiments on the IWSLT14DE-EN dataset that has not undergone data distillation. The experimental results show that the updateable non-autoregressive machine translation model improves the Bleu score by 1.42 on the basic model, which proves that the updateable non-autoregressive machine translation model is robust. Figure 6 shown.

[0113] Enter the sentence length

[0114] This example also explores the effect of input sentence length on the model. This example divides the original input length in the test data set into five intervals and calculates the Bleu score in each interval. The experimental results show that the model of this example surpasses the basic model in each interval, especially when the sentence length is between 20 and 40, the updateable non-autoregressive machine translation model achieves excellent results. The experimental results are shown in Figure 2. Figure 7 shown.

[0115] like Figure 8 As shown, this embodiment provides two specific machine translation examples, which are from the test data set of WMT14DE→DN. From Example 1 (Example 1), this embodiment can find that the masked language model decoding module of GLAT and the updateable non-autoregressive machine translation model generates results containing repeated words, while the updateable non-autoregressive machine translation model containing the masked language model decoding module updates the potential results and then gives a fluent sentence without repeated words. In addition, there are some spelling errors on GLAT and the updateable non-autoregressive machine translation model without the masked language model decoding module, as shown in Example 2 (Example 2), where "sun-hunghungers" and "hungey ry" are typical errors. The updateable non-autoregressive machine translation model successfully fixes this spelling error and makes the sentence more fluent. In short, although the results generated by the updateable non-autoregressive machine translation model still contain some errors, the method of this embodiment can actually alleviate the problems of non-autoregressive neural machine translation, such as repeated markings, spelling errors, etc., and generate more fluent and meaningful translation results.

[0116] Of course, in the experimental comparison, the mask based on the formula method still has a good effect. Due to space limitations, this embodiment will not be described in detail.

[0117] Embodiment 2

[0118] This embodiment provides a non-autoregressive machine translation method, including:

[0119] Step S1: Serialize and encode the source sequence X in sequence to obtain an encoded sequence;

[0120] Step S2: Taking the serialized source sequence X as the reference, the coded sequence is decoded to obtain the sequence Y with noise pot ;

[0121] Step S3: For the noisy sequence Y potMask the error information in to obtain a mask sequence;

[0122] Step S4: using the encoding sequence as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result;

[0123] Step S5: Compare the prediction result with the noisy sequence Y pot The unmasked information in is combined to obtain the translation result.

[0124] Embodiment 3

[0125] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the non-autoregressive machine translation method described in Embodiment 2 are implemented.

[0126] Embodiment 4

[0127] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the non-autoregressive machine translation method described in the second embodiment are implemented.

[0128] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiment, complete software embodiment, or the embodiment in combination with software and hardware. Moreover, the present application can adopt the form of the computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can adopt various computer languages ​​to realize, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0133] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A non-autoregressive machine translation system, characterized in that: include: Encoder module: used to serialize and encode the source sequence X in sequence to obtain the encoded sequence; Decoding module: used to decode the coded sequence based on the serialized source sequence X to obtain the sequence Y with noise pot ; Mask module: used to mask the noisy sequence Y pot Mask the error information in to obtain a mask sequence; Prediction module: used for using the encoding sequence as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result; Combination module: used to combine the prediction result with the noisy sequence Y pot Combine the unmasked information in to obtain the translation result; The mask module also includes a search module, which is used to search for the noisy sequence Y during inference. pot When masking the error information in , find the location of the error information; The method used in the search module to search for the position of the error information includes a threshold method and a formula method; The threshold method includes: Get the noisy sequence Y pot The confidence level of each mark in Compare the confidence of each identification with a first preset threshold α1, if the identification is higher than the first preset threshold α1, then the identification is regarded as correct information; if the identification is lower than the first preset threshold α1, then the identification is regarded as wrong information; The formula method satisfies: Wherein, mean(confidence) represents the average confidence of all identifiers in the current generated sentence, α2 is the second preset threshold, when mean(confidence)>α2, scale is set to α2, and the identifier with confidence less than α2 in the current generated sentence is used as error information; when mean(confidence)≤α2, scale is set to the average confidence value mean(confidence), and the identifier with confidence less than mean(confidence) in the current generated sentence is used as error information.

2. The non-autoregressive machine translation system according to claim 1, characterized in that: The decoding module comprises a non-autoregressive translation decoding module, and the non-autoregressive translation decoding module is used to decode the encoding sequence; The formula of the non-autoregressive translation decoding module is: in, represents the hidden state of the last layer of the non-autoregressive translation decoding module, Dec 1:N-k represents the non-autoregressive translation decoding module, represents the 1st to NKth decoder layers of the non-autoregressive translation decoding module, Y pot represents the noisy sequence output by the non-autoregressive translation decoding module, H represents a soft copy or uniform copy of the word embedding representation from the source sequence X, and E X represents a coding sequence; The loss formula of the non-autoregressive translation decoding module during training is: Among them, T represents the total length of the correct target sequence during training, and y t1 is a noisy sequence Y pot The corresponding correct result, X is the source sequence, and θ is the model parameter.

3. The non-autoregressive machine translation system according to claim 2, characterized in that: The prediction module includes a masked language model decoding module, and the masked language model decoding module is used to predict the masked error information in the mask sequence; The formula of the masked language model decoding module is: Among them, E Y Represents the hidden state of the last layer of the masked language model decoding module, Dec N-k+1:N represents the masked language model decoding module, represents the N-K+1th layer to the Nth layer of the masked language model decoding module, Y' represents the mask sequence, and emb(·) is the word embedding function that maps discrete words to vector representations; The loss formula of the masked language model decoding module during training is: Among them, y t2 represents the correct result corresponding to the masked part in the correct target sequence during training, Y obs represents the unmasked part of the correct target sequence during training, X represents the source sequence, and θ represents the model parameters.

4. The non-autoregressive machine translation system according to claim 3, characterized in that: The total loss formula generated by the non-autoregressive translation decoding module in the decoding module and the masked language model decoding module in the prediction module during the training process is: L = L pot +L mlm +L len , where L len Represents the length loss of predicted translation results during training.

5. The non-autoregressive machine translation system according to claim 4, characterized in that: It also includes an optimization module: used to optimize the model parameter θ by the total loss L during training, the formula is: Among them, γ represents the learning rate, and ← represents replacing the original model parameter symbol with the new model parameter.

6. A non-autoregressive machine translation method, characterized in that: include: Step S1: Serialize and encode the source sequence X in sequence to obtain an encoded sequence; Step S2: Taking the serialized source sequence X as the reference, the coded sequence is decoded to obtain the sequence Y with noise pot ; Step S3: For the noisy sequence Y pot Mask the error information in to obtain a mask sequence; Step S4: using the encoding sequence as an attention mechanism to predict the masked error information in the mask sequence to obtain a prediction result; Step S5: Compare the prediction result with the noisy sequence Y pot Combine the unmasked information in to obtain the translation result; The step S3 also includes the step of performing the inference on the noisy sequence Y pot When masking the error information in , find the location of the error information; The method used to find the position of the error information includes a threshold method and a formula method; The threshold method includes: Get the noisy sequence Y pot The confidence level of each mark in Compare the confidence of each identification with a first preset threshold α1, if the identification is higher than the first preset threshold α1, then the identification is regarded as correct information; if the identification is lower than the first preset threshold α1, then the identification is regarded as wrong information; The formula method satisfies: Wherein, mean(confidence) represents the average confidence of all identifiers in the current generated sentence, α2 is the second preset threshold, when mean(confidence)>α2, scale is set to α2, and the identifier with confidence less than α2 in the current generated sentence is used as error information; when mean(confidence)≤α2, scale is set to the average confidence value mean(confidence), and the identifier with confidence less than mean(confidence) in the current generated sentence is used as error information.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the non-autoregressive machine translation method according to claim 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the non-autoregressive machine translation method according to claim 6 are implemented.

Citation Information

Patent Citations

  • Sound signal processing apparatus, sound signal processing method, and program

    CN102610227A

  • Knowledge-enhanced non-autoregressive neural machine translation method and device

    CN114611488A

  • NER-oriented Chinese clinical text data enhancement method and device

    CN114861600A