Multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables

Abstract

The present invention proposes a multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables. Domain adaptation can effectively solve the problem of poor translation performance in specific domains. Existing methods usually rely on mixing data from multiple domains to obtain a single general multi-domain neural machine translation model. However, the mixing of large-scale general domain data with data from multiple specific domains will cause problems such as catastrophic forgetting and inter-domain parameter interference in the final model, resulting in performance degradation and the contradiction of insufficient capacity of the general domain model. Therefore, the present invention uses the Gumbel‑Softmax reparameterization technique to simultaneously learn model parameters and latent variables. The obtained model can learn knowledge in specific domains and share general domain knowledge through latent variables. Experimental results show that in multi-domain neural machine translation, the present invention improves the BLEU values ​​by an average of 3.2 and 1.68 in English-German and Chinese-English multi-domain neural machine translation compared with the baseline model, respectively.

Description

Technical Field

[0001] The present invention relates to a multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables, and belongs to the technical field of natural language processing. Background Art

[0002] Neural machine translation relies on deep learning and uses an end-to-end approach to translate the source language into the target language. Among them, domain neural machine translation is one of the important research directions. In current neural machine translation models, the effect of translating general domains is often better, but it usually performs poorly when translating texts in new domains with different styles or vocabulary. This is called the catastrophic forgetting problem. Not only that, the amount of general domain data available for training is much larger than the amount of specific domain data, which is particularly evident in Chinese and English data. To this end, the present invention uses the Transformer Base and Big models to conduct preliminary experiments on each domain on Chinese and English data. The experimental results in Table 1 show that general domains require more model parameters than specific domains to achieve the same translation performance, reflecting the contradiction between the insufficient capacity of general domain models and the redundancy of specific domain model parameters. In addition, the problem of model parameter interference caused by domain transfer is also common in multi-domain neural machine translation.

[0003] Table 1 Preliminary experimental results of various fields on Chinese and English datasets

[0004] General fields paper Spoken educate Transformer 18.68 14.38 15.22 15.43 Transformer-big 20.74 14.59 15.22 15.73

[0005] To address these issues, many researchers have conducted their own explorations. First, domain-based neural machine translation aims to establish a unified model. By mixing data from multiple domains and jointly training rich and low-resource domains, it can effectively mine resource-rich domain information and improve performance in multiple specific domains, thereby accurately translating terminology and text styles from different domains. Second, to address the catastrophic forgetting problem caused by fine-tuning, Chu et al. proposed a hybrid fine-tuning method, in which they fine-tuned a general-domain model using a mixture of general-domain and oversampled in-domain data. Kobus et al. minimized the loss of domain information by adding domain labels to each sentence. Meanwhile, Britz et al. established relationships between domains and extracted common features by adding an identifier. Furthermore, Khayrallah et al. attempted to keep model parameters close to their original values ​​by adding regularization terms. Furthermore, Bapna and Firat et al. added a domain-specific adapter module to the general-domain model and fine-tuned the adapter using domain data. Unfortunately, as the number of domains increases, the model parameters of the above methods will increase dramatically. Once the model parameters are reduced, the performance of some domains will decline, especially for general domains. Moreover, as the number of domains increases, the parameter interference problem between domains becomes more serious. Summary of the Invention

[0006] To address the above problems, the present invention proposes a multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables. This method not only solves the contradiction between the insufficient capacity of general domain models and the parameter redundancy of specific domain models, but also maintains good translation performance in multiple domains.

[0007] The technical solution of the present invention is a multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables. The specific steps of the multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables are as follows:

[0008] Step 1: Download the training dataset from the machine translation competition website and use tools to perform data preprocessing operations such as word segmentation and byte pairing.

[0009] Step 2: In the framework of the Transformer model, use the "Pre-LN" (forward normalization) operation to insert domain-specific sub-layer latent variables and use the Gumbel-Softmax reparameterization technique to simultaneously learn model parameters and latent variables;

[0010] Step 3. Finally, a unified multi-domain neural machine translation model is obtained by jointly training the general domain and specific domain data controlled by domain labels.

[0011] As a preferred embodiment of the present invention, the specific steps of Step 1 are:

[0012] Download general and domain datasets from websites such as WMT, CCMT, and UM-Corpus, and use sentencespiece, Stanford NLP, and moses tokenizer tools to perform word segmentation and byte pair processing on the data to complete data preprocessing;

[0013] As a preferred embodiment of the present invention, the specific steps of Step 2 are:

[0014] First, the present invention uses "Pre-LN" (forward normalization) instead of "Post-LN" (backward normalization) on the Transformer architecture during training to improve model stability;

[0015] Secondly, domain-specific sub-layer latent variables are incorporated to control whether the model skips the current sub-layer. When the latent variable value is 0, the current sub-layer is skipped, and when the latent variable value is 1, the current sub-layer is selected. To this end, the present invention introduces a posterior probability distribution to approximate the true distribution of the latent variables and introduces variational inference to solve for the domain-specific sub-layer latent variables.

[0016] Finally, we sample from the approximate posterior probability distribution and use the Gumbel-Softmax reparameterization technique to simultaneously learn the model parameters and latent variables.

[0017] The beneficial effects of the present invention are:

[0018] 1. The present invention first adds learnable domain-specific sub-layer latent variables to the encoder and decoder layers of the Transformer, which determines whether the non-residual connection blocks in the current sub-layer participate in parameter updates, thereby reducing redundant parameters of the specific domain model and maintaining the capacity of the general domain model; secondly, the Gumbel-Softmax reparameterization technique is used to train the model parameters and the domain-specific sub-layer latent variables; finally, a mixture of general corpus and domain-specific corpus data with added domain labels is used for joint training to obtain a unified multi-domain neural machine translation model. Experimental results show that the method proposed in the present invention can effectively improve the performance of the multi-domain neural machine translation model on English-German and Chinese-English public datasets;

[0019] 2. The domain-specific sub-layer latent variables incorporated into the present invention can effectively solve the parameter interference problem by learning specific domain knowledge and sharing public domain information, prevent parameter redundancy in specific domains, maintain the model capacity of general domains, and avoid catastrophic forgetting problems in general domains, thereby improving the performance of multi-domain neural machine translation models.

[0020] 3. The method proposed in this paper is simpler than traditional multi-domain neural machine translation. Experimental results show that the BLEU value of this method is generally improved compared with the baseline system. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall flow chart of the present invention;

[0022] Figure 2 This is a Transformer model diagram that incorporates domain-specific sub-layer latent variables in the present invention; DETAILED DESCRIPTION

[0023] Example 1: Figure 1-Figure 2 As shown in Figure 1, a multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables is proposed. To test the effectiveness of the proposed method in multi-domain adaptive neural machine translation tasks, experiments were conducted on English-to-German and Chinese-to-English translation, and various baseline models were compared. The specific steps of the multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables are as follows:

[0024] Step 1: For English-to-German translation, we used the WMT14 news translation task as the general-domain training corpus, and newstest2013 and newstest2014 as the validation and test sets, respectively. The experiment was conducted in three different domains: TED Talks, biomedicine, and fiction. For the TED Talks domain, we used IWSLT14 as the training corpus, dev2010 and tst2014 as the validation and test sets, respectively. For the biomedicine domain, we used the EMEA News Crawl dataset for training and evaluation, and the Khresmoi Medical Abstract Translation Test Data 2.0 as the validation and test sets. For the fiction domain, we used the OPUS book dataset as the training set, and randomly selected chapters from Jane Eyre and The Metamorphosis as the validation and test sets.

[0025] For Chinese-to-English translation, the experiment uses the WMT17 news translation task dataset as the general-domain training corpus, newsdev2017 and newstest2017 as the validation set and test set, respectively, and selects data from three fields, namely papers, spoken language, and education, from UM-Corpus as the specific-domain experimental corpus.

[0026] The statistical information of the dataset is shown in Table 2.

[0027] The present invention performs text preprocessing operations on English-German and Chinese-English respectively. On the English-German dataset, the sentencespiece word segmentation tool is used to segment the data. Since English and German belong to the West Germanic language branch, a shared vocabulary operation can be used, wherein the size of the jointly learned vocabulary is 32768. On the Chinese-English dataset, the Stanford NLP and moses tokenizer word segmentation tools are used to segment Chinese and English respectively. However, due to the large difference between the two languages, the shared vocabulary operation is not performed. Byte Pair Encoding (BPE) is used to segment Chinese and English respectively into subwords, and the resulting vocabulary sizes are 44K and 33K respectively.

[0028] Table 2 Dataset statistics

[0029]

[0030] Due to the differences between the two experimental data sets, English and German both belong to the West Germanic branch, so shared vocabulary operations are performed on the English-German data set. However, for Chinese and English, the two languages ​​are very different, so shared vocabulary operations are not performed. For English to German, the present invention uses the sentencepiece tool to segment the data, where the size of the shared dictionary is 32,768. For Chinese to English, the present invention uses the Stanford NLP Segmenter and Moses tokenizer tools to segment Chinese and English, respectively, and Truecase them, and uses byte pair encoding (BPE) for encoding, respectively, where the Chinese and English dictionary sizes are 44K and 33K, respectively.

[0031] During inference, the beam search was set to 4 for English-German and Chinese-English, and the length penalty was set to 0.6 for English-German and 1.0 for Chinese-English. Furthermore, we used the Sacre-BLEU tool to evaluate the performance of translation models in all experiments. For simplicity, we only report the BLEU results from the best multi-domain neural machine translation model.

[0032] Step 2. The experimental environment is Linux Ubuntu 20.04 system, PyTorch version is 1.7.1+cu110, and the compilation language is Python 3.7. The method proposed in the present invention is applied to the Transformer framework, in which the number of encoder and decoder layers is 6, the word embedding dimensions in the encoder and decoder and the hidden units of the feedforward network are 1024 and 4096 respectively, and the number of multi-head attention heads for self-attention and cross-attention are both set to 16. The learning rate during training is adjusted by using the Adam optimizer. In addition, in order to prevent overfitting problems, dropout is set to 0.1, and all models are trained on the NVIDIA RTX3090GPU with a global batch size of 4096 words. The experimental parameter settings are shown in Table 3.

[0033] Table 3 Experimental parameter settings

[0034] system Ubuntu 20.04 Pytorch version 1.7.1+cu110 Compiled Language Python 3.7 equipment 3090 Model Transformer encoder 6th floor Decoder 6th floor Word embedding dimension 1024 Feedforward network hidden units 4096 Self-attention multi-head attention head number 16 Cross-attention multi-head attention head number 16 Optimizer Adam Dropout 0.1 Each batch contains the number of words 4096

[0035] The Transformer model consists of a framework of multiple stacked network layers based on the encoder-decoder structure. The encoder consists of N identical layers, each with two sublayers. The first is a multi-head self-attention layer, and the second is a fully connected feedforward network layer. Both sublayers perform residual connection operations and layer normalization operations. The decoder also consists of N identical layers. In addition to the two sublayers in the encoder layer, the decoder also inserts a multi-head cross attention sublayer, which performs a multi-head attention operation on the output of the encoder. For the input sequence x = (x1, x2, ... x i), the output of each sub-layer in the encoder or decoder is LayerNorem(SubLayer+x), where SubLayer(·) represents a multi-head self-attention, multi-head cross-attention or feed-forward network module.

[0036] First, the architecture of the present invention uses "Pre-LN" (forward normalization) instead of "Post-LN" (backward normalization) during training to improve model stability. The output of each sub-layer becomes:

[0037] x′=x+SubLayer(LayerNorm(x)) (1)

[0038] It is worth noting that the present invention regards SubLayer(·) as a function like Figure 2 As shown, the present invention inserts a domain-specific sub-layer latent variable To control the function Whether to skip the current sub-layer. When the latent variable value is 0, the current sub-layer will be skipped. When the latent variable value is 1, the current sub-layer will be selected.

[0039] More specifically, for each layer L in a Transformer encoder or decoder stack, we consider the non-residual connected blocks as sublayers And hope to learn the domain-specific sub-layer hidden variables that can determine whether the current sub-layer is selected Right now:

[0040]

[0041] in is the control coefficient of the ith sublayer of the lth layer.

[0042] In order to learn the domain-specific sub-layer latent variables that obey the Bernoulli distribution proposed in the framework of this invention The present invention expresses the Ground-Truth sequence as and represents the translation as And the multi-domain translation task target p(y|x) is shared by the model parameters θ and the domain-specific sub-layer hidden variables The N fields represented are:

[0043]

[0044] in and

[0045]

[0046] Here, variational inference is introduced to solve the domain-specific sub-layer hidden variables The true distribution is p {y|x；θ} (z). For this purpose, a posterior probability distribution is also introduced To approximate the latent variable The true distribution of , and solve it using KL divergence:

[0047]

[0048] Further simplification and introduction of the evidence lower bound (ELBO) yield:

[0049]

[0050] In the experiment, due to the domain-specific sub-layer latent variables is a discrete value, the present invention can directly use the Gumbel-Softmax reparameterization technique to simultaneously learn the model parameters θ and latent variables First, the present invention approximates the posterior probability distribution This can make the neural network with discrete variables differentiable during back propagation. Secondly, in order to achieve the selection of sub-layers, the present invention generates If q(z) becomes discrete, the present invention extracts hard samples during inference, so that the present invention can directly learn the logits parameter α of the i-th sub-layer sl :

[0051]

[0052] in is a Gumbel distribution, τ is a hyperparameter, initialized to 5, and can be continuously learned during training.

[0053] Step 3. Finally, a unified multi-domain neural machine translation model is obtained by jointly training the general domain and specific domain data controlled by domain labels.

[0054] Table 4 Results of each model on the English-German dataset

[0055] Model General Corpus TED Talks Biomedical Sciences novel average Common Domain Model 28.70 28.50 28.40 14.50 25.03 Hybrid Domain Model 27.90 31.30 32.00 21.20 28.10 Fine-tuning - 31.50 29.70 23.40 28.20 Sequence trimming 28.40 31.90 30.10 23.60 28.50 Methods 31.02 32.71 33.04 24.93 30.43

[0056] In order to verify the effectiveness of the multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables proposed in this paper, we compared it with some classic methods and the most advanced state-of-the-art (SOTA) methods:

[0057] 1) General domain model: Only parallel corpus data from general domains is used to train the model.

[0058] 2) Mixed-domain model: Mix the general domain and all domain-specific data to train the model.

[0059] 3) Fine-tuning: First train the model with general domain data, and then fine-tune it with domain-specific data to continue training.

[0060] 4) Multiple-output Layer Learning (MLL): Adding domain-specific output layers to the general domain model and learning the parameters of these domain-specific output layers according to their respective learning objectives.

[0061] 5) Adapter: A domain adapter corresponding to a specific domain is added to each layer of the general domain model, which includes a normalization layer and two linear projection layers, and is fine-tuned using domain-specific data.

[0062] 6) Pruning Then Expanding (PTE): A pruned model is trained on a general domain, and then the pruned model is fine-tuned using domain-specific data and knowledge distillation, where the unpruned model serves as the teacher and the pruned model serves as the student. Finally, the pruned model is fine-tuned and expanded to its original size.

[0063] 7) Sequential Prune-Tune: Find the most informative parameters on the general domain model and freeze them, then prune unnecessary parameters, and finally fine-tune the domain-specific sub-model network using domain-specific data.

[0064] The results of each model on the English-German dataset and the Chinese-English dataset are shown in Table 4 and Table 5 respectively.

[0065] Table 5 Results of each model on Chinese and English datasets

[0066] Model General Corpus paper Spoken educate average Common Domain Model 20.74 12.60 16.47 17.85 16.92 Hybrid Domain Model 20.49 14.55 15.73 17.20 16.99 Fine-tuning 12.58 16.99 18.64 19.43 16.91 Multi-output layer learning 22.60 16.24 18.27 18.39 18.88 adapter 23.26 15.82 17.83 18.68 18.90 Trim and expand 23.78 16.85 18.69 19.55 19.72 Methods 24.19 19.90 17.71 23.79 21.40

[0067] Tables 4 and 5 show the results of multi-domain adaptive neural machine translation experiments conducted on public datasets in English and German and Chinese and English, respectively. Since the English-German and Chinese-English datasets in the referenced best model papers are different, they are compared with the method proposed in the present invention respectively. In addition, the results of the general domain model and mixed domain model of Chinese and English are missing in the best model, so the present invention supplemented them with relevant experiments. It can be found that on the English-German and Chinese-English datasets, the BLEU values ​​reported by the method proposed in the present invention are better than other comparison models in most fields. The results of multi-domain neural machine translation in English and Chinese and English can strongly demonstrate the effectiveness of the method proposed in the present invention.

[0068] Furthermore, the present invention draws the following conclusions: First, for most specific domains, the proposed method surpasses common methods in domain-adaptive neural machine translation, such as general domain models, hybrid domain models, fine-tuning, multi-output layer learning, adapters, pruning and expansion, and sequence pruning and tuning. This confirms the effectiveness of learning domain-specific sub-layer latent variables, and the proposed method can obtain a single general multi-domain neural machine translation model with good performance from large-scale general domain data and small-scale specific domain data.

[0069] Therefore, the present invention can know that the method of the present invention can effectively alleviate the parameter interference between multiple fields by learning knowledge in specific fields and sharing common field information, thereby improving the performance of the multi-field neural machine translation model. Secondly, compared with fine-tuning, the model of the present invention can effectively avoid the catastrophic forgetting problem in general fields. Although multi-output layer learning, adapters and pruning expansion can also solve this problem to a certain extent, the method proposed in the present invention is more effective. The fundamental reason is that the multi-field neural machine translation model can maintain the model capacity of the general domain and reduce the model parameter redundancy of the specific field by learning the domain-specific sub-layer hidden variables, thereby alleviating the performance degradation problem in the general field and improving the performance of multiple specific fields.

[0070] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables, characterized by: The specific steps of the method are as follows: Step 1: Download the training dataset and use tools to preprocess the data. Step 2: Based on the Transformer model, we use the "Pre-LN" forward normalization operation to insert domain-specific sub-layer latent variables and use the Gumbel-Softmax reparameterization technique to simultaneously learn model parameters and latent variables. Step 3: Finally, we perform translation by jointly training a unified multi-domain neural machine translation model using a mixture of general domain and specific domain data controlled by domain labels. The specific steps of Step 2 are: First, during training, "Pre-LN" forward normalization is used on the Transformer architecture instead of "Post-LN" backward normalization to improve model stability; Secondly, domain-specific sub-layer latent variables are inserted to control whether the model skips the current sub-layer. When the latent variable value is 0, the current sub-layer will be skipped, and when the latent variable value is 1, the current sub-layer will be selected. To this end, the posterior probability distribution is introduced to approximate the true distribution of the latent variable, and variational inference is introduced to solve the domain-specific sub-layer latent variables. Finally, we sample from the approximate posterior probability distribution and use the Gumbel-Softmax reparameterization technique to simultaneously learn the model parameters and latent variables.

2. The multi-domain adaptive neural machine translation method based on domain-specific sub-layer latent variables according to claim 1 is characterized by: The specific steps of Step 1 are: Download the general dataset and domain dataset from the machine translation competition website, and use the sentencespiece, StanfordNLP, and moses tokenizer tools to perform word segmentation and byte pair processing on the data to complete data preprocessing.

Images