A neural machine translation method based on feature attenuation screening

Abstract

The present application relates to a kind of neural machine translation method based on feature attenuation screening, belong to natural language processing machine neural translation field.This method is for the translation task of source language to target language, first using parallel data training a reverse translation model of target language to source language, then using the translation model to target language monolingual data set is reversed, obtain preliminary pseudo-parallel data set.According to the source language text of parallel data set, sentence feature set is created, and initial score is assigned to each sentence feature, according to feature attenuation algorithm, pseudo-data set is iteratively selected, and screened pseudo-data set is obtained.Synthetic data set is constructed by combining screened pseudo-data set with parallel data set, and finally using synthetic data set trains the neural machine translation model of source language to target language.This method avoids the adverse effects of poor quality pseudo-data on model training, effectively improves the translation effect of model.

Description

TECHNICAL FIELD

[0001] The application relates to a neural machine translation method based on feature attenuation screening and belongs to the technical field of machine translation in natural language processing. BACKGROUND

[0002] Since the birth of computers, people have always hoped to find a way to enable computers to understand human language. The development of natural language processing (NLP) technology makes this idea possible. Machine translation (MT) as an important branch of natural language processing aims to transform one language into another language through certain transformation and keep the semantic information unchanged.

[0003] Most machine translation systems are based on neural networks, and with the help of large-scale parallel data and deep learning technology, neural machine translation has achieved results comparable to human translation in some translation tasks. However, for most languages in the world, it is difficult to construct large-scale parallel data sets. At present, using data enhancement methods is an effective way to solve the problem of insufficient data, among which back translation (BT) is relatively effective and has been widely used in various low-resource neural machine translation methods.

[0004] However, due to the quality of the back translation model, the quality of the generated pseudo data is uneven, and since the amount of pseudo data is considerable, it directly affects the final translation model, so there is an urgent need for an effective method to remove the poor quality part of the pseudo data, improve the quality of the pseudo data, and enhance the translation effect of the translation model. SUMMARY

[0005] The purpose of the application is to solve the technical problem of poor translation effect caused by poor quality of pseudo data of the back translation model of the neural machine translation method, and a neural machine translation method based on feature attenuation screening is proposed. The innovation of the method is that by applying a feature attenuation algorithm, the poor quality data in the back translation process is effectively screened and removed, and the translation effect of the final model is enhanced.

[0006] First, the relevant concepts are explained:

[0007] 1. Source language and target language

[0008] The source language represents the input language of the machine translation task, and the target language represents the output language of the machine translation task.

[0009] 2. Parallel data set D

[0010] This represents a dataset containing source language text and corresponding target language text, in the form: D = {(x1,y1),(x2,y2),…,(x...} i ,y i ),…,(x M ,y M )}, where the subscript M indicates that the dataset contains M data items, (x i ,y i () represents a data point, consisting of source language text x i and the corresponding target language text y i composition.

[0011] 3. Target Language Monolingual Dataset D y

[0012] A dataset consisting solely of target language text is represented in the form: D y ={y1,y2,…,y L The subscript L indicates that there are L data entries in the dataset.

[0013] 4. BPE (Byte Pair Encoding) is a byte-pair encoding technology.

[0014] This technology compresses data by continuously combining the most frequently occurring adjacent characters, thereby reducing the number of entries in the translation dictionary.

[0015] 5. Sentence Features

[0016] This refers to the n-gram model. The n-gram model divides text according to the number of words, where n is any positive integer. For example, in the sentence "I like to eat apples," "apple" is a 2-gram, and "eat apples" is a 3-gram. The 2-gram and 3-gram are used as sentence features.

[0017] 6. Transformer Translation Model

[0018] A baseline model for the translation system is constructed based on this method.

[0019] A neural machine translation method based on feature decay screening includes the following steps:

[0020] Step 1: Compare the parallel dataset D and the target language monolingual dataset D y Perform data preprocessing, including data cleaning, word segmentation, and BPE processing.

[0021] Specifically, the following methods can be used:

[0022] Step 1.1: clean the data using Moses scripts, including removing data with too large length difference between source language text and target language text (e.g., data with length difference between source language text and target language text exceeding 10), lowercasing the text, and punctuating the data (uniformly replacing punctuations with English punctuations), wherein Moses is an open-source preprocessing tool for machine translation.

[0023] Step 1.2: perform word segmentation on the data using an external open-source word segmentation tool.

[0024] Step 1.3: perform BPE processing on the data using an external open-source BPE tool.

[0025] Step 2: input the parallel data set D into a Transformer model to train a reverse translation model Model YX from the target language to the source language.

[0026] Step 3: use the reverse translation model Model YX to translate the monolingual data set D y in the target language into a data set D X in the source language, in the form of {x1, x2, …, x L}.

[0027] Step 4: concatenate the data in D x and D Y to construct a pseudo-parallel data set D syn in the form of D syn = {(x1, y1), (x1, y1), …, (x L , y L )}.

[0028] Step 5: apply a feature decay algorithm to filter the pseudo-parallel data set D syn obtained in Step 4.

[0029] Specifically, the following steps are included:

[0030] Step 5.1: perform sentence feature statistics on the source language text in the parallel data set D and sort the sentence features in descending order of quantity, taking the top K sentence features to construct a feature set F s .

[0031] Step 5.2: assign an initial score init(f) to each sentence feature in the feature set F s .

[0032] Step 5.3: traverse the pseudo-parallel data set D syn and score the source language text of each data, as follows:

[0033]

[0034] Where s represents the source language sentence, score(s) represents the score of the sentence, word_counts represents the number of words in s, and value(f) represents the score of the sentence feature f in the current iteration round. In the first iteration, value(f) = init(f).

[0035] Step 5.4: Transfer the pseudo-parallel dataset D syn All data are sorted in descending order of score. The top K highest-scoring data points are selected to construct the first iteration's data selection set S1. These K data points are then removed from the pseudo-parallel dataset D. syn Remove from the middle.

[0036] Step 5.5: Perform statistical analysis on the sentence features in the data selection set S1, and update the scores of the sentence features by decaying, as follows:

[0037] value(f) = init(f) 0.5 C(f) (2)

[0038] Here, C(f) represents the number of times sentence feature f appears in the selection set S1. The more times sentence feature f is selected, the more its score decreases.

[0039] Step 5.6: Iterate through steps 5.3 to 5.5 N times to obtain N data selection sets {S1, S2, ..., S...} N}, and merge them to obtain the filtered pseudo-dataset D. f .

[0040] Step 6: Filter the dataset D obtained in Step 5. f By merging with parallel dataset D, a synthetic parallel dataset D is obtained. final .

[0041] Step 7: Synthesize parallel dataset D final The source language is input into the Transformer model, which is used to train a neural machine translation model from the source language to the target language. The translation is then performed using this model.

[0042] Thus far, through steps 1 to 7, an initial pseudo dataset is first obtained through back translation, then the pseudo dataset is filtered using the feature decay method, and finally the filtered pseudo parallel dataset is added to the model for training. The resulting model has a further improvement in translation quality compared to the unfiltered method.

[0043] Beneficial effects

[0044] Compared with the prior art, the method has the following advantages:

[0045] The method aims at the problem of uneven quality of pseudo data in the method of enhancing the machine translation effect by using the reverse translation method, and by introducing the feature attenuation screening process, effectively eliminates the pseudo data with low quality, thereby reducing the adverse effects of noise data on model training, and improving the model translation effect. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The flowchart of the method of the present application. DETAILED DESCRIPTION

[0047] The method of the present application will be further described in detail below in conjunction with the drawings and examples of the specification.

[0048] EMBODIMENT

[0049] Taking the German-English data set and the English monolingual data set in the IWSLT2014 data set as an embodiment, the method of the present application will be described in detail with specific examples.

[0050] A neural machine translation method based on feature attenuation screening, as shown in Figure 1 The method comprises the following steps:

[0051] Step 1: Preprocess the parallel data set and monolingual data set data.

[0052] The IWSLT2014 German-English data set and the English monolingual data set are subjected to data cleaning and word segmentation, and a tool Moses script is used, and then BPE processing is performed. For example, the original data is ("was ist mit langlebigkeit?", "What about longevity?"), and after data cleaning, word segmentation and BPE processing, the processed result is ("was ist mitlang@@le@@big@@keit?", "what about lon@@ge@@vity?"). The processed German-English parallel data set is 150K, and the English monolingual data set is 130K.

[0053] Step 2: Train the reverse translation model.

[0054] The 150K German-English parallel data set is used to train the English-German reverse translation model, the model used is the Transformer-base model, and the parameter setting uses the recommended parameters of the original model.

[0055] Step 3: Construct a pseudo parallel data set.

[0056] The English monolingual dataset is back-translated using the English-to-German translation model obtained in Step 2, resulting in a German-English pseudo-parallel dataset of 130K.

[0057] Step 4: Data filtering.

[0058] First, the German data in the parallel dataset is sentence feature statistics, and the feature set is constructed. For example, for the sentence "was ist mit lang@@le@@big@@keit?", using 2-gram and 3-gram as sentence features, the sentence features are {“was ist”, “ist mit”, “mit lang@@”, “lang@@le@@”, “le@@big@@”, “big@@keit”, “keit?”, “was ist mit”, “ist mit lang@@”, “mit lang@@le@@”, “lang@@le@@big@@”, “le@@big@@keit”, “big@@keit?”}. After the sentence feature statistics of all data in the dataset, the top 5K sentence features are taken to constitute the feature set. Then traverse the pseudo-parallel dataset, score all sentences in the pseudo-parallel dataset according to the formula described in Step 5, and select the top 10K data with the highest score to join the filtered pseudo-parallel dataset. Iterate this process 6 times to get the filtered pseudo-parallel dataset of 60K.

[0059] Step 5: Construct a synthetic dataset.

[0060] The filtered pseudo-parallel dataset obtained in Step 4 is combined with the parallel dataset to form a synthetic dataset for training the final target translation model, totaling 210K data.

[0061] Step 6: Train the target translation model.

[0062] The synthetic dataset is used to train the source language to target language translation model.

Claims

1. A neural machine translation method based on feature decay filtering, characterized in that, The method comprises the following steps: Step 1: Data preprocessing is performed on the parallel dataset D and the target language monolingual dataset D y including data cleaning, word segmentation, and byte pair encoding technique (BPE) processing; Wherein, parallel dataset D represents a dataset containing source language text and corresponding target language text, in the form: D={(x1,y1),(x2,y2),…,(x i ,y i ),…,(x M ,y M )}, where the subscript M indicates that the dataset contains M data items, (x i ,y i () represents a data point, consisting of source language text x i and the corresponding target language text y i composition; Target Language Monolingual Dataset D y A dataset consisting solely of target language text is represented in the form: D y ={y1,y2,…,y L The subscript L indicates that there are L data entries in the dataset; Step 1.1: cleaning data using Moses scripts, including removing data with too large length difference between source language text and target language text, lowercasing text letters, and punctuating data, wherein Moses is an open source preprocessing tool for machine translation; wherein the data with too large length difference refers to data with length difference between source language text and target language text exceeding 10; Step 1.2: performing word segmentation on data using an external open source word segmentation tool; Step 1.3: performing BPE processing on data using an external open source BPE tool; Step 2: Input the parallel dataset D into the Transformer translation model, training to obtain a reverse translation model Model from the target language to the source language YX ; The source language represents the input language of the machine translation task, and the target language represents the output language of the machine translation task; Step 3: Use the back-translation model Model YX， to translate the target language monolingual dataset D y into a source language form dataset D X in the form {x1, x2,..., x L} Step 4: D x with D Y corresponding data stitching, constructing a pseudo-parallel dataset D syn , in the form of: D syn = {(x1,y1),(x1,y1),…,(x L ,y L )}. Step 5: Apply a feature decay algorithm to the pseudo-parallel dataset D syn Screening; comprising the steps of: Step 5.1: Perform sentence feature statistics on the source language text in the parallel data set D, and sort them in descending order of quantity, take the top K sentence features to construct the feature set F s ; The sentence feature is a n-gram model, which divides the text according to the number of words, and n is any positive integer; Step 5.2: for each sentence feature in the feature set F s assign an initial score init(f); Step 5.3: Traverse the pseudo-parallel dataset D syn Score the source language text for each piece of data as follows: Wherein s represents a source language sentence, score(s) represents the score of the sentence, word_counts represents the number of words in s, and value(f) represents the score of the sentence feature f in the current iteration round, value(f)=init(f) in the first iteration; Step 5.4: Arrange all data of the pseudo-parallel dataset D syn in descending order of scores, select the top K data with the highest scores to construct the data selection set S1 of the first iteration, and remove these K data from the pseudo-parallel dataset D syn . Step 5.5: counting the sentence features in the selection set S1 and updating the score of the sentence feature: value(f) = init(f)0 5 C(f) (2) Wherein C(f) represents the number of times that the sentence feature f appears in the selection set S1. Step 5.6: iterate the above steps 5.3 to 5.5 N times to obtain N sets of data selection {S1, S2, …, SN}, and combine them to obtain the screened pseudo-data set D N . f ​ Step 6: The filtered dataset D obtained in step 5 is combined with the parallel dataset D to obtain a synthetic parallel dataset D f final ;​ Step 7: Synthesizing parallel dataset D final In an input Transformer model, a neural machine translation model is trained from a source language to a target language, and the model is used for translation.

Images