A knowledge fusion method for machine translation

Abstract

The invention discloses a knowledge fusion method applied to machine translation, and the steps are: using a bilingual corpus to train one or more neural machine translation models as knowledge providers, that is, teacher models; translating the source language S in the corpus and extracting the knowledge of the teacher model; performing knowledge fusion; copying the source language S and splicing it, and finally obtaining the corpus after knowledge fusion; training the neural machine translation model used finally; decoding the source sentences in the bilingual corpus, firstly using the same sentence splicing method for the corresponding source sentences, and using separators to separate them, and then sending them into the neural machine translation model finally trained for decoding, and then separating the decoded results, and the separation mark is the separator used when the sentence is spliced, so as to realize knowledge fusion. The invention solves the problem that knowledge often carries many errors, and can integrate the knowledge of the teacher model and the knowledge in the real corpus to form a knowledge-rich corpus.

Description

Technical Field

[0001] The present invention relates to a neural machine translation technology, specifically a knowledge fusion method, which can improve the performance of a trained model. Background Art

[0002] Neural network machine translation usually adopts an encoder-decoder structure to achieve modeling of variable-length input sentences. The encoder realizes "understanding" of the source language sentence and forms a floating-point vector of a specific dimension. Then the decoder generates the translation result of the target language word by word based on this vector. In the early stage of the development of neural network machine translation, recurrent neural network (RNN) was widely used as the network structure of encoder and decoder. This network is good at modeling natural language. RNN networks represented by long short-term memory networks (LSTM) and gated recurrent unit networks (GRU) "remember" more important words in the sentence through the gating mechanism, so that the "memory" can be preserved for a longer time. In 2017, two works successively proposed the use of convolutional neural network (CNN) and self-attention network (Transformer) as encoder and decoder structures. They not only greatly surpassed the RNN-based neural network in translation effect, but also achieved improved training efficiency through parallelization during training. Currently, the mainstream framework for machine translation in the industry uses the self-attention network (Transformer), which is not only used in machine translation, but also has outstanding performance in fields such as self-supervised learning.

[0003] The achievements of neural machine translation are closely related to the unique network structure and algorithms of neural networks. The model architecture of neural machine translation is an encoder-decoder framework, which is characterized by using an encoder to convert the input text sequence into a vector containing text information, and then using a decoder to convert this vector into the target language. The encoder and decoder are mainly composed of different intermediate network layers. Commonly used intermediate layers include recurrent neural networks, convolutional neural networks, and attention-based networks. Among them, convolutional neural networks share convolution kernels and can efficiently process high-dimensional data. In machine translation, they can be used to extract features, while recurrent neural networks can be used to describe the output state that is continuous in time. They have memory functions and fit the characteristics of text sequences. They can be used by the decoder to decode the target file in sequence.

[0004] The early encoder-decoder framework used a convolutional neural network to encode the source language sequence, and a recurrent neural network to convert the model intermediate vector into the target language. The successful application of the attention mechanism in neural machine translation has brought machine translation to a new level. The attention mechanism imitates the human translation process. When decoding, the model does not treat all words the same as the recurrent neural network, but assigns different weights to different words. Words with high relevance to the words being translated at the current moment will receive higher attention weights. It can be seen that the introduction of the attention mechanism alleviates the shortcomings of the recurrent neural network.

[0005] In the actual process, whether it is machine translation or other tasks based on deep neural networks, they are constantly simulating human behavior, and are constantly innovating and breaking through. This is the case in the process of neural machine translation. Imagine that if there is a model that first learns the knowledge of the corpus it has, and then borrows the knowledge learned by this model, it may be able to help other models learn better.

[0006] Although there are some knowledge fusion methods, such as fusing syntax trees and word alignment information, they have the following shortcomings:

[0007] Knowledge is difficult to obtain, such as syntax tree information. When obtaining it, a large amount of corpus is often required for training and extraction, which causes certain losses in manpower and material resources;

[0008] Knowledge may carry a large amount of error propagation and error bias. For example, the quality of word alignment is often not very good, which greatly brings certain difficulties to the downstream knowledge fusion process.

[0009] Knowledge fusion methods are complex; existing knowledge fusion methods are relatively complex and difficult to implement, and often require professionals to design them skillfully.

[0010] At present, there is no report on the knowledge fusion method of applied machine translation that can solve the above problems. Summary of the invention

[0011] In view of the problem of limited knowledge during training in existing machine translation, which may lead to insufficient model learning, the present invention provides a knowledge fusion method applied to machine translation, so that the model can draw on the knowledge learned by other models and then help itself to learn, so as to obtain better performance, and the knowledge learned in the training process of the model is more sufficient.

[0012] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0013] The present invention provides a knowledge fusion method applied to machine translation, comprising the following steps:

[0014] 1) Use the bilingual corpus to train one or more neural machine translation models as knowledge providers, i.e. teacher models;

[0015] 2) Using the trained teacher model to translate the source language S in the corpus, extracting the teacher model knowledge using the method of translating the source language S in the corpus, the extracted result is the carrier T' of the teacher model knowledge;

[0016] 3) Use sentence concatenation to fuse knowledge. First, concatenate the carrier T' of the teacher model knowledge and the real target sentence T, adding a separator in the middle to separate them. Then copy the source language S and use the same sentence concatenation method to concatenate them. Finally, obtain the corpus S after knowledge fusion. joint and T joint ;

[0017] 4) Use the corpus S after knowledge fusion joint and T joint To train the neural machine translation model that is ultimately used, the training includes learning the knowledge in the real corpus and the knowledge learned by the teacher model;

[0018] 5) Decode the source sentences in the bilingual corpus. First, use the same sentence concatenation method to concatenate the corresponding source sentences and separate them with delimiters. Then send them to the final trained neural machine translation model for decoding. After that, separate the decoded results. The delimiter marks are used as the delimiters used in sentence concatenation to achieve knowledge fusion.

[0019] In step 1), a bilingual corpus is used to train a teacher model. A teacher model is obtained by training with the existing corpus, which is expressed as:

[0020] TeacherModel=ModelTrain(S,T)

[0021] ModelTrain is a model training function, which means training a single model or integrating multiple models together.

[0022] Step 2) Let the source language set in the bilingual corpus be S = (s 0 ,s 1 ,s 2 ,......,s m ), the target language set is T = (t 0 ,t 1 ,t 2 ,......,t m );Then use the teacher model to translate the source language S, and the corresponding sentence translated is T'=(t'0 ,t 1 ',t' 2 ,......,t' m ), taking T' as the carrier of teacher model knowledge;

[0023] Step 3) Knowledge fusion is performed by sentence concatenation. The specific steps are as follows:

[0024] 301) First, the corresponding target sentence translated by the teacher model and the real target are concatenated and separated by a separator to obtain the final target sentence T after integrating knowledge. joint ;

[0025] 302) Copy the source sentence and concatenate it using the same sentence concatenation method, also separated by separators, to obtain the source sentence S that integrates the knowledge joint .

[0026] Step 4) By using the bilingual corpus S that integrates knowledge joint and T joint To train the final neural machine translation model, during the model training process, it learns the knowledge of real bilingual corpus and also learns the knowledge learned by the teacher model, making the information it learns more sufficient and diverse.

[0027] In step 5), the final neural machine translation model trained in step 5) is used for decoding. The specific steps are as follows:

[0028] 501) copy the sentence to be translated, separate it with a delimiter, and then send it to the final neural machine translation model trained in step 4) for decoding;

[0029] 502) Perform a delimiting operation on the decoding result to find out whether there is a delimiter in the decoding result. If so, extract the part after the delimiter as the final decoding result. If not, use the entire complete decoding result as the final decoding result.

[0030] The present invention has the following beneficial effects and advantages:

[0031] 1. The method of the present invention can obtain the knowledge to be integrated in a simple way. Through simple conventional training means of machine translation, the neural machine translation model is trained using the bilingual corpus data to act as a teacher model in the technical solution, providing the knowledge to be integrated for the neural machine translation model to be used in the final training.

[0032] 2. The method of the present invention solves the problem that knowledge often contains many errors, and uses a more powerful integrated teacher model to obtain high-quality knowledge, which contains less false information and false bias.

[0033] 3. The method of the present invention proposes a knowledge fusion method that is easy to operate and implement in time, and uses a simple sentence splicing method to fuse knowledge, which can effectively integrate the knowledge of the teacher model and the knowledge in the real corpus to form a knowledge-rich corpus.

[0034] 4. The method of the present invention is also very flexible. For example, the teacher model can adopt an integrated model with relatively strong performance, and multiple strategies such as Beam Search and Gray Search can be used in the process of translating the teacher model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the overall process of the present invention;

[0036] Figure 2 This is a diagram of the knowledge fusion process of the target sentence of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings.

[0038] The present invention provides a knowledge fusion method using machine translation, the specific process is as follows Figure 1 As shown, the following steps are included:

[0039] 1) Use a bilingual corpus to train one or more neural machine translation models as knowledge providers, i.e. teacher models.

[0040] 2) Using the trained teacher model to translate the source language S in the corpus, extracting the teacher model knowledge using the method of translating the source language S in the corpus, the extracted result is the carrier T' of the teacher model knowledge;

[0041] 3) Use sentence concatenation to fuse knowledge. First, concatenate the carrier T' of the teacher model knowledge and the real target sentence T, adding a separator in the middle to separate them. Then copy the source language S and use the same sentence concatenation method to concatenate them. Finally, obtain the corpus S after knowledge fusion. joint and T joint ;

[0042] 4) Use the corpus S after knowledge fusion joint and T joint To train the neural machine translation model that is ultimately used, the training includes learning the knowledge in the real corpus and the knowledge learned by the teacher model;

[0043] 5) Decode the source sentences in the bilingual corpus. First, use the same sentence concatenation method to concatenate the corresponding source sentences and separate them with delimiters. Then send them to the final trained neural machine translation model for decoding. After that, separate the decoded results. The delimiter marks are used as the delimiters used in sentence concatenation to achieve knowledge fusion.

[0044] In step 1), a teacher model is trained using the bilingual corpus owned by the user. A teacher model is obtained by training with the existing corpus, which is expressed as:

[0045] TeacherModel=ModelTrain(S,T)

[0046] ModelTrain is a model training function, which means training a single model or integrating multiple models together.

[0047] During the training process, in order to improve the performance of neural machine translation, position encoding and multi-head attention are used to capture different relationships between sentences and bilingual sentence pairs in different representation spaces, so that the model can learn knowledge in more dimensions; specifically:

[0048] 101) Calculate the position encoding of each word and add it to the word embedding vector to generate the input vector required by the model. The calculation method is as follows:

[0049]

[0050]

[0051] PE(.) is the position encoding function, pos is the position of the current word, i represents the dimension in the position encoding vector, d model It is a basic parameter of the translation model, indicating the hidden layer size at each position. ;

[0052] 102) The model vectors Q, K, and V are mapped into h subsets (h heads), and point multiplication and concatenation operations are performed on them. The specific calculation is as follows:

[0053]

[0054] MultiHead(Q,K,V)=Concat(head 1 ,…,head h )W o

[0055] head i is the vector of the i-th head, W is the translation model parameter, Attention(.) is the attention mechanism function, and Concat(.) is the vector concatenation function;

[0056] 103) Use a fully connected network to map the vector representation after the attention mechanism operation to the new space. The specific implementation is as follows:

[0057] FFN(x)=max(0,xW 1 +b 1 )W 2 +b 2

[0058] Among them, W 1 , W 2 , b 1 , b 2 are the parameters of the translation model, which are automatically learned by the model.

[0059] 104) Train the model using the corresponding cross entropy loss function;

[0060] 105) Test the model and use the BLEU value as a measurement indicator; and here you can also train multiple teacher models and use model integration technology to obtain a teacher model with better performance.

[0061] In step 2), let the source language set in the bilingual corpus be S = (s 0 ,s 1 ,s 2 ,……,s m ), the target language set is T = (t 0 ,t 1 ,t 2 ,……,t m );Then use the teacher model to translate the source language S, and the corresponding sentence translated is T'=(t' 0 ,t 1 ',t' 2 ,......,t' m ), taking T' as the carrier of teacher model knowledge; the specific steps are:

[0062] 201) Extracting the source language from the bilingual corpus;

[0063] 202) In order to improve the diversity and difference of languages, the extracted source languages ​​are randomly masked, where the probability interval of the mask is [0, 0.1].

[0064] 203) Use the teacher model in step 1) to translate the processed source language. During the translation process, Beam Search or Gray Search or other decoding methods can be used.

[0065] In step 3), knowledge fusion is performed by sentence concatenation, and the specific steps are as follows:

[0066] 301) First, the corresponding target sentence translated by the teacher model and the real target are concatenated and separated by a separator to obtain the final target sentence T after integrating knowledge. joint ;

[0067] 302) Copy the source sentence and concatenate it using the same sentence concatenation method, also separated by separators, to obtain the source sentence S that integrates the knowledge joint .

[0068] In step 4), by using the bilingual corpus S with fusion knowledge joint and T joint To train the final NMT model, during the model training process, it learns the knowledge of real bilingual corpus and also learns the knowledge learned by the teacher model, making the information it learns more sufficient and diverse.

[0069] In step 5), the final neural machine translation model trained in step 4) is used for decoding, and the final decoding result is extracted, such as Figure 2 As shown, the specific steps are as follows:

[0070] 501) Copy the sentence to be translated and use " <split>" delimiter, and then sent to the final neural machine translation model trained in step 4) for decoding.

[0071] 502) Separate the decoding result and find out whether there is " <split>"Symbol, if present, extract" <split>"The part after is used as the final decoding result. If it does not exist, the entire decoding result is used as the final decoding result.

[0072] In this embodiment, the source sentences in the bilingual corpus are decoded. First, the corresponding source sentences are concatenated using the same sentence concatenation method and separated by a separator symbol " <split>" is separated and then sent to the final trained neural machine translation model for decoding. The decoded results are then separated and marked as " <split>", in the actual test process, it was found that for all the test solutions, the decoded" <split>"The performance of the corresponding result behind is better than that of the corresponding result in front, so extract" <split>The result after " is taken as the corresponding final decoding result.

[0073] For example, when translating the Chinese sentence "You are a student." into English, the actual target sentence is "You area student.", and the teacher model translates it as "You were a student.". Then the concatenated result is "You area student. <split>You were a student.". You need to use the concatenation method to construct the corresponding source sentence, such as "You are a student." The result of concatenating this source sentence is "You are a student. <split>You are a student. ”

[0074] The present invention uses a sentence concatenation method to fuse the knowledge in the real corpus and the knowledge of the pre-trained teacher model, and provides the final fusion result to the neural machine translation to be used as the training corpus, so that it can not only learn knowledge from the real corpus, but also learn knowledge from the teacher model, thereby improving the performance of the final training result of the neural machine translation.

[0075] The present invention uses a sentence concatenation method and proposes a knowledge fusion method for machine translation. The present invention experiments are conducted on two translation tasks: IWLST14 German-English and NIST12 Chinese-English. The experimental results are shown in the following table. It can be seen that compared with the baseline model, the model after knowledge fusion has a significant speed improvement.

[0076]

[0077]

[0078] The present invention also randomly extracts the knowledge of the teacher model in the experiment to test its knowledge quality. The test results of NIST12 Chinese-English translation are shown in the following table. It can be seen that the knowledge carrier generated by the teacher model has less error information and higher quality.

[0079] < / split> < / split> < / split> < / split> < / split> < / split> < / split> < / split> < / split>

Claims

1. A knowledge fusion method applied to machine translation, characterized in that The following steps are involved: 1) Use the bilingual corpus to train one or more neural machine translation models as knowledge providers, i.e. teacher models; 2) Use the trained teacher model to train the source language in the corpus S Translate the source language in the corpus S The translation method is used to extract teacher model knowledge, and the extracted result is the carrier of teacher model knowledge. T′ ; 3) Use sentence concatenation to integrate knowledge. First, the carrier of teacher model knowledge T′ and the actual target sentence T Splice and add separators in the middle to separate them; then copy the source language S , add a separator in the middle to separate, and finally obtain the corpus after knowledge fusion S joint and T joint ; 4) Using the corpus after knowledge fusion S joint and T joint To train the neural machine translation model that is ultimately used, the training includes learning the knowledge in the real corpus and the knowledge learned by the teacher model; 5) Decode the source sentences in the bilingual corpus. First, use the same sentence concatenation method to concatenate the corresponding source sentences and use separators to separate them. Then, send them to the final trained neural machine translation model for decoding. After that, separate the decoded results. The separator marks are used as the separators used in sentence concatenation to achieve knowledge fusion. In step 5), the final neural machine translation model trained in step 4) is used for decoding. The specific steps are as follows: 501) Copy the sentence to be translated and separate it with a delimiter, and then send it to the final neural machine translation model trained in step 4) for decoding; 502) Perform a delimiting operation on the decoding result to find out whether there is a delimiter in the decoding result. If so, extract the part after the delimiter as the final decoding result. If not, use the entire decoding result as the final decoding result.

2. The knowledge fusion method for machine translation according to claim 1, characterized in that: In step 1), a teacher model is trained using a bilingual corpus. A teacher model is obtained by training with the existing corpus, which is expressed as: TeacherModel = ModelTrain ( S , T ) in ModelTrain It is a model training function, which means training a single model or integrating multiple models together.

3. The knowledge fusion method for machine translation according to claim 1, characterized in that: Step 2) Assume that the source language set in the bilingual corpus is S = ( s 0, s 1, s 2,…… s m ), the target language set is T =( t 0, t 1, t 2,…… t m ); then use the teacher model to translate the source language S , the corresponding sentence translated is T′ =( t′ 0, t′ 1, t′ 2,…… t′ m ),Bundle T′ As a carrier of teacher model knowledge.

4. The knowledge fusion method for machine translation according to claim 1, characterized in that: Step 3) Knowledge integration is performed through sentence concatenation. The specific steps are as follows: 301) First, concatenate the corresponding target sentence translated by the teacher model and the real target, and separate them with a delimiter to obtain the final target sentence after integrating knowledge. T joint ; 302) Copy the source sentence and separate it with a separator to obtain the source sentence with integrated knowledge S joint .

5. The knowledge fusion method for machine translation according to claim 1, characterized in that: Step 4) By using bilingual corpora that integrate knowledge S joint and T joint To train the final neural machine translation model, during the model training process, it learns the knowledge of real bilingual corpus and also learns the knowledge learned by the teacher model, making the information it learns more sufficient and diverse.

Images