A method for constructing a multi-domain machine translation system based on pre-training guidance

Abstract

The present invention discloses a method for constructing a multi-domain machine translation system based on pre-training guidance. The steps are as follows: obtaining an open-source pre-trained model through a website; obtaining an open-source dataset through a website and cleaning it; encoding the source language text that needs to be translated currently using the pre-trained model; encoding the context of the text that needs to be translated currently using the pre-trained model; extracting features from the encoding results of the context using a convolutional neural network; processing the encoding results of the source language text and the feature extraction results of the context using an enhanced attention mechanism; using a Transformer decoder to receive the results processed by the enhanced attention mechanism and generate the target translation; training the machine translation system using the dataset to achieve the construction of the system. The present invention enhances the translation effect of the machine translation system on texts in a wide range of fields, reduces the training cost of the translation model, improves the quality of the generated translation, and approaches the "smoothness" and "elegance" levels in the translation field.

Description

Technical Field

[0001] The present invention relates to a machine translation technology, specifically a method for constructing a multi-domain machine translation system based on pre-training guidance. Background Art

[0002] At the present stage, the machine translation system based on neural network is one of the best in terms of translation effect, and most of such systems belong to the "encoding-decoding" framework. This framework divides translation into two steps, which are completed by two model structures respectively. In the first step, "encoding" is to encode and analyze the text to be translated, which is completed by the encoder in the machine translation system, and the result analyzed by the encoder is sent to the decoder; in the second step, "decoding" is to generate the target translation based on the analysis result transmitted by the encoder, which is completed by the decoder.

[0003] Pre-training is one of the directions with breakthrough progress in the field of natural language processing in recent years. The main idea of this direction is to train a model with a very large number of parameters (such as GPT3 has 175 billion parameters) without considering downstream tasks using a huge amount of monolingual corpus (for example, the training data of the pre-trained model GPT3 has 0.5 trillion words). Let this model be able to learn some general knowledge and have a stronger text analysis ability than ordinary models. Since the data that the pre-trained model "has seen" is in many fields, its encoding and analysis ability for texts in various fields is very strong.

[0004] One of the main functions of pre-training is to analyze and encode text, which can guide the downstream machine translation model in many aspects, including encoding the source language text to be translated currently and encoding the context of the text to be translated currently. During the encoding process of the pre-trained model, the knowledge learned in each field during the training stage of the pre-trained model will be incorporated. Therefore, the pre-trained model is very suitable as an encoder in the machine translation system to encode the source language text and the context.

[0005] As one of the ultimate tasks to achieve seamless communication between computers and humans, the machine translation task is extremely difficult. Its goal is to generate a text in the target language based on a given source language text through machine analysis and calculation. The generated target language text is required to be at least grammatically correct and not change the meaning of the source language text. On the basis of trying to meet the requirement of "faithfulness" (i.e., no partial translation or omission), "expressiveness" (not being restricted by the form of the original text, and the translation is smooth and clear), and "elegance" (pursuing elegance and appropriateness of the translation) are taken as the ultimate translation goals. At present, machine translation can only reach the stage of "faithfulness" in specific fields, such as news translation, instruction manual translation, weather forecast translation, etc., while in other fields, the translation effect is often not good, and there is a large gap compared with the translation quality in specific fields. The main reason for the low universality of machine translation systems is that not every field has enough bilingual data for training, and the performance of machine translation systems is highly dependent on training data. Even when a machine translation system is well-trained in a certain field, it is still likely to perform poorly in fields with insufficient training. Summary of the Invention

[0006] In view of the deficiencies in the prior art, such as the large difference in performance of machine translation systems in different fields, the technical problem to be solved by the present invention is to provide a method for constructing a multi-domain machine translation system based on pre-training guidance that can enable the machine translation system to achieve good translation effects in multiple fields.

[0007] To solve the above technical problem, the technical solution adopted by the present invention is:

[0008] The present invention provides a method for constructing a multi-domain machine translation system based on pre-training guidance, including the following steps:

[0009] 1) Obtain an open-source pre-trained model trained on a large-scale corpus through a website;

[0010] 2) Obtain an open-source dataset for training the machine translation system through a website and clean it to obtain a clean dataset that can be directly used for training the machine translation system;

[0011] 3) Encode the source language text that needs to be translated currently using the pre-trained model;

[0012] 4) Encode the context of the text that needs to be translated currently using the pre-trained model;

[0013] 5) Use a convolutional neural network to extract features from the results of encoding the context by the pre-trained model;

[0014] 6) Use an enhanced attention mechanism to process the results of encoding the source language text in step 3) and the results of feature extraction from encoding the context in step 5) (such as Figure 2as shown);

[0015] 7) Use the Transformer decoder to receive the processing result of the enhanced attention mechanism and generate the target translation;

[0016] 8) Use the dataset cleaned in step 2) to train the complete machine translation system to build a multi-domain machine translation system.

[0017] In step 2), obtain the open-source dataset for training the machine translation system through the website and clean it, including filtering special characters, filtering overly long sentences, length ratio filtering, removing HTML tags, and language detection.

[0018] In step 3), use the pre-trained model to encode the source language text that needs to be translated currently. First, input the source language text into the pre-trained model, and then add up the calculation results of each layer structure in the pre-trained model as the result of the pre-trained model encoding the source language text.

[0019] In step 4), use the pre-trained model to encode the context of the text that needs to be translated currently. First, input the context of the source language text into the pre-trained model, and then add up the calculation results of each layer structure in the pre-trained model as the result of the pre-trained model encoding the processed source language text context. Specifically:

[0020] 401) Add the first symbol in the middle of the above and below of the source language text to separate the above and below;

[0021] 402) Let the fixed input length of the pre-trained model be N, then use the second symbol to pad the context content of the source language text. Let the length of the padded context be M, then M satisfies the following conditions, where k is a positive integer, that is, the pre-trained model needs to be encoded k times to complete the encoding of all contexts:

[0022] M = kN, k ∈ {1, 2, 3...}

[0023] 403) Input the processed context into the pre-trained model, and the pre-trained model performs calculations. Add up the calculation results of each layer structure in the pre-trained model as the result of the pre-trained model encoding the processed source language text context.

[0024] In step 5), use the convolutional neural network to extract features from the result of the pre-trained model encoding the context, which is to compress the context encoding result into a vector of the same length as the pre-trained model encoding the source language text. Use the convolutional neural network to compress the context encoding result as follows:

[0025] 501) First, concatenate the k results encoded by the pre-trained model to form a result with a length of kd, where d represents the length of the result generated by encoding once with the pre-trained model;

[0026] 502) Use a convolutional neural network to extract features from the concatenated result x, while retaining the keyword features, reduce the result x with a length of kd to a context representation y with a length of d, satisfying the following conditions:

[0027] y = CNN(x)

[0028] where CNN represents a convolutional neural network.

[0029] Step 6) Use an enhanced attention mechanism to process the source language text encoding result and the context encoding result of the translated text as follows: Adopt an enhanced attention mechanism, which is divided into two parts. The first part is the traditional attention mechanism, which selects the context information required for translation according to the source language text; the second part is the synthesis mechanism, which combines the source language text information with the context information, and the combined result is sent to the decoder of the machine translation system as the information of all the source language texts seen by the model. Specifically:

[0030] 601) The traditional attention mechanism uses the following formula for feature fusion. The encoding result t of the source language text by the pre-trained model is used as the Query of the traditional attention mechanism, and the encoding results y of the context by the pre-trained model are used as the Key and Value of the traditional attention mechanism, satisfying the following conditions:

[0031]

[0032] where Query represents the query content for attention calculation, Key and Value represent the content to be queried for attention calculation, and the dimensions of Query, Key, and Value are the same, and the dimension is represented by d k denotes, and Softmax represents the activation function; 602) The synthesis mechanism uses the following formula for feature fusion. The calculation result Attention(t, y, y) of the traditional self-attention mechanism passes through a feed-forward neural network (FNN), and then is added to the encoding result t of the source language text by the pre-trained model to obtain the information z of all the source language texts seen by the model, satisfying the following conditions:

[0033] z = t + FFN(Attention(t, y, y)).

[0034] Step 7) Use the Transformer decoder to receive the processing result of the enhanced attention mechanism: Adopt the same decoder structure as the Transformer. The decoder consists of many layers, and the structure of each layer is the same and is composed of Self-Attention, Cross-Attention, and FNN. Use the information z of all source language texts in the calculation result of step 6) as the input of Cross-Attention. The Self-Attention and FNN parts of the decoder are the same as those of the Transformer.

[0035] The present invention has the following beneficial effects and advantages:

[0036] 1. The present invention innovatively introduces a pre-trained model with extensive general knowledge in the field of natural language processing into the machine translation system, enabling the original machine translation system to perform well even in under-trained fields; the machine translation system of the present invention has a wider application range and also reduces the training cost of the translation model.

[0037] 2. The present invention not only uses the pre-trained model to encode the source language text, but also uses the pre-trained model to encode the context of the source language text, so that the decoder can not only see the source language text but also the context information when generating the translation, thereby deepening the model's understanding of the meaning expressed by the source language text and further improving the quality of the generated translation, approaching the "smoothness" and "elegance" levels in the translation field.

[0038] 3. The design concept of the present invention can also be applied to other sequence-to-sequence systems, such as human-machine dialogue. Only by adapting and adjusting the enhanced attention mechanism proposed by the present invention for relevant tasks can it be applied to other tasks, expanding the application scope. Brief Description of the Drawings

[0039] Figure 1 It is a diagram of the overall process of the present invention;

[0040] Figure 2 It is a diagram of the enhanced attention mechanism in the present invention. Detailed Embodiment

[0041] As Figure 1 shown, the present invention provides a method for constructing a multi-domain machine translation system based on pre-training guidance, including the following steps:

[0042] 1) Obtain an open-source pre-trained model trained on a large-scale corpus through a website;

[0043] 2) Obtain an open-source dataset for training the machine translation system through a website and clean it to obtain a clean dataset that can be directly used for training the machine translation system;

[0044] 3) Encode the source language text that needs to be translated currently using a pre-trained model;

[0045] 4) Encode the context of the text that needs to be translated currently using a pre-trained model;

[0046] 5) Use a convolutional neural network to extract features from the result of encoding the context by the pre-trained model;

[0047] 6) Use an enhanced attention mechanism to process the result of encoding the source language text in step 3) and the result of feature extraction from encoding the context in step 5);

[0048] 7) Use a Transformer decoder to receive the processing result of the enhanced attention mechanism and generate the target translation;

[0049] 8) Use the cleaned dataset in step 2) to train the complete machine translation system to build a multi-domain machine translation system.

[0050] In step 1), select an open-source pre-trained model that is allowed by the hardware conditions and trained on a large-scale corpus on the Hugging Face website, such as BERT.

[0051] In step 2), select an open-source dataset provided in the machine translation competition and perform cleaning including the following five steps: filtering special characters, filtering overly long sentences, length ratio filtering, removing tags such as HTML, and language detection.

[0052] In step 3), first input the source language text into the pre-trained model, and then add up the calculation results of each layer structure in the pre-trained model as the result of encoding the source language text by the pre-trained model.

[0053] In step 4), use the pre-trained model to encode the context of the text that needs to be translated currently. First, input the context of the source language text into the pre-trained model, and then add up the calculation results of each layer structure in the pre-trained model as the result of encoding and processing the context of the source language text. Specifically:

[0054] 401) Add the first symbol in the middle of the above and below the source language text <cla>”, used to separate the above text and the following text; when the source language text has both the above text and the following text, for example, if the above text is "What is your favourite food" and the following text is "Because it can make me sleep better", then a symbol "" needs to be added between the above text and the following text. <cla>”, becomes "What is your favourite food <cla>Because it can make me sleep better”。

[0055] 402) Let the fixed input length of the pre-trained model be N, then use the second symbol " <pad>"Fill in the context content of the source language text. Let the length of the filled context be M, and M satisfies the following conditions, where k is a positive integer, that is, the pre-trained model needs to be encoded k times to complete the encoding of all contexts:

[0056] M = kN, k ∈ {1, 2, 3...}

[0057] Therefore, in the above example, assuming N is 5 (the actual input length of the pre-trained model may be 512 or even longer), and the merged context "What is your favourite food" <cla>Because it can make me sleep better” length is 13, so 2 "" need to be filled <pad>” becomes "What is your favourite food <cla>Because it can make me sleep better <pad> <pad>”.

[0058] 403) Input the processed context into the pre-trained model. The pre-trained model performs calculations and extracts the calculation results of each layer structure in the pre-trained model as the result after the pre-trained model encodes and processes the source language text context.

[0059] Before inputting into the pre-trained model, the context content needs to be segmented. For the sentence in the above example, it will be segmented into "What is your favourite food", " <cla>Because it can make”、"me sleep better <pad> <pad>”. Then, the segmented sentences are sequentially fed into the pre-trained model for encoding.

[0060] In step 5), since the length of the source language text context itself is not fixed, the length of the result encoded by the pre-trained model for the context is also not fixed. At this time, the context encoding result needs to be compressed into a vector with the same length as the source language text encoded by the pre-trained model. This system uses a convolutional neural network to compress the context encoding result, as follows:

[0061] 501) First, splice the k results encoded by the pre-trained model to form a result with a length of kd, where d represents the length of the result generated by the pre-trained model for one encoding.

[0062] 502) Use a convolutional neural network (also known as CNN) to extract features from the spliced result x. While retaining important features, the result x with a length of kd is reduced in dimension to a context representation y with a length of d, satisfying the following condition:

[0063] y = CNN(x)

[0064] where CNN represents a convolutional neural network.

[0065] In step 6), an enhanced attention mechanism is adopted, which is divided into two parts. The first part is the traditional attention mechanism, whose purpose is to enable the model to select the context information required for translation according to the source language text; the second part is the synthesis mechanism, whose purpose is to combine the source language text information with the context information, and the combined result is sent to the decoder of the machine translation system as the information of all the source language texts seen by the model. Specifically:

[0066] 601) The traditional attention mechanism is as follows. This mechanism is part of the Transformer model invented by the Google team in 2017. Its input consists of three parts: Query, Key, and Value. Generally, Key and Value are the same content. Its core idea is to extract valuable information from Value according to Query. Specifically, first, find the most similar place in the Key matrix to Query by taking the dot product of Query and Key. In other words, determine the position in the Key matrix where the useful information in Key can be obtained based on Query. Then, the result of multiplying Query and Key is passed through the Softmax function to obtain the weight matrix, and Value is multiplied by the weight matrix to obtain the information extracted from Value according to Query. This mechanism can achieve the purpose of expanding the important information in Value and shielding the noise information in Value.

[0067] In the present invention, since it is necessary to translate the source language text, the encoding result t of the source language text by the pre-trained model is used as the Query of the traditional attention mechanism. At the same time, the encoding result y of the pre-trained model for the context is used as the Key and Value of the traditional attention mechanism, satisfying the following conditions:

[0068]

[0069] Among them, Query represents the query content for attention calculation, Key and Value represent the content to be queried for attention calculation, and Query, Key, and Value have the same dimension, and the dimension is represented by d k Softmax represents the activation function;

[0070] The purpose of doing this is to be oriented by the source language text to find the information in the context that is useful for the model to analyze the source language text;

[0071] 602) The synthesis mechanism is as follows. The calculation result Attention(t, y, y) of the traditional self-attention mechanism is passed through a feed-forward neural network (FNN), and then added to the encoding result t of the pre-trained model for the source language text to obtain the information z of all the source language texts seen by the model, satisfying the following conditions:

[0072] z = t + FFN(Attention(t, y, y))

[0073] The role of FNN is to adjust the fusion ratio of the context information and the source language text information. It not only prevents the context information from being too high and directly covering up the source language text information, but also prevents the context information from being too little to help the model translate the source language text.

[0074] In step 7), the same decoder structure as that of the Transformer is adopted. The decoder contains many layers, and the structure of each layer is the same and consists of Self-Attention, Cross-Attention, and FNN. In the present invention, the calculation result of step 6) - the information z of all the source language texts is used as the input of Cross-Attention, and the Self-Attention and FNN parts of the decoder are the same as those of the Transformer. The following is an illustration with a specific translation example.

[0075] The present invention first reads the source language text to be translated and its context. For example, the source language text is "I like it" and the context is "Because coffee keeps me awake". These two sentences are sent to the machine translation system. The system will first run the pre-trained model to encode the source language text and save the encoded result. Then the pre-trained model is run to encode the context and save the encoded result. Next, the system will run the enhanced attention mechanism and send the result processed by the enhanced attention mechanism to the decoder. After receiving the information, the decoder generates a translation. Since the pre-trained model has strong text analysis capabilities and the enhanced attention mechanism also introduces contextual information, the present invention can have strong translation capabilities and can translate "I like it" into "I like coffee" instead of "I like it".

[0076] To address the shortcomings of existing machine translation systems in their domain adaptability, the present invention proposes a method for constructing a multi-domain machine translation system based on pre-training guidance. First, a powerful pre-trained model from the natural language processing field is used to encode the source text and context. Then, an improved enhanced attention mechanism is used to fuse the source text and context information. The fused result is fed to a decoder for analysis and translation. This method improves the translation capabilities of machine translation systems in under-trained domains. By utilizing a powerful pre-trained model from the natural language processing field, pre-training is used to encode the source text and context. Because the pre-trained model has been trained on extensive and massive amounts of data, it possesses strong text analysis and encoding capabilities. Therefore, pre-training allows the machine translation system to "observe" deeper meaning in the source text. The present invention also innovatively proposes an enhanced attention mechanism to incorporate contextual information of the translated text into the model, thereby enabling the model to "see" a wider range of the source text.< / pad> < / pad> < / cla> < / pad> < / pad> < / cla> < / pad> < / cla> < / pad> < / cla> < / cla> < / cla>

Claims

1. A method for constructing a multi-domain machine translation system based on pre-training guidance, characterized in that It includes the following steps: 1) Obtain an open-source pre-trained model trained on a large-scale corpus through a website; 2) Obtain an open-source dataset for training a machine translation system through a website and clean it to obtain a clean dataset that can be directly used for training the machine translation system; 3) Use the pre-trained model to encode the source language text that needs to be translated currently; 4) Use the pre-trained model to encode the context of the text that needs to be translated currently; 5) Use a convolutional neural network to extract features from the result of encoding the context by the pre-trained model; 6) Use an enhanced attention mechanism to process the result of encoding the source language text in step 3) and the result of feature extraction of encoding the context in step 5); 7) Use a Transformer decoder to receive the result processed by the enhanced attention mechanism and generate the target translation; 8) Use the dataset cleaned in step 2) to train the complete machine translation system to build a multi-domain machine translation system; In step 4), when using the pre-trained model to encode the context of the text that needs to be translated currently, first input the context of the source language text into the pre-trained model, and then add the calculation results of each layer structure in the pre-trained model as the result after encoding and processing the context of the source language text by the pre-trained model. Specifically: 401) Add a first symbol in the middle of the above and below texts of the source language text to separate the above and below texts; 402) Assume that the fixed input length of the pre-trained model is N, then use a second symbol to pad the context content of the source language text. Assume that the length of the padded context is M, then M satisfies the following conditions, where k is a positive integer, that is, the pre-trained model needs to encode k times to complete the encoding of all contexts: M = kN, k ∈ {1, 2, 3…} 403) Input the processed context into the pre-trained model, and the pre-trained model performs calculations. Add the calculation results of each layer structure in the pre-trained model as the result after encoding and processing the context of the source language text by the pre-trained model; In step 5), when using a convolutional neural network to extract features from the result of encoding the context by the pre-trained model, it is to compress the context encoding result into a vector with the same length as the encoding of the source language text by the pre-trained model. Use a convolutional neural network to compress the context encoding result, specifically as follows: 501) First, splice the k results after encoding by the pre-trained model to form a result with a length of kd, where d represents the length of the result generated by encoding the pre-trained model once; 5) Use a convolutional neural network to extract features from the spliced result x. While retaining the keyword features, reduce the result x with a length of kd to a context representation y with a length of d, satisfying the following conditions: y = CNN(x) where CNN represents a convolutional neural network.

2. The method for constructing a multi-domain machine translation system based on pre-training guidance according to claim 1, wherein: In step 2), obtaining an open-source dataset for training a machine translation system through a website and cleaning it includes filtering special characters, filtering overly long sentences, length ratio filtering, removing HTML tags, and language detection.

3. The method for constructing a multi-domain machine translation system based on pre-training guidance according to claim 1, wherein: Step 3) Encode the source language text that needs to be translated currently using the pre-trained model. First, input the source language text into the pre-trained model, and then add up the calculation results of each layer structure in the pre-trained model as the result of the pre-trained model encoding the source language text.

4. The method for constructing a multi-domain machine translation system based on pre-training guidance according to claim 1, wherein: Step 6) Process the source language text encoding result and the context encoding result of the translated text using the enhanced attention mechanism as follows: The enhanced attention mechanism is divided into two parts. The first part is the traditional attention mechanism, which selects the context information required for translation according to the source language text. The second part is the synthesis mechanism, which combines the source language text information with the context information, and the combined result is sent to the decoder of the machine translation system as the information of all the source language texts seen by the model. Specifically: 601) The traditional attention mechanism performs feature fusion using the following formula. The encoding result t of the pre-trained model for the source language text is used as the Query of the traditional attention mechanism, and the encoding results y of the pre-trained model for the context are used as the Key and Value of the traditional attention mechanism, satisfying the following conditions: Among them, Query represents the query content for attention calculation, Key and Value represent the content to be queried for attention calculation, and Query, Key, and Value have the same dimension, which is represented by d k denotes, and Softmax represents the activation function; 602) The synthesis mechanism performs feature fusion using the following formula. The calculation result Attention(t, y, y) of the traditional self-attention mechanism is passed through a feed-forward neural network (FNN), and then added to the encoding result t of the pre-trained model for the source language text to obtain the information z of all source language texts seen by the model, satisfying the following conditions: z = t + FFN(Attention(t, y, y)).

5. The method for constructing a multi-domain machine translation system based on pre-training guidance according to claim 1, wherein Step 7) Use the Transformer decoder to receive the processing result of the enhanced attention mechanism as follows: Adopt the same decoder structure as the Transformer. The decoder contains many layers, and the structure of each layer is the same and consists of Self-Attention, Cross-Attention, and FNN. Use the information z of all the source language texts in the calculation result of Step 6 as the input of Cross-Attention. The Self-Attention and FNN parts of the decoder are the same as those of the Transformer.

Images