A method for constructing and training reverse translation data based on domain classification

Abstract

A method for constructing and training reverse translation data based on domain classification, which relates to the fields of natural language processing, machine translation, and data augmentation. The method includes: training a bilingual model based on the original corpus: using the Transformer neural network model to train two neural machine translation models by taking the bilingual original corpus as the source - target end and the target - source end respectively; 2) constructing a domain monolingual dataset based on keyword and topic similarity; 3) generating a domain pseudo - parallel dataset based on reverse translation: taking the obtained domain monolingual dataset as the source - end dataset, using the trained bilingual model to perform reverse translation to obtain pseudo - sentence pairs, and forming a domain pseudo - parallel dataset after collation and alignment; 4) joint training based on domain - classification learning: marking the pseudo - parallel dataset as domain corpus and performing joint training with general data. It can obtain high - quality knowledge in relevant fields while effectively utilizing general - domain knowledge, improve translation performance, and ensure translation quality.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing, machine translation, and data augmentation, and particularly to a method for constructing and training reverse translation data based on domain classification. Background Art

[0002] In recent years, neural machine translation has become the mainstream model of machine translation methods. However, neural machine translation is highly dependent on data. Generally speaking, the larger the training data, the better the translation quality. For machine translation of a few languages, the lack of bilingual parallel corpora often becomes an important factor restricting its translation quality. Especially in machine translation of special fields, high-quality domain corpora are very scarce compared with a large amount of general corpora. Some data augmentation methods need to be adopted to improve the translation quality. A typical method is to use monolingual corpora for reverse translation to obtain pseudo-parallel corpora. Therefore, how to effectively obtain a large amount of high-quality monolingual corpora and ensure less quality loss in the reverse translation process to improve the machine translation model has become one of the research goals. On the other hand, compared with the huge monolingual general corpora, the monolingual data in special fields are even scarcer and the imbalance problem leads to the neglect of domain knowledge during the training process, and the translation effect of the translation model in the actual field is not good.

[0003] Back-translation is a commonly used method to improve the quality of training data using monolingual corpora. Applying the back-translation method to machine translation tasks with scarce corpora has been verified to be effective in different literatures. Among them, Edunov et al. (Sergey Edunov, et al. 2018. Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics) studied back-translation in different scenarios and proved that back-translation can improve BLEU.Sennrich et al. (Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational Linguistics.) used the monolingual data obtained by back-translation and mixed it with the original corpus and then added it to the training to improve the translation quality. At the same time, their further research proved that this synthetic data of the target monolingual language is very helpful for domain adaptation (Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Linguistics). In the construction of domain data, generally, the existing manually labeled domain data sets are directly adopted or the data sets are constructed by means of replacing with synonym similarity and sentence similarity. In the face of the sampling problem of general data and in-domain data, it is a common method to increase the domain-related weights and change the model training objectives.As Chen et al. (Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. 2017a. Cost weighting for neural machine translation domain adaptation. In Proceedings of the First Workshop on Neural Machine Translation, pages 40–46, Vancouver.) pre-trained a domain classifier and transferred the output probabilities into the translation model to replace the actual weights, while Wang et al. (Rui Wang, Masao Utiyama, Andrew Finch, Lemao Liu, Kehai Chen, and Eiichiro Sumita. 2018. Sentence selection and weighting for neural machine translation domain adaptation. IEEE / ACM Transactions on Audio, Speech, and Language Processing) performed joint training through sentence selection and domain weighting. However, these methods have some deficiencies: First, there is less domain data marked manually, and the substitution-based method is too mechanical and does not conform to the real distribution; in learning from domain and general datasets, a model lacking domain weights cannot solve the problem of unbalanced sample distribution and cannot learn domain knowledge well, while pre-training an additional classifier still requires a round of weight adjustment during the transfer process to adapt to the model. Summary of the Invention

[0004] The object of the present invention is to provide a method for constructing and training reverse translation data based on domain classification for the problem of low translation quality caused by lack of corpus during the translation of minority languages, especially in special domains. The method constructs pseudo-parallel corpora from documents using the method of domain topic similarity to enhance the domain dataset, and improves the domain adaptation ability through joint training of domain classification, enhances the domain recognition ability and generalization performance of the model. On the one hand, it can mine high-quality domain monolingual data faster and better, and on the other hand, it enables the model to better learn domain knowledge, thereby enhancing the translation quality of the neural translation model for domain translation.

[0005] The present invention includes the following steps:

[0006] 1) Training a bilingual model based on the original corpus: Using the Transformer neural network model, train two neural machine translation models by taking the bilingual original corpus as the source - target end and the target - source end respectively.

[0007] 2) Constructing a domain - specific monolingual dataset based on keyword and topic similarity;

[0008] 3) Generating a domain - specific pseudo - parallel dataset based on back - translation: Taking the domain - specific monolingual dataset obtained in step 2) as the source - end dataset, using the trained bilingual model in step 1) to perform back - translation to obtain pseudo - sentence pairs, and forming a domain - specific pseudo - parallel dataset after collation and alignment;

[0009] 4) Joint training based on domain - classification learning: Marking the pseudo - parallel dataset obtained in step 3) as domain - specific corpus and mixing it with general data for joint training.

[0010] In step 1), the specific steps of training the bilingual model based on the original corpus can be as follows:

[0011] The language model is based on the neural network model of open - source Transformer. This model is an encoder - decoder model based on self - attention and consists of four modules: input, encoding, decoding, and output. The input module reads the input text sequence Input = {s1, s2, s3, …, st} from the training file, where st represents the i - th word, obtains the initial word embedding and connects it with the positional encoding to get the input vector Input Embedding = {x1, x2, …, xt}, and then sends it to the encoder. Each layer of the encoder consists of a multi - head attention mechanism, a feed - forward neural network, a normalization layer, and a residual connection. Among them, the core model is the multi - head attention mechanism evolved from the attention mechanism, and the expression is:

[0012]

[0013] After the encoder obtains the abstract feature representation of the input, it is input into the decoder for decoding. The decoder has basically the same structure as the encoder and, conversely, performs decoding by connecting the target embedding, the encoder state, and the decoder state. The decoding result passes through a linear layer and a softmax layer to output the probability distribution of the overall label. For the machine - translation bilingual model, the commonly used training loss function is the cross - entropy loss. Based on the use of this model, after pre - processing the well - organized high - quality original bilingual dataset S, such as alignment, tokenization, BPE, statistical vocabulary, and training - set segmentation, train a unidirectional model distributively. Based on the constructed bilingual model, train it with the original corpus to obtain two bilingual models;

[0014] In step 2), the specific steps of constructing a domain - specific monolingual dataset based on keyword and topic similarity can be as follows:

[0015] In the relevant chapters of the field, the dataset is expanded through document topic similarity to avoid the inefficiency of sentence-by-sentence selection; for any field, the frequently occurring keywords within the field can represent the theme of this field, but some pronouns, personal names and other meaningless words need to be excluded, which can be constructed by the automatic acquisition method of the program. The specific construction method is as follows:

[0016] (1) Construct a basic vocabulary. Use the python toolkit to perform word frequency statistics on the existing field dataset, obtain the first 500 common words and then manually screen out a vocabulary, filter out some common words, and obtain a basic vocabulary of size 256 as the field theme representative D = [d1, d2,..., dn], where di represents the i-th word;

[0017] (2) Extract the theme of the document. It is similar to the above process and can also be obtained by word frequency statistics; for any article P = [p1, p2, p3..., pn] in the document library, pi represents the i-th sentence; perform content selection to extract the important sentences of the document, providing a mixture of two extraction methods. One is extraction based on the main structure. Generally speaking, the important parts of a document are mainly concentrated in the title, the first paragraph and the end, and these parts are extracted according to the convention; the other is to select the sentences containing certain informative keywords (such as "generally speaking") for extraction; the extraction by the two methods constitutes a subset P1 and satisfies the extraction total constraint. Generally, these sentences are filtered by word frequency statistics to obtain the article keyword representative set Ds = [dp1, dp2,..., dpn]; in this way, the topic similarity can be transformed into the similarity of two keyword sets representing the topics, and the similarity between the two can be calculated by the vector representation method;

[0018] (3) Use the BERT word vectors to transform D and Ds into vector representations, and sum all the word vectors to obtain the topic vector representation and

[0019]

[0020]

[0021] Use the Pearson correlation coefficient to represent the similarity between the two, which is used to judge the similarity between the text theme and the words in the vocabulary. The calculation formula is:

[0022]

[0023] Calculate the similarity for all documents in the entire document library, take 10% of the documents in the document library as the candidate set of field-related data, and construct a new field monolingual dataset after splitting the documents based on regular expressions and performing deduplication and filtering

[0024] In step 3), the specific steps for generating a domain pseudo-parallel dataset based on back translation can be as follows:

[0025] Take the obtained domain monolingual dataset as the source-side dataset input x, and use the source language to target language translation model trained in step 1) to obtain pseudo-sentence pairs After sorting and alignment, form a pseudo-bilingual parallel dataset S1; when there is a large amount of original parallel corpus, the sampling method uses beam search plus noise, and when the parallel corpus is small, the sampling method used is beam search;

[0026] In step 4), the specific steps for the joint training based on domain classification learning include:

[0027] To enable the model to better learn domain-related knowledge, effectively distinguish noise, and at the same time ensure the matching degree of the model, mix and retrain the generated pseudo-bilingual parallel dataset S1 and the original parallel corpus S; the training model uses the Transformer model, and add a domain classification task to the model during training for joint training; add a label to the source language input to distinguish domain data and non-domain data for classification; given the source sentence X = {x1,..., x w} of the sentence pair, the label domian = {normal|special} and the target sentence Y = {y1,..., y t |, where W and T are the number of tokens of the source sentence and the target sentence respectively, and the training objective function of machine translation is as follows:

[0028]

[0029] where θ is the learnable model parameter;

[0030] The objective function of domain classification is as follows:

[0031]

[0032] where is the probability of the model's domain classification, π y is the correct classification label, 1 represents special, and 0 represents normal;

[0033] The two parameters λ1 and λ2 are parameters to be adjusted, and the final loss function can be expressed as:

[0034]

[0035] The present invention provides a method for constructing and training reverse translation data based on domain classification. Based on the concept of domain-based classification, domain-related texts are screened from network texts through keyword and topic similarity and segmented to form a domain monolingual dataset. At the same time, two bilingual translation models are trained with the original corpus dataset, and the obtained domain monolingual corpus is reverse-translated to form pseudo-parallel bilingual corpus. These pseudo-parallel data are labeled as domain corpus and general data for joint training, and domain classification is added to the training model to improve the translation quality within the domain. The present invention can quickly and efficiently construct a domain dataset by using the method of keyword and topic similarity, and enhance the translation effect by reverse translation. To ensure that the model can better learn the distribution of domain and general data, the joint training method of domain classification can enable the model to increment on general data while maintaining a good recognition effect on domain data. At the same time, this reverse translation is beneficial to increasing the data quantity, ensuring the improvement of the model quality, obtaining high-quality knowledge in related fields, effectively using general domain knowledge, improving the translation performance, and ensuring the translation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 FIG. is a flowchart of the implementation process of the method for constructing and training reverse translation data based on domain classification.

[0037] Figure 2 FIG. is a flowchart of the training of a bidirectional model.

[0038] Figure 3 FIG. is a flowchart of constructing a domain monolingual dataset based on topic similarity.

[0039] Figure 4 FIG. is a model diagram of joint training based on domain classification learning. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The following embodiments further illustrate the present invention in conjunction with the accompanying drawings, but are not used to limit the scope of the present invention. The following embodiments are the preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

[0041] The present invention aims to solve the problem of lack of corpus for machine translation models in the translation of minority languages, especially in special fields. It constructs a reverse translation domain dataset from documents using topic similarity, and at the same time improves translation quality through joint training of domain classification. Translation of minority languages usually faces the problem of lack of corpus. Constructing a pseudo-parallel dataset using reverse translation is one of the simplest and most practical data augmentation methods. However, for domain translation, it is difficult to obtain high-quality monolingual data. Such a small amount of reverse translation is not enough to improve translation quality, and a large amount of easily obtained domain-irrelevant unidirectional data will reduce the quality of domain translation. Therefore, a domain dataset is constructed from a large number of network documents by using keyword-based topic similarity matching. To improve the domain recognition ability and generalization performance of the model, general corpus is added to form a new pseudo-parallel corpus after reverse translation, and then large-scale retraining is carried out on the model based on the domain classifier. In this way, the model can obtain high-quality relevant domain knowledge while effectively using general domain knowledge, improve translation performance, and ensure translation quality. As Figure 1 shown, the construction of the reverse translation system in the present invention mainly consists of four parts: training a bilingual model based on the original corpus, constructing a domain monolingual dataset based on topic similarity, generating a domain pseudo-parallel dataset based on reverse translation, and joint training based on domain classification learning.

[0042] 1) Training a bilingual model based on the original corpus:

[0043] The Transformer model used in the present invention (Vaswani, A., et al. (2017). Attention is all you need. arXiv) consists of four modules: input, encoding, decoding, and output. Specifically, the open-source neural machine translation toolkit THNMT of Tsinghua University (Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, Yang Liu. 2017. THUMT: An Open Source Toolkit for Neural Machine Translation. arXiv:1706.06415) can be used as a training tool. Taking Mongolian-Chinese as an example, two neural machine translation models are trained with Mongolian and Chinese as the source-target and target-source respectively. The training process specifically requires several steps: a) Prepare parallel corpora. b) Download mosesdecoder and stanford-segmenter. c) (For Mongolian, it can be pre-converted to its Latin form before the following operations) For languages such as English and Mongolian, use mosesdecoder for word segmentation, and for Chinese, use stanford-segmenter for word segmentation. d) Simply filter the sentences with a length exceeding maxLen (usually set to 256) from the segmented data. e) Use the BPE tool in subword-nmt (Rico Sennrich, Barry Haddow and Alexandra Birch (2016): Neural Machine Translation of Rare Words with Subword Units Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). Berlin, Germany) for subword segmentation. f) Divide the training set, test set, and validation set. g) Use the THNMT tool to generate the vocabulary, that is, count the word frequencies and proportions of the source language sentences and target language sentences in the bilingual sentence pairs after automatic word segmentation, and determine the sizes of the target vocabulary and the source vocabulary. h) Use THNMT for training until convergence, and the same method is used for training in both directions. i) Use multi-bleu for validation and evaluation. After the above word segmentation and filtering of 600,000 Mongolian-Chinese torture bidirectional corpora, 550,000 training corpora can be obtained. The generated vocabulary sizes are 20,000 for Mongolian and 30,000 for Chinese. After 100,000 steps, the training can converge to convergence. The overall training process is asFigure 2 as shown

[0044] 2) Construct a monolingual domain dataset based on topic similarity:

[0045] In order to obtain more and faster monolingual domain datasets, by directly searching for domain-related texts, that is, expanding the dataset through document topic similarity to avoid the inefficiency of sentence-by-sentence selection; for any domain, the keywords frequently appearing within the domain can represent the theme of this domain. For example, the keywords in the field of torture are generally "torture, police, crime, capture", etc.; a representative vocabulary of a domain can be easily obtained manually. On the other hand, frequently appearing words usually also mean they are related to the domain theme, but some meaningless words such as pronouns and personal names need to be excluded. It can be constructed by the automatic acquisition method of the program. The specific steps are as follows:

[0046] A) Domain theme representation: For any domain, a basic vocabulary needs to be constructed first to represent the theme expressed by the current domain. Use Python toolkits such as Jieba Chinese word segmentation to segment words. For Mongolian, the existing domain dataset is segmented by spaces to perform word frequency statistics, and the top 500 common words are obtained. Manually screen and filter out some common words to obtain a basic vocabulary of size 256 as the domain theme representative D = [d1, d2,... di…, dn], where di represents the i-th word.

[0047] B) Document library Here, some news web pages such as the Mongolian version of People's Daily can be selected for crawling and collection. For any article P = [p1, p2,... pi…, pn] in the document library, pi represents the i-th sentence.

[0048] C) Execute content selection, extract important sentences from the document, and provide a mixture of two extraction methods. First, extract important parts of an article such as the title, the first paragraph, and the end, and then select sentences containing certain informative keywords (such as "generally speaking") for extraction. Mix the data extracted by the two methods to form a subset P1 and require that this subset meet a certain total number constraint. Generally, word frequency statistics and filtering are performed on these sentences to obtain the article keyword representative set Ds = [dp1, dp2,…, dpn].

[0049] D) Use the pre-trained word vector model of BERT (Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.) to convert D and Ds into vector representations, and then sum all the word vectors to obtain the topic vector representation. and

[0050]

[0051]

[0052] Use the Pearson correlation coefficient to represent the similarity between the two, and use it to judge the similarity between the text topic and the words in the thesaurus. The calculation formula is:

[0053]

[0054] Calculate the similarity for all documents in the entire document library, and select the documents in the document library with a similarity exceeding a certain threshold as the candidate set of domain-related data.

[0055] E) Based on regular expressions, Chinese is segmented by sentences according to punctuation marks (such as 。!?), while Mongolian segments the document according to its special stop symbols and constructs a new domain monolingual dataset after deduplication and filtering. As an example, when selecting 120,000 documents from Mongolian news and the average similarity of the documents is 25.2%, select the documents with a similarity exceeding 40%. Through text segmentation, finally obtain about 9.6 million monolingual sentences to construct a domain monolingual dataset.

[0056] The overall construction process is as Figure 3 shown.

[0057] 3) Generate a domain pseudo-parallel dataset based on back translation

[0058] Use the obtained domain monolingual dataset as the source-side data input x, and use the trained source language to target language translation model in step 1) to obtain pseudo sentence pairs After sorting and alignment, a pseudo-bilingual parallel dataset S1 is formed. When there is a large amount of original parallel corpus, the sampling method uses beam search plus noise. When the parallel corpus is small, the sampling method used here is beam search.

[0059] 4) Joint training based on domain classification learning:

[0060] Compared with the original model of Transformer, the embodiment of the present invention adds a task of domain classification learning to help the model better distinguish noise and learn domain knowledge. Mix the generated pseudo-bilingual parallel dataset S1 and the original parallel corpus S, and the mixing ratio is usually 2:1. During the mixing process, a prefix needs to be added before each bilingual sentence pair to distinguish which dataset it comes from, and then retraining is carried out on the new model. The training model is still based on the Transformer model, and no changes are required for the encoder and decoder parts. In the input module, the sentence pair prefix is read as the label domian and added to distinguish domain data and non-domain data for classification. Given the source sentence X = {x1,..., x w} of the sentence pair, the label domian = {normal|special}, and the target sentence Y = {y1,..., y t}, where W and T are the number of tokens in the source sentence and the target sentence respectively. The training objective function of machine translation is as follows. Where θ is the learnable model parameter:

[0061]

[0062] In the output layer after the decoding layer, a linear layer and a softmax module are used to obtain the classification prediction probability, and the binary cross-entropy loss is used for calculation. The objective function of domain classification is as follows, is the probability of the model's domain classification, π y is the correct classification label, 1 represents special, and 0 represents normal.

[0063]

[0064] The two parameters λ1 and λ2 are parameters to be adjusted, and the final loss function can be expressed as:

[0065]

[0066] The specific translation model is as follows Figure 4 shown as follows

[0067] In the example of the present invention, in the translation of Mongolian-Chinese torture corpora, the M01 dataset represents the original bilingual corpus dataset, the M02 dataset represents the bilingual corpus mixed with the pseudo-parallel corpus S1 and the original parallel corpus S, and the M03 dataset is the bilingual corpus obtained by randomly selecting 800,000 monolingual corpora from the document library without passing through the processing of Step 2, performing back translation, and then mixing them with the original parallel corpus S

[0068] Table 1

[0069]

[0070] Table 1 shows the performance of the present invention and the basic Transformer neural machine translation model (NMT) on different datasets (M01, M02, M03) in the field of Mongolian-Chinese torture. It can be found that although the back-translated corpus without processing (the third row, M03) enhances and supplements the data, its poor translation quality and inadaptability to the field lead to a decrease rather than an increase in translation quality. After translation, the domain monolingual dataset selected based on topic similarity can bring a 1.07 BLEU value improvement to the model (the second row, M02), and after further adding domain classification learning, the machine translation system applying the present invention has a 1.91 BLEU value improvement

[0071] The present invention constructs a back-translation domain dataset from documents using topic similarity, and at the same time improves translation quality through joint training of domain classification. Minority language translation usually faces the problem of lack of corpora. Constructing a pseudo-parallel dataset using back translation is one of the simplest and most practical data augmentation methods. However, for domain translation, it is difficult to obtain high-quality monolingual data. Such a small amount of back translation is not enough to improve translation quality, while a large amount of easily available domain-irrelevant unidirectional data will reduce the quality of domain translation. Therefore, a domain dataset is constructed from a large number of network documents by using keyword-based topic similarity matching. To improve the domain recognition ability and generalization performance of the model, a new pseudo-parallel corpus is formed by adding general corpora after back translation, and then large-scale retraining is performed on a model based on a domain classifier. In this way, the model can obtain high-quality relevant domain knowledge while effectively using general domain knowledge, improve translation performance, and ensure translation quality

Claims

1. A method for constructing and training reverse translation data based on domain classification, characterized in that It includes the following steps: 1) Training a bilingual model based on the original corpus: Using the Transformer neural network model, train two neural machine translation models by taking the bilingual original corpus as the source - target end and the target - source end respectively, 2) Constructing a domain - specific monolingual dataset based on keyword and topic similarity; 3) Generating a domain - specific pseudo - parallel dataset based on back - translation: Taking the domain - specific monolingual dataset obtained in step 2) as the source - end dataset, use the trained bilingual model in step 1) to perform back - translation to obtain pseudo - sentence pairs, and after sorting and alignment, form a domain - specific pseudo - parallel dataset; The specific steps for generating the domain - specific pseudo - parallel dataset based on back - translation are: The obtained monolingual dataset of the domain is used as the source dataset x, and the trained translation model from the source language to the target language in step 1) is used to obtain pseudo-sentence pairs After sorting and alignment, a pseudo-bilingual parallel dataset S1 is formed; when there is a large amount of original parallel corpus, the sampling method is beam search plus noise, and when the parallel corpus is small, the sampling method used is beam search; 4) Joint training based on domain - classification learning: Mark the pseudo - parallel dataset obtained in step 3) as domain - specific corpus and conduct joint training with general data.

2. The method for constructing and training reverse translation data based on domain classification according to claim 1, wherein In step 1), the specific steps for training the bilingual model based on the original corpus are: The language model is based on the open - source Transformer neural network model, which is an encoder - decoder model based on self - attention and consists of four modules: input, encoding, decoding, and output. The input module reads the input text sequence Input = {s1, s2, s3, …, su} from the training file, where su represents the u - th word, obtains the initial word embedding and connects it with the position encoding to get the input vector Input Embedding = {x1, x2, …, xe}, and then sends it to the encoder. Each layer of the encoder consists of a multi - head attention mechanism, a feed - forward neural network, a normalization layer, and a residual connection. Among them, the core model is the multi - head attention mechanism evolved from the attention mechanism, and the expression is: After the encoder obtains the abstract feature representation of the input, input it into the decoder for decoding; The decoder has the same structure as the encoder and, conversely, performs decoding by relating the target embedding, the encoder state, and the decoder state; The decoding result passes through a linear layer and a softmax layer to output the probability distribution of the overall label. For the machine - translation bilingual model, the commonly used training loss function is the cross - entropy loss. Based on the use of this model, after pre - processing the sorted high - quality original bilingual dataset S by alignment, word - segmentation, BPE, statistical vocabulary, and training - set splitting, train a unidirectional model distributively. Based on the constructed bilingual model, train with the original corpus to obtain two bilingual models.

3. The method for constructing and training reverse translation data based on domain classification according to claim 1, wherein In step 2), the specific steps for constructing a domain - specific monolingual dataset based on keyword and topic similarity are: Search for domain - related passages, that is, expand the dataset through document - topic similarity to avoid the inefficiency of selecting sentence by sentence. For any domain, the keywords that frequently appear within the domain represent the theme of this domain, but some pronouns, personal names, and meaningless words need to be excluded. It is constructed by an automatic program - acquisition method. The specific construction method: (1) Construct a basic vocabulary. Use a Python toolkit to perform word frequency statistics on the existing domain dataset, obtain the top 500 common words, and then manually screen out a vocabulary, filtering out some common words to obtain a basic vocabulary of size 256 as the domain theme representative D = [d1, d2, …, dn], where di represents the i-th word; (2) Extract the theme of the document, which is similar to the above process and is also obtained through word frequency statistics; for any article P = [p1, p2, p3…, pm] in the document library, pm represents the m-th sentence; perform content selection to extract the important sentences of the document, providing a mixture of two extraction methods. One is extraction based on the main structure. The important parts of a document are mainly concentrated in the title, the first paragraph, and the end. Extract these parts. The other is to select sentences containing certain informative keywords for extraction; form a subset P1 through the extraction of the two methods and meet the extraction total constraint. Obtain the article keyword representative set Dp = [dp1, dp2, …, dpj] by performing word frequency statistics filtering on these sentences, where dpj represents the j-th word extracted from article P; in this way, the theme similarity is transformed into the similarity of two keyword sets representing the themes, and the similarity between the two is calculated through vector representation; (3) Convert D and Ds into vector representations using BERT word vectors, and sum all the word vectors to obtain the topic vector representation and Calculate the similarity Sim using the Pearson correlation coefficient topic , which is used to determine the similarity between the text theme and the words in the thesaurus. The calculation formula is as follows: Calculate the similarity of all documents in the entire document library, take 10% of the documents in the document library as the candidate set of domain-related data, and construct a new domain monolingual dataset after segmenting the documents based on regular expressions and performing deduplication filtering 4. The method for constructing and training reverse translation data based on domain classification according to claim 1, wherein In step 4), the specific steps of the joint training based on domain classification learning include: The generated pseudo-bilingual parallel dataset S1 and the original parallel corpus S are mixed and retrained; the training model uses the Transformer model, and a domain classification task is added to the model during training for joint training; a label is added to the source language input to distinguish domain data from non-domain data for classification; given the source sentence X = {x1, …, x w} of the sentence pair, the label domian = {normal|special} and the target sentence Y = {y1, …, y t}, where W and T are the number of tokens in the source sentence and the target sentence respectively, and the training objective function of machine translation is as follows: where θ is the learnable model parameter; The objective function of domain classification is as follows: Among them, is the probability of the model domain classification, π y is the correct classification label, where 1 represents special and 0 represents normal; The two parameters λ1 and λ2 are parameters to be adjusted, and the final loss function is expressed as:

Images