A training method for a vertical domain machine translation engine

Abstract

The application discloses a training method for a vertical field machine translation engine, comprising cleaning of vertical field data, mining of vertical field parallel corpus, fine tuning of a general field machine translation model, mining of vertical field monolingual data, generation of vertical field pseudo-parallel corpus and training of a vertical field machine translation model. The application reduces the risk of model overfitting through two-round fine tuning training, improves the accuracy of the model on field data, expands the field data set through various data enhancement methods, reduces the risk of model overfitting through multiple rounds of iterative training, and improves the accuracy of field translation.

Description

Technical Field

[0001] This invention relates to a training method for an engine, specifically a training method for a vertical domain machine translation engine, belonging to the field of machine translation technology in natural language processing. Background Technology

[0002] Generally speaking, Natural Language Processing (NLP) refers to the technology of using computers to process and analyze human language, specifically including: 1) Machine translation: the process of automatically translating text from one language to another using artificial intelligence; 2) BT data: BT (back translation) is a data augmentation technique used in the field of machine translation, which constructs parallel corpora from the source language to the target language by translating the source language into the target language; 3) FT data: FT (forward translation) is a data augmentation technique used in the field of machine translation, which constructs parallel corpora from the source language to the target language by translating the source language into the target language.

[0003] General-purpose machine translation engines are typically designed to solve broad and widespread translation tasks across multiple domains. In contrast, vertical-domain machine translation engines are machine translation systems specifically designed for a particular domain. Existing technologies, such as the personalized machine translation training method and system disclosed in CN114564933A, achieve relatively low-cost, highly personalized incremental training of machine translation engines by fusing personalized incremental training with personalized corpora and interactive translation techniques. However, domain-specific corpora often contain a large number of specialized terms, specific contexts, and industry-specific jargon. These specialized terms are often poorly handled by general-purpose machine translation engines, leading to a decline in translation quality. To address this issue, vertical-domain machine translation models improve their understanding of domain-specific data and translation accuracy by learning from domain-specific corpora. However, for specific domains, the limited training corpora can result in insufficient training of the machine translation model, failing to achieve the desired results. Summary of the Invention

[0004] The purpose of this invention is to provide a training method for a vertical domain machine translation engine in order to solve at least one of the above-mentioned technical problems, by fine-tuning an existing general domain machine translation model to make it applicable to vertical domains.

[0005] This invention achieves the above objective through the following technical solution: a training method for a vertical domain machine translation engine, the training method comprising the following steps: Step 1: Cleaning vertical domain data: Collect parallel corpus data in the vertical domain and clean the parallel corpus data in the vertical domain to obtain cleaned parallel corpus data in the vertical domain. Step 2: Mining of parallel corpora in vertical domains: Mining parallel corpora in vertical domains based on general domain parallel corpora data to obtain the mined parallel corpora in vertical domains; Step 3: Fine-tuning of the general domain machine translation model: Fine-tuning the general domain machine translation model using the cleaned vertical domain parallel corpus and the mined vertical domain parallel corpus to obtain the fine-tuned general domain machine translation model; Step 4: Mining monolingual data in vertical domains; Step 5: Generation of pseudo-parallel corpora in vertical domains; Step 6: Training the vertical domain machine translation model.

[0006] As a further technical solution of the present invention, in step two, the mining of the vertical domain parallel corpus: 1) Obtain the source language domain monolingual data and the target language domain monolingual data from the cleaned vertical domain parallel corpus, wherein the source language domain monolingual data is denoted as S and the target language domain monolingual data is denoted as T. 2) Collect parallel corpus data in general domains to obtain general data for single languages ​​in the source language domain and general data for single languages ​​in the target language domain; 3) Construct domain classification models for the source language domain monolingual data and the target language domain monolingual data: Train a binary classification model for source language text based on the source language domain monolingual data and the source language domain general monolingual data, and train a binary classification model for target language text based on the target language domain monolingual data and the target language domain general monolingual data; 4) Predict the source language data in the general domain parallel corpus data using the source language text binary classification model, and select the source language data as the source language parallel corpus of the specified domain; The target language data in the general domain parallel corpus is predicted by the target language text binary classification model, and the selected target language data is the target language parallel corpus of the specified domain. Specifically, the selected source language data and the selected target language data are combined to obtain the mined vertical domain parallel corpus.

[0007] As a further technical solution of the present invention, step four, the mining of the vertical domain monolingual data includes the following steps: 1) Collect source language monolingual data and target language monolingual data; 2) Train a monolingual language model; 3) Use a monolingual language model to filter source language monolingual data and target language monolingual data; 4) Use the source language text binary classification model to filter out monolingual data in the source language domain, and use the target language text binary classification model to filter out monolingual data in the target language domain.

[0008] As a further technical solution of the present invention, step five, the generation of domain pseudo-parallel corpus, specifically includes: 1) The source language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is sampling, and the result is used as the BT data of ST for training the TS model. 2) The source language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is beam_search, and the result is used as the FT data of ST for training the ST model. 3) The target language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is sampling, and the result is used as the BT data of TS to train the ST model. 4) The target language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is beam_search, and the result is used as the TS FT data for training the TS model.

[0009] As a further technical solution of the present invention, the specific steps for training the vertical domain model in step six are as follows: 1) Using the vertical domain parallel corpus, the mined vertical domain parallel corpus, the FT data of the ST and the BT data of the TS, the final vertical domain ST machine translation model is obtained by fine-tuning the fine-tuned general domain machine translation model. 2) Using the vertical domain parallel corpus, the mined vertical domain parallel corpus, the FT data of the TS, and the BT data of the ST, the final vertical domain TS machine translation model is obtained by fine-tuning the fine-tuned general domain machine translation model.

[0010] The beneficial effects of this invention are: 1) Collect parallel corpus data in vertical domains, clean the parallel corpus data in vertical domains, and filter out parallel corpus data in vertical domains from the parallel corpus data in general domains. That is, more vertical domain datasets were obtained through corpus mining methods. 2) Vertical domain BT data was generated using a general domain machine translation model and the mined domain monolingual data. Pseudo-parallel corpora were generated by combining the machine translation model, which further expanded the training set. 3) Based on the cleaned vertical domain parallel corpus and the mined vertical domain parallel corpus, the general domain machine translation model is fine-tuned. Through two rounds of fine-tuning training, the risk of overfitting of the machine translation model is reduced and the accuracy of the model on domain data is improved. 4) The domain dataset was expanded using various data augmentation methods to train the vertical domain model, and the risk of model overfitting was reduced through multiple rounds of iterative training, thereby improving the accuracy of domain translation. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the classification model process of the present invention; Figure 2 This is a schematic diagram of the corpus enhancement process of the present invention; Figure 3 This is a schematic diagram of the first round of model training process of the present invention; Figure 4 This is a schematic diagram of the second round of model training process of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0013] Example 1: The general domain refers to a domain that covers all industries, while the vertical domain refers to a sub-domain of the general domain.

[0014] like Figures 1 to 4 As shown, a training method for a vertical domain machine translation engine is described, which includes: First: Cleaning data from vertical industries.

[0015] Collect parallel corpus data in vertical domains, and clean the parallel corpus data in vertical domains to obtain cleaned parallel corpus data in vertical domains.

[0016] Collecting parallel corpus data in vertical domains refers to collecting parallel corpus data in vertical domains from general domain corpus data. Parallel corpus data includes source language corpus and target language corpus.

[0017] Cleaning refers to the preprocessing of collected parallel corpora in the vertical field, using Chinese and English corpora. The preprocessing of this pair of parallel corpora includes: (1) removing duplicate corpus pairs; (2) removing Chinese sentences with more than 30% English characters; (3) removing English sentences containing Chinese characters; (4) removing HTML tags from sentences; (5) removing sentences that are too long or too short; (6) removing invisible characters from sentences; (7) removing sentences with a length ratio greater than 3 or less than 1 / 3; and (8) converting traditional Chinese characters to simplified Chinese characters and converting full-width characters in English characters to half-width characters.

[0018] Second: Mining parallel corpora in vertical domains.

[0019] Since general domain parallel corpus data is easier to obtain and the amount of data is large enough, vertical domain parallel corpus data is mined based on general domain parallel corpus data to obtain the mined vertical domain parallel corpus.

[0020] The specific operating method is as follows: 1) Obtain the source language domain monolingual data and the target language domain monolingual data from the cleaned vertical domain parallel corpus, wherein the source language domain monolingual data is denoted as S and the target language domain monolingual data is denoted as T; 2) Collect parallel corpus data in general domains to obtain general data for single languages ​​in the source language domain and general data for single languages ​​in the target language domain; 3) Construct domain classification models for the source language domain monolingual data and the target language domain monolingual data: Train a binary classification model for source language text based on the source language domain monolingual data and the source language domain general monolingual data, and train a binary classification model for target language text based on the target language domain monolingual data and the target language domain general monolingual data; 4) Predict the source language data in the general domain parallel corpus data using the source language text binary classification model, and select the source language data as the source language parallel corpus of the specified domain; The target language data in the general domain parallel corpus is predicted by the target language text binary classification model, and the selected target language data is the target language parallel corpus of the specified domain. Specifically, the selected source language data and the selected target language data are combined to obtain the mined vertical domain parallel corpus.

[0021] Third: Fine-tuning of general-domain machine translation models.

[0022] By fine-tuning the general domain machine translation model using cleaned vertical domain parallel corpora and mined vertical domain parallel corpora, a fine-tuned general domain machine translation model is obtained. Taking Chinese-English translation as an example, some parameters of a general-domain machine translation model can be set to max_epoch=10 and learning_rate=5*10-5. These parameters can be set according to specific circumstances. Since the training data in this round is relatively small, the maximum number of training epochs and the learning rate should not be set too high; otherwise, overfitting may occur during training.

[0023] Fourth: Mining monolingual data in vertical domains.

[0024] 1) Collect source language monolingual data and target language monolingual data; 2) Train a monolingual language model; 3) Use monolingual language models to filter source language monolingual data and target language monolingual data; 4) Use the source language text binary classification model to filter out the source language domain monolingual data, and use the target language text binary classification model to filter out the target language domain monolingual data.

[0025] Taking Chinese-English translation as an example: 1) Collect Chinese monolingual data and English monolingual data; 2) Train a monolingual language model; 3) Use the monolingual language model to filter Chinese monolingual data and English monolingual data (filter out low-quality monolingual data); 4) Use a Chinese text binary classification model to filter out Chinese domain monolingual data, and use an English text binary classification model to filter out English domain monolingual data.

[0026] Fifth: Generation of pseudo-parallel corpora in vertical domains.

[0027] 1) The source language domain monolingual data is used as input, and the prediction is performed by a fine-tuned general domain machine translation model. The decoding method is sampling. The result is used as the ST (source language - target language) BT data to train the TS model. Here, the TS model refers to the target language - source language template. 2) The source language domain monolingual data is used as input, and the prediction is performed by a fine-tuned general domain machine translation model. The decoding method is beam_search. The result is used as the ST (source language-target language) FT data to train the ST model. Here, the ST model refers to the source language-target language model. 3) The target language domain monolingual data is used as input, and the prediction is performed by a fine-tuned general domain machine translation model. The decoding method is sampling, and the result is used as TS (target language - source language) BT data to train the ST model. 4) The target language domain monolingual data is used as input, and the prediction is performed by a fine-tuned general domain machine translation model. The decoding method is beam_search. The result is used as TS (target language - source language) FT data to train the TS model.

[0028] Sixth: Training of vertical domain machine translation models.

[0029] 1) Using vertical domain parallel corpora, mined vertical domain parallel corpora, ST's FT data and TS's BT data, the final vertical domain ST machine translation model is obtained by fine-tuning the fine-tuned general domain machine translation model. 2) Using vertical domain parallel corpora, mined vertical domain parallel corpora, TS FT data and ST BT data, the final vertical domain TS machine translation model is obtained by fine-tuning the fine-tuned general domain machine translation model.

[0030] Example 2: The vertical field of this example is finance, and the machine translation model is a Chinese-French general translation model.

[0031] 1. Collect parallel corpus data on Chinese and French discourse in the financial field, and collect general parallel corpus data on Chinese and French discourse; 2. Clean the Sino-French parallel corpus data in the financial field. The cleaning process includes removing duplicate sentences, removing special characters, filtering based on semantic similarity, removing sentences with excessively long lengths, converting traditional Chinese characters to simplified Chinese characters, and converting full-width French characters to half-width characters. This results in a cleaned Sino-French parallel corpus (financial field). 3. Collect general Chinese and French corpora to obtain general Chinese and French monolingual corpora (general domains); 4. Train a Chinese language model based on the collected Chinese monolingual corpus and general Chinese monolingual corpus in the financial field, and train a French language model based on the collected French monolingual corpus and general French monolingual corpus in the financial field. 5. Filter the Chinese monolingual corpus in the financial field using a Chinese language model, and filter the French monolingual corpus in the financial field using a French language model to obtain high-quality Chinese monolingual corpus and French monolingual corpus in the financial field. 6. Train a domain classification model using monolingual general data (Chinese monolingual general corpus and French monolingual general corpus) and monolingual financial data (high-quality Chinese monolingual corpus and French monolingual corpus in the financial field); 7. Parallel financial corpus was extracted from the Sino-French common parallel corpus data using a domain classification model, resulting in the mined parallel financial corpus. 8. By fine-tuning the Sino-French general translation model using cleaned and mined parallel corpora in the financial field, a Sino-French financial translation model is obtained; 9. Mining financial monolingual data through domain classification models; 10. Generate BT and FT data using a Sino-French financial translation model; 11. By using parallel financial data, mined parallel financial data, BT data, and FT data, the Sino-French financial translation model is fine-tuned to obtain the final Sino-French financial translation model.

[0032] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0033] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A training method for a vertical domain machine translation engine, characterized in that, The training method includes the following steps: Step 1: Cleaning of vertical domain data: Collect parallel corpus data in the vertical domain and clean the parallel corpus data in the vertical domain to obtain cleaned parallel corpus data in the vertical domain; wherein, the cleaned parallel corpus data in the vertical domain includes source language domain monolingual data and target language domain monolingual data, and the source language domain monolingual data is denoted as S and the target language domain monolingual data is denoted as T; Step 2: Mining of parallel corpora in vertical domains: Mining parallel corpora in vertical domains based on general domain parallel corpora data to obtain the mined parallel corpora in vertical domains; Step 3: Fine-tuning of the general domain machine translation model: Fine-tuning the general domain machine translation model using the cleaned vertical domain parallel corpus and the mined vertical domain parallel corpus to obtain the fine-tuned general domain machine translation model; Step 4: Mining monolingual data in vertical domains; Step 5: Generation of pseudo-parallel corpora in vertical domains; Step five specifically includes the generation of pseudo-parallel corpora in vertical domains: 1) The source language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is sampling, and the result is used as the BT data of ST for training the TS model. 2) The source language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is beam_search, and the result is used as the FT data of ST for training the ST model. 3) The target language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is sampling, and the result is used as the BT data of TS to train the ST model. 4) The target language domain monolingual data is used as input, and the fine-tuned general domain machine translation model is used for prediction. The decoding method is beam_search, and the result is used as the TS FT data for training the TS model. Step Six: Training the Vertical Domain Machine Translation Model; In step six, the specific steps for training the vertical domain machine translation model are as follows: 1) Using the cleaned vertical domain parallel corpus, the mined vertical domain parallel corpus, the FT data of the ST, and the BT data of the TS, the final vertical domain ST machine translation model is obtained by fine-tuning the fine-tuned general domain machine translation model. 2) Using the cleaned vertical domain parallel corpus, the mined vertical domain parallel corpus, the FT data of the TS, and the BT data of the ST, the final vertical domain TS machine translation model is obtained by fine-tuning the fine-tuned general domain machine translation model.

2. The training method according to claim 1, characterized in that: In step two, the mining of the vertical domain parallel corpus includes: 1) Obtain the source language domain monolingual data and the target language domain monolingual data from the cleaned vertical domain parallel corpus, wherein the source language domain monolingual data is denoted as S and the target language domain monolingual data is denoted as T; 2) Collect parallel corpus data in general domains to obtain general data for single languages ​​in the source language domain and general data for single languages ​​in the target language domain; 3) Construct domain classification models for the source language domain monolingual data and the target language domain monolingual data: Train a binary classification model for source language text based on the source language domain monolingual data and the source language domain general monolingual data, and train a binary classification model for target language text based on the target language domain monolingual data and the target language domain general monolingual data; 4) Predict the source language data in the general domain parallel corpus data using the source language text binary classification model, and select the source language data as the source language parallel corpus of the specified domain; The target language data in the general domain parallel corpus is predicted by the target language text binary classification model, and the selected target language data is the target language parallel corpus of the specified domain. Specifically, the selected source language data and the selected target language data are combined to obtain the mined vertical domain parallel corpus.

3. The training method according to claim 2, characterized in that: In step four, the mining of the vertical domain monolingual data includes the following steps: 1) Collect source language monolingual data and target language monolingual data; 2) Train a monolingual language model; 3) Filter the source language monolingual data and the target language monolingual data using the monolingual language model; 4) Use the source language text binary classification model to filter out the source language domain monolingual data, and use the target language text binary classification model to filter out the target language domain monolingual data.

Images