A personalized machine translation training method and system

Abstract

The application particularly relates to a personalized machine translation training method and system, the personalized machine translation training method comprising the steps of: uploading historical corpus by a translator; automatically preprocessing the uploaded corpus; manually auditing the preprocessed corpus; incrementally training a model on the audited corpus; extracting part of a test set from the incrementally trained model to automatically evaluate the quality of a translation; after the automatic quality evaluation is completed, the translator manually evaluates the quality; after the manual quality evaluation is passed, the translator can quickly put the incrementally trained machine translation engine into a pre-translation process through a machine translation scheduling system; pre-translation; interactive translation; and continuous incremental training. The personalized machine translation training method realizes the incremental training of a machine translation engine with low cost and high personalization by combining the incremental training of personalized corpus and the interactive translation technology.

Description

Technical Field

[0001] This invention belongs to the fields of natural language processing, machine translation, and computer-aided translation, and specifically relates to a personalized machine translation training method and system that integrates incremental learning and interactive translation. Background Technology

[0002] Machine translation technology refers to the technology of using computers and other computing devices to translate a source text (generally called the source language) into a target language (generally called the target language). Because this technology is performed by machines, it can process a large volume of translation work in a relatively short time compared to human translation. In recent years, machine translation technology has developed rapidly.

[0003] Machine translation technology can be broadly divided into two categories:

[0004] One approach is to train a vertical domain translation engine, which involves subdividing translation scenarios into areas such as medicine, culture, law, and patents, compiling relevant domain corpora, and training a translation engine specifically for that domain or industry. Alternatively, it could allow specific users to upload their own corpora for incremental training. Essentially, this involves learning from the translators' translation history.

[0005] Another approach is interactive translation, which involves collecting corpora during the translator's process. Essentially, it's incremental learning through real-time interaction. This involves summarizing the corpora accumulated during the translator's work, collecting terminology, input style, and habits, and then using this data to train and update the machine translation engine in real-time or periodically.

[0006] Vertical domain machine translation offers some quality improvement over general-purpose engines in specific domains. However, given the broad concept of "domain," the reality is that the corpus and the content to be translated often cannot be strictly categorized. For example, content in patent documents may belong to both the patent domain and another specific domain. Methods that allow translators to upload corpora for incremental training meet translators' requirements in terms of domain segmentation, but require periodic corpus aggregation, and the machine cannot learn in real-time from the knowledge gained during the translator's interaction.

[0007] Interactive machine translation allows the machine to incrementally learn from the translator's input and the translation results in real time. However, the translator interacts with a general machine translation engine and cannot interact with their own incrementally trained engine in real time to continuously learn during subsequent use. Furthermore, because updating the model through interactive translation requires time to accumulate before noticeable effects are achieved, it is difficult to solve the system's cold start problem.

[0008] Since deep neural networks were used in machine translation technology, the quality of machine translation has been greatly improved, and general machine translation has entered a wider range of application scenarios. However, in professional, vertical or personalized fields, current machine translation technology has problems such as the inability to use a large amount of translators' translation history knowledge for machine learning, low degree of personalization, and high cost of incremental training. Summary of the Invention

[0009] To address the problems of low personalization and high incremental training costs in existing machine translation technologies, this invention provides a personalized machine translation training method and system. This personalized machine translation training method integrates incremental training with personalized corpora and interactive translation technology to achieve low-cost, high-personality incremental training of a machine translation engine. The technical problem to be solved by this invention is achieved through the following technical solution:

[0010] A personalized machine translation training method includes the following steps:

[0011] Step 1: The translator uploads historical corpus data;

[0012] Step 2: Automatically preprocess the uploaded corpus;

[0013] Step 3: Manually review the preprocessed corpus;

[0014] Step 4: Perform incremental model training on the approved corpus;

[0015] Step 5: Extract a portion of the test set from the incrementally trained model to perform automatic translation quality assessment;

[0016] Step Six: After the automatic quality assessment is completed, the translator performs a manual quality assessment;

[0017] Step 7: After the manual quality assessment is passed, the translator can quickly put the incrementally trained machine translation engine into the pre-translation process through the machine translation scheduling system;

[0018] Step 8: Pre-translation;

[0019] Step Nine: Interactive Translation;

[0020] Step 10: Continuous incremental training.

[0021] Furthermore, after step three above, the corpus that fails the review is returned to step one, and the translator re-uploads the historical corpus.

[0022] Furthermore, after step six above, if the manual quality assessment fails, the process returns to step one, where the translator re-uploads the historical corpus.

[0023] Furthermore, the aforementioned historical corpus includes a translation memory and a terminology database.

[0024] Furthermore, the preprocessing of the above-mentioned corpus refers to calculating the similarity between the original text and the source text through edit distance, automatically deduplicating the corpus and removing duplicate text, and at the same time calculating the semantic similarity between the original text and the translation in the uploaded corpus and removing the original text and translation with poor semantic similarity.

[0025] Furthermore, the above-mentioned incremental training of the model refers to initializing the parameter statistics of the original model, and then performing iterative parameter training to obtain the incrementally trained model.

[0026] Furthermore, the aforementioned manual quality assessment refers to comparing the translations of the original machine translation engine and the incrementally trained translation engine.

[0027] Furthermore, in the aforementioned pre-translation project, an incrementally trained engine is used to pre-translate the content to be translated. During the translation process in the editor, after the translator inputs part of the content, the incrementally trained engine performs constraint decoding based on this constraint and provides the re-decoded and generated translation in real time.

[0028] Furthermore, in the interactive translation process described above, the translator modifies the translation in real time, and the translation model re-decodes based on the modification results to generate a new translation recommendation. This process is repeated until the current sentence segment is translated.

[0029] Furthermore, in the aforementioned continuous incremental training, after the current sentence segment is translated, the translated sentence segment is stored in the translation memory in real time, and the machine translation engine is updated through the real-time online update algorithm.

[0030] A personalized machine translation training system corresponding to the above-mentioned personalized machine translation training method includes an incremental training module, an engine quality evaluation module, and a translation module;

[0031] The incremental training module includes a historical corpus upload module, a corpus preprocessing module, and a corpus review module.

[0032] The engine quality assessment module includes an automatic quality assessment module and a manual quality assessment module;

[0033] The translation module includes a pre-translation module and an interactive translation module.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] The personalized machine translation training method of the present invention solves the cold start problem by incrementally training on the translator's historical corpus. On the other hand, it solves the real-time learning problem during the process by real-time interaction between the translator and the engine after incremental training. The two form a closed-loop data feedback and seamless integration, which significantly reduces the cost of personalized machine translation for translators and improves the efficiency of data flow and machine learning. Attached Figure Description

[0036] Figure 1 This is a flowchart of the personalized machine translation training method in this embodiment. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0038] To address the issues of low personalization and high incremental training costs in existing machine translation technologies, this embodiment provides a personalized machine translation training method that integrates incremental learning and interactive translation. The flowchart of this personalized machine translation training method is attached. Figure 1 .

[0039] The overall process of the personalized machine translation training method in this embodiment is as follows: the translator uploads historical corpora, incrementally trains the initial machine translation model, and seamlessly integrates the trained engine into the pre-translation and interactive translation processes. During subsequent interactive translation, the system continues to perform incremental training. This personalized machine translation training method achieves low-cost, highly personalized incremental training of the machine translation engine by integrating personalized corpus incremental training and interactive translation technology.

[0040] A personalized machine translation training method includes the following steps:

[0041] Step 1: Translators upload historical data. Translators are allowed to upload a certain amount of historical data, such as translation memory and terminology database.

[0042] Step 2: After uploading the corpus, the system automatically preprocesses the uploaded corpus, calculates the similarity between the original text and the source text using edit distance, and automatically removes duplicate text. Simultaneously, it calculates the semantic similarity between the original text and the translated text in the uploaded corpus, removing those with poor semantic similarity.

[0043] Step 3: To ensure the effectiveness of incremental training, after the corpus preprocessing is completed, the corpus is manually reviewed, mainly considering fidelity and fluency.

[0044] Step 4: Perform incremental training on the reviewed corpus, initialize the parameter statistics of the original model, and then perform iterative parameter training to obtain the incrementally trained model.

[0045] Step 5: Extract a portion of the test set from the incrementally trained model to perform automatic translation quality assessment and obtain the BLEU score (Bilingual Evaluation Understudy) for the current specific domain.

[0046] Step Six: After the automatic quality assessment is completed, translators are allowed to perform a manual quality assessment, comparing the translations from the original machine translation engine and the incrementally trained translation engine. This concludes the incremental training process based on the translator's historical corpus.

[0047] Step 7: After the manual quality assessment is passed, the translator can quickly put the incrementally trained machine translation engine into the pre-translation process through the machine translation scheduling system;

[0048] Step 8: Pre-translation. In the pre-translation project, an incrementally trained engine is used to pre-translate the content to be translated. During the translation process in the editor, after the translator inputs part of the content, the incrementally trained engine performs constraint decoding based on this constraint and provides the re-decoded and generated translation in real time.

[0049] Step Nine: Interactive translation. The translator modifies the translation in real time, and the translation model decodes it again based on the modification results to generate a new translation recommendation... This process is repeated until the current sentence segment is translated.

[0050] Step 10: Continuous incremental training. After the current sentence segment is translated, the sentence segment is stored in the translation memory in real time, and the machine translation engine is updated through the real-time online learning algorithm.

[0051] In this way, incremental training and interactive translation methods are seamlessly integrated, enabling a personalized machine engine tailored to the translator and continuously updated during use.

[0052] The personalized machine translation training method in this embodiment solves the cold start problem by incrementally training on the translator's historical corpus. On the other hand, it solves the real-time learning problem by real-time interaction between the translator and the engine after incremental training. The two form a closed-loop data feedback and seamless integration, which significantly reduces the cost of personalized machine translation for translators and improves data flow and machine learning efficiency.

[0053] This embodiment also provides a personalized machine translation training system, which corresponds to the personalized machine translation training method in this embodiment. The personalized machine translation training system includes an incremental training module, an engine quality assessment module, and a translation module. The incremental training module includes an uploading historical corpus module, a corpus preprocessing module, and a corpus review module; the engine quality assessment module includes an automatic quality assessment module and a manual quality assessment module; and the translation module includes a pre-translation module and an interactive translation module.

[0054] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A personalized machine translation training method, characterized in that, Includes the following steps: Step 1: The translator uploads historical corpus data; Step 2: Automatically preprocess the uploaded corpus; Step 3: Manually review the preprocessed corpus; corpus that fails the review is returned to Step 1, and the translator re-uploads the historical corpus. Step 4: Perform incremental training of the model on the approved corpus; the incremental training of the model refers to initializing the parameter statistics of the original model, and then performing iterative training of the parameters to obtain the incrementally trained model; Step 5: For the incrementally trained model, automatically extract a portion of the test set to perform automatic translation quality assessment and obtain the BLEU score for the current domain; Step Six: After the automatic quality assessment is completed, the translator performs a manual quality assessment; Step 7: After the manual quality assessment is passed, the incrementally trained machine translation engine is placed into the pre-translation process through the machine translation scheduling system; Step 8: Pre-translation; In the pre-translation project, the machine translation engine trained incrementally is used to pre-translate the content to be translated. During the translation process in the editor, after the translator inputs part of the content, the engine trained incrementally performs constraint decoding based on the part of the translator's input and provides the re-decoded and generated translation in real time. Step Nine: Interactive Translation; In this interactive translation, the translator modifies the translation in real time, and the incrementally trained machine translation engine re-decodes based on the modification results to generate a new translation recommendation. This interactive translation process in Step Nine is repeated until the current sentence segment is translated. Step 10: Continuous incremental training; The continuous incremental training involves storing the current sentence segment in the translation memory in real time after the translation is completed, and updating the machine translation engine through a real-time online update algorithm. Incremental training and interactive translation methods are seamlessly integrated, enabling a personalized machine engine tailored to the translator and continuously updated during use; The cold start problem is solved by incremental training on the translator's historical corpus, and the real-time learning problem is solved by real-time interaction between the translator and the incrementally trained engine. The two form a closed-loop data feedback and seamless integration, reducing the cost of personalized machine translation for translators.

2. The personalized machine translation training method according to claim 1, characterized in that, Step six also includes returning to step one if the manual quality assessment fails, and the translator re-uploads the historical corpus.

3. The personalized machine translation training method according to claim 1, characterized in that, The corpus preprocessing refers to calculating the similarity between the original text and the edit distance, automatically deduplicating the corpus and removing duplicate text, and simultaneously calculating the semantic similarity between the original text and the translated text in the uploaded corpus and removing the original text and translated text with poor semantic similarity.

4. The personalized machine translation training method according to claim 1, characterized in that, The manual quality assessment refers to comparing the translations from the original machine translation engine and the incrementally trained translation engine.

5. A personalized machine translation training system corresponding to the personalized machine translation training method according to any one of claims 1-4, characterized in that, It includes an incremental training module, an engine quality assessment module, and a translation module; the incremental training module includes an uploading historical corpus module, a corpus preprocessing module, and a corpus review module; the engine quality assessment module includes an automatic quality assessment module and a manual quality assessment module; the translation module includes a pre-translation module and an interactive translation module.

Images