Deep learning-based Chinese-Zhuang language bidirectional machine translation system and construction method

Through the combination of efficient parameter fine-tuning technology and pre-training models, the problems of high training cost and large data demand in Chinese-Zhuang machine translation system are solved, and efficient and low-cost deployment and translation effect of Chinese-Zhuang two-way machine translation system are achieved. Improvement.

CN119026609APending Publication Date: 2024-11-26GUANGXI UNIV FOR NATITIES
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411046374.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing Chinese-Zhuang language machine translation system based on deep learning has problems such as high computing costs, high storage costs, reduced model performance and large data demand when training unknown languages. The huge amount of parameters of the pre-trained model leads to difficulty in deployment.

Method used

Using parameter efficient fine-tuning technology (PEFT), combining pre-training model and PEFT method, select some parameters for training, form a pre-training model-PEFT technology pairing, realize Chinese-Zhuang bidirectional machine translation through less data and computing resources, and use The PyTorch framework is converted into callable binary files and deployed through Gradio or Qt development interactive interface.

Benefits of technology

It significantly reduces computational costs and storage costs, improves model performance, reduces training data requirements, realizes rapid deployment and model switching, and ensures the quality and efficiency of translation effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026609A_ABST
    Figure CN119026609A_ABST
Patent Text Reader

Abstract

The invention provides a Chinese-Zhuang language two-way machine translation system based on deep learning and a construction method, a system interaction front end formed by a user and a Gradio or Qt GUI is used for interacting and inputting a text needing to be translated to the interaction front end, and the interaction front end preprocesses the input of the user and removes redundant blank texts; the interaction front end interacts with the core component, transmits the processed user input to the core component, and prepares to receive a corresponding translation text output by the core component; the core component calls a corresponding Chinese-Zhuang language word segmentation device, a pre-training model and one or more corresponding candidate PEFT models according to a text input by the interaction front end, and transmits the text subjected to word segmentation by the Chinese-Zhuang language word segmentation device to one or more combined PEFT models to obtain a single-model translation result or a multi-model joint translation result; and the interaction front end pulls the translation result output by the core component and displays the translation result to a user. According to the method, Chinese-Zhuang language two-way smooth, accurate and rapid translation is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention provides a Chinese-Zhuang bidirectional machine translation system based on deep learning and a construction method thereof, belonging to the technical field of translation thereof. Background Art

[0002] Machine translation technology is an automatic language conversion technology that combines computer science, artificial intelligence and natural language processing to achieve cross-language conversion of texts. It has undergone evolution from rule-based, statistical and neural network-based. At present, neural network-based machine translation technology has been widely used and become a mainstream technology due to its high performance, high quality and high efficiency. Neural network-based machine translation technology consists of three basic components:

[0003] Corpus: Contains a large number of corresponding text pairs in the source language and the target language, also known as a parallel corpus.

[0004] Model: Models used for machine translation tasks usually adopt an encoder-decoder model architecture.

[0005] Word segmenter: word segmentation rules for bilingual languages ​​obtained by processing a bilingual parallel corpus with a statistical algorithm.

[0006] In the early stages of development, machine translation systems based on deep learning technology only used bilingual corpora segmented by word segmenters and were trained from scratch using a specific structural model. That is, each translation direction (such as Chinese-English, Chinese-French) corresponded to a specific model.

[0007] After the concept of pre-training became popular, machine translation systems based on deep learning no longer trained a model of a specific structure from scratch, but used pre-trained models that were pre-trained on multilingual data. The pre-training process injects multilingual knowledge into the model parameters, and then uses bilingual corpora of a certain translation direction for further training, using the multilingual knowledge of the pre-trained model to learn the conversion relationship between the two languages. The above method is summarized as a new paradigm: pre-training + downstream task fine-tuning.

[0008] However, there are three major drawbacks to using pre-trained models for further training on a model-unknown language such as Zhuang:

[0009] (1) Training for unknown languages ​​requires adjusting all parameters in the pre-trained model (i.e., full fine-tuning), which may damage the knowledge learned by the pre-trained model and cause the model's performance to degrade in other languages.

[0010] (2) Full fine-tuning requires saving a complete set of model parameters for each newly added language, that is, a model dedicated to Chinese-Zhuang translation. However, the pre-trained model files are usually very large, and saving these fine-tuned models requires a lot of storage space.

[0011] (3) The number of parameters of the pre-trained model is very large, usually hundreds of millions to hundreds of billions. Adjusting all the parameters during the training process will require a large amount of GPU RAM, even reaching 780GB at one point (NVIDIA A100 GPU RAM has only 80GB). Such a high computing cost is unaffordable for most people.

[0012] Due to the above problems with full-scale fine-tuning, in recent years, many studies have proposed a new technology called Parameter-Efficient Fine-Tuning (PEFT) to reduce computing costs and storage costs while ensuring that model performance is almost unaffected.

[0013] Parameter efficient fine-tuning technology is a method of optimization based on pre-trained models, which aims to improve the generalization ability of the model on new tasks while reducing training time and computing resources. Parameter efficient fine-tuning technology does not need to adjust all model parameters in the pre-trained model. It selects a small group of parameters in the pre-trained model, or introduces a small group of new untrained parameters, while keeping most or all parameters in the pre-trained model fixed, and only trains the selected parameters or newly introduced parameters. Parameter efficient fine-tuning technology has been proven to significantly reduce computing costs, reduce storage costs, and achieve model performance comparable to full fine-tuning.

[0014] Using a randomly initialized model, we directly perform word segmentation and model training on the Chinese-Zhuang bilingual data to obtain a Chinese-Zhuang machine translation model. This technology has the following disadvantages:

[0015] (1) It takes a lot of experiments to determine which model structure is best.

[0016] (2) A large amount of bilingual data is required, but it is difficult to collect the amount of data required to achieve the target effect for Zhuang language.

[0017] (3) A large number of experiments are required to determine the optimal training configuration, which is computationally expensive.

[0018] In addition, there is also a pre-trained model with good performance in multilingual translation, which is used to segment words and fine-tune the model in full on a small amount of high-quality bilingual data to obtain a Chinese-Zhuang bidirectional machine translation model. This technology has the following disadvantages:

[0019] (1) Pre-trained models with good performance often have a huge number of parameters. Full fine-tuning requires a large amount of GPU RAM, and the computational cost is very high.

[0020] (2) The resulting model often requires a large amount of storage space and is difficult to deploy.

[0021] (3) A large number of experiments are required to determine the optimal training configuration, and the computational cost is usually higher than that of the method in technique 1. Summary of the invention

[0022] In view of the above technical problems, the present invention provides a Chinese-Zhuang bidirectional machine translation system and construction method based on deep learning to achieve the following objectives:

[0023] (1) Achieve fluent, accurate and fast translation between Chinese and Zhuang.

[0024] (2) Reduce the computational cost, storage cost, and deployment cost of model training, use a better model at a lower cost, and ensure that the model's performance is almost unchanged.

[0025] (3) Make full use of the knowledge learned by the pre-trained model to achieve better translation results with less data.

[0026] (4) The word segmenter can correctly segment Zhuang texts, and enable the pre-trained model to learn relevant language knowledge of Zhuang without destroying the original parameters of the pre-trained model as much as possible, so that it can correctly recognize Zhuang.

[0027] To achieve the above objectives, the technical solution adopted by the present invention is:

[0028] A Chinese-Zhuang bidirectional machine translation system based on deep learning, including:

[0029] Interactive front end, the system interactive front end consisting of the user and Gradio or Qt GUI, is used for interaction. The text to be translated is input into the interactive front end, which pre-processes the user input and removes redundant blank text.

[0030] Core component, the interactive front end interacts with the core component, passes the processed user input to the core component, and is ready to receive the corresponding translation text output by the core component;

[0031] It also includes a Chinese-Zhuang word segmenter, a pre-trained model, and one or more corresponding candidate PEFT models; the core component calls the corresponding Chinese-Zhuang word segmenter, the pre-trained model, and one or more corresponding candidate PEFT models according to the text input by the interactive front end, and passes the text after word segmentation by the Chinese-Zhuang word segmenter to the combined one or more PEFT models to obtain a single model translation result or a multi-model joint translation result; the interactive front end pulls the translation result output by the core component and displays it to the user.

[0032] The method for constructing a Chinese-Zhuang bidirectional machine translation system based on deep learning includes the following steps:

[0033] (1) Determine the components to be used: Select a model suitable for the Chinese-Zhuang bidirectional translation task; Apply the PEFT method to the training process of the Chinese-Zhuang machine translation model, and combine several alternative pre-trained models and PEFT technical methods in pairs to form a pre-trained model-PEFT technology pairing as an alternative solution;

[0034] (2) Data collection and corpus construction: We collected Chinese-Zhuang bilingual texts from the Internet, books, and literature, divided the bilingual texts into sentences, removed sentences with less than 5 words and more than 512 words, and aligned them to ensure that the Chinese texts and Zhuang texts corresponded one to one, and constructed a parallel corpus.

[0035] (3) Training model and model screening: Experiments are conducted on the pre-trained model-PEFT technology pairing. The translation results generated by the trained machine translation model are scored by machine using the evaluation indicators commonly used in machine translation tasks. The results are then reviewed and scored manually, and the results together with the machine scores are used as the final evaluation results. Based on the evaluation results, the model is adjusted and optimized, and the model with the highest score is used as the final model.

[0036] (4) Model deployment and software engineering: One or more final models obtained in the model training phase are converted into callable binary files using relevant methods in the PyTorch deep learning framework. A user interaction interface is developed through software development tools and integrated with the model binary file to complete model deployment and software development. The software is put into use and optimized based on real-time feedback from users in actual usage scenarios.

[0037] Among them, the model training method:

[0038] (3.1) Word segmentation training stage:

[0039] Select a word segmenter of a language close to Zhuang, use the algorithm to learn the bilingual corpus data separately, and add the Chinese-Zhuang dictionary as supplementary information to obtain a word segmenter that can correctly segment Chinese-Zhuang words;

[0040] (3.2) Data preprocessing stage:

[0041] Use the obtained Chinese-Zhuang word segmenter to segment Chinese-Zhuang bilingual text pairs and convert them into data acceptable to the model;

[0042] (3.3) Model training and evaluation phase:

[0043] Experiments were conducted based on the pre-trained model-PEFT technology pairing obtained during the technical research phase. The BLEU word-level indicator and CHRF character-level indicator commonly used in machine translation tasks were used to score the translation results generated by the model. The scores were automatically generated by the script, and practitioners of Chinese-Zhuang translation were asked to score the translation quality at the sentence level. The above scores were combined as the final score of the model. The training process was adjusted according to the final score. The above process was repeated, and finally all pre-trained model-PEFT technology pairs were ranked according to the scores. The score calculation formula is as follows:

[0044] Final Score=0.1×CHRF+0.2×BLEU+0.7×Human Score

[0045] (3.4) Select one or more pre-trained model-PEFT technology pairs with the highest scores to form the final Chinese-Zhuang machine translation model. Only one pre-trained model and a PEFT model with extremely small parameters need to be saved.

[0046] The technical effects of the present invention are as follows:

[0047] (1) Use models with larger parameters and better effects at lower computational, storage, and deployment costs, with almost no loss in model performance.

[0048] (2) The training cost is greatly reduced, and multiple high-performance PEFT models with small parameters and small storage space requirements can be quickly obtained.

[0049] (3) During deployment, since the PEFT model is very small (compared to the pre-trained model), model switching can be performed quickly to obtain different model outputs.

[0050] (4) The required training data is consistent with that required for full fine-tuning, and much less data is required than training a randomly initialized model from scratch. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the structure of the Chinese-Zhuang machine translation system of the present invention;

[0052] Figure 2 It is a flow chart of constructing the Chinese-Zhuang machine translation system of the present invention;

[0053] Figure 3 It is a flow chart of the training of the Chinese-Zhuang machine translation model of the present invention. DETAILED DESCRIPTION

[0054] The specific technical solution of the present invention is described in conjunction with the accompanying drawings.

[0055] like Figure 1As shown in the figure, the Chinese-Zhuang bidirectional machine translation system based on deep learning includes:

[0056] Interactive front end, the system interactive front end consisting of the user and Gradio or Qt GUI, is used for interaction. The text to be translated is input into the interactive front end, which pre-processes the user input and removes redundant blank text.

[0057] Core component, the interactive front end interacts with the core component, passes the processed user input to the core component, and is ready to receive the corresponding translation text output by the core component;

[0058] It also includes a Chinese-Zhuang word segmenter, a pre-trained model, and one or more corresponding candidate PEFT models; the core component calls the corresponding Chinese-Zhuang word segmenter, the pre-trained model, and one or more corresponding candidate PEFT models according to the text input by the interactive front end, and passes the text after word segmentation by the Chinese-Zhuang word segmenter to the combined one or more PEFT models to obtain a single model translation result or a multi-model joint translation result; the interactive front end pulls the translation result output by the core component and displays it to the user.

[0059] A method for constructing a Chinese-Zhuang bidirectional machine translation system based on deep learning, such as Figure 2 As shown, the following steps are included:

[0060] (1) Determine the components to be used: Through technical research, select commonly used pre-training models and PEFT methods; compare and analyze existing pre-training models that support multiple languages, such as MBART, MT0[, and NLLB-200-Distilled, which have been proven to have excellent performance in multiple languages, and select models suitable for Chinese-Zhuang bidirectional translation tasks; study a series of commonly used PEFT methods such as LoRA, P-Tuning, and Prefix Tuning, and apply them to the training process of the Chinese-Zhuang machine translation model; combine several alternative pre-training models and PEFT technical methods in pairs to form a (pre-training model, PEFT technology) pairing as an alternative experimental plan;

[0061] (2) Data collection and corpus construction: We collected Chinese-Zhuang bilingual texts from the Internet, books, and literature using web crawlers and manual collation. We divided the bilingual texts into sentences, removed short (less than 5 words) and long (more than 512 words) sentences, and aligned them to ensure that the Chinese and Zhuang texts corresponded one to one, thus building a parallel corpus.

[0062] (3) Training model and model screening: Experiments are conducted on the pairs (pre-trained models, PEFT technology) determined in the technical research phase. Common evaluation indicators for machine translation tasks such as BLEU and CHRF are used to perform machine scoring on the translation results generated by the trained machine translation model. Manual review and scoring are then performed, and the results together with the machine scores are used as the final evaluation results. Based on the evaluation results, the model is adjusted and optimized to improve the translation quality, and the model with the highest score is used as the final model.

[0063] (4) Model deployment and software engineering: Convert one or more final models obtained in the model training phase into callable binary files using relevant methods in the PyTorch deep learning framework, develop a user interaction interface through software development tools such as the Gradio library or Qt, integrate it with the model binary file, complete model deployment and software development, put the software into use, and optimize the software based on real-time feedback from users in actual usage scenarios.

[0064] The model training method in the present invention is as follows Figure 3 :

[0065] (3.1) Tokenizer training stage: Since Zhuang is an unknown language for the pre-training model, the tokenizer needs to correctly segment the training data to ensure that the model can correctly learn language knowledge. A tokenizer for a language close to Zhuang (such as Thai) is selected, and the bilingual corpus data is learned using algorithms such as BPE, WordPiece, and ULM. A Chinese-Zhuang dictionary is added as supplementary information to obtain a tokenizer that can correctly segment Chinese and Zhuang words.

[0066] (3.2) Data preprocessing stage: Use the obtained Chinese-Zhuang word segmenter to segment the Chinese-Zhuang bilingual text pairs and convert them into data acceptable to the model;

[0067] (3.3) Model training and evaluation phase: Experiments were conducted based on the (pre-trained model-PEFT technology) pairings obtained in the technical research phase. The BLEU (word level) and CHRF (character level) indicators commonly used in machine translation tasks were used to score the translation results generated by the model. The scores were automatically generated by the script, and relevant practitioners of Chinese-Zhuang translation were asked to score the translation quality (sentence level). The above scores were combined as the final score of the model. The training process was adjusted according to the final score. The above process was repeated, and finally all (pre-trained model-PEFT technology) pairs were ranked according to the scores. The score calculation formula is as follows:

[0068] Final Score=0.1×CHRF+0.2×BLEU+0.7×Human Score

[0069] (3.4) Select one or more (pre-trained model-PEFT technology) pairs with the highest scores to form the final Chinese-Zhuang machine translation model. Only one pre-trained model and a PEFT model with extremely small parameters need to be saved.

Claims

1. A Chinese-Zhuang bidirectional machine translation system based on deep learning, It is characterized in that include: Interactive front end, the system interactive front end consisting of the user and Gradio or Qt GUI, is used for interaction. The text to be translated is input into the interactive front end, which pre-processes the user input and removes redundant blank text. Core component, the interactive front end interacts with the core component, passes the processed user input to the core component, and is ready to receive the corresponding translation text output by the core component; It also includes a Chinese-Zhuang word segmenter, a pre-trained model, and one or more corresponding candidate PEFT models; the core component calls the corresponding Chinese-Zhuang word segmenter, the pre-trained model, and one or more corresponding candidate PEFT models according to the text input by the interactive front end, and passes the text segmented by the Chinese-Zhuang word segmenter to the combined one or more PEFT models to obtain a single model translation result or a multi-model joint translation result; The interactive front end pulls the translation results output by the core component and displays them to the user.

2. Construction method of Chinese-Zhuang bidirectional machine translation system based on deep learning, It is characterized in that The following steps are involved: (1) Determine the components to be used: Select a model suitable for the Chinese-Zhuang bidirectional translation task; Apply the PEFT method to the training process of the Chinese-Zhuang machine translation model, and combine several alternative pre-trained models and PEFT technical methods in pairs to form a pre-trained model-PEFT technology pairing as an alternative solution; (2) Data collection and corpus construction: We collected Chinese-Zhuang bilingual texts from the Internet, books, and literature, divided the bilingual texts into sentences, removed sentences with less than 5 words and more than 512 words, and aligned them to ensure that the Chinese texts and Zhuang texts corresponded one to one, and constructed a parallel corpus. (3) Training model and model screening: Experiments are conducted on the pre-trained model-PEFT technology pairing. The translation results generated by the trained machine translation model are scored by machine using the evaluation indicators commonly used in machine translation tasks. The results are then reviewed and scored manually, and the results together with the machine scores are used as the final evaluation results. Based on the evaluation results, the model is adjusted and optimized, and the model with the highest score is used as the final model. (4) Model deployment and software engineering: One or more final models obtained in the model training phase are converted into callable binary files using relevant methods in the PyTorch deep learning framework. A user interaction interface is developed through software development tools and integrated with the model binary file to complete model deployment and software development. The software is put into use and optimized based on real-time feedback from users in actual usage scenarios. Among them, the model training method: (3.1) Word segmentation training stage: Select a word segmenter of a language close to Zhuang, use the algorithm to learn the bilingual corpus data separately, and add the Chinese-Zhuang dictionary as supplementary information to obtain a word segmenter that can correctly segment Chinese-Zhuang words; (3.2) Data preprocessing stage: Use the obtained Chinese-Zhuang word segmenter to segment Chinese-Zhuang bilingual text pairs and convert them into data acceptable to the model; (3.3) Model training and evaluation phase: Experiments were conducted based on the pre-trained model-PEFT technology pairing obtained during the technical research phase. The BLEU word-level indicator and CHRF character-level indicator commonly used in machine translation tasks were used to score the translation results generated by the model. The scores were automatically generated by the script, and practitioners of Chinese-Zhuang translation were asked to score the translation quality at the sentence level. The above scores were combined as the final score of the model. The training process was adjusted according to the final score. The above process was repeated, and finally all pre-trained model-PEFT technology pairs were ranked according to the scores. The score calculation formula is as follows: Final Score=0.1×CHRF+0.2×BLEU+0.7×Human Score (3.4) Select one or more pre-trained model-PEFT technology pairs with the highest scores to form the final Chinese-Zhuang machine translation model. Only one pre-trained model and a PEFT model with extremely small parameters need to be saved.