A machine translation model training method and system based on parameter pruning and tuning
Through the parameter pruning and tuning method, the catastrophic forgetting problem of multilingual neural machine translation model in continuous learning is solved, and new language knowledge is learned while the primitive translation performance is not degraded, which improves the model's continuous learning ability .
Patent Information
- Application Number
- CN202410963942.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-07-18
AI Technical Summary
Multilingual neural machine translation models face catastrophic forgetting problems during continuous learning, resulting in a decline in original translation performance. Existing methods such as playback method, adapter method and parameter isolation method have large training data sets, long training time or increased model size The problem.
Using a machine translation model training method based on parameter pruning tuning, the pruning model is constructed by obtaining incremental data sets and pseudo-target language data sets, evaluating the importance of FFN layer parameters, replacing unimportant parameters, and using the optimized CPO The algorithm fine-tunes the FFN layer to form a new model after parameter pruning.
Effectively solve the problem of continuous learning and catastrophic forgetting of the machine translation model of Transformer architecture, ensuring that the translation performance of the primitive language does not decline, and at the same time learn new language knowledge, improve the continuous learning ability of the model, and verify the new model through multiple benchmark test sets. effectiveness.
Smart Images
Figure CN118504642B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine translation, and in particular to a machine translation model training method and system based on parameter pruning and tuning. Background Art
[0002] The ability of a machine translation model to continuously learn new translation directions through a newly added data set is called continuous learning, which can continuously expand the types of translations that can be performed by the machine translation model without re-establishing a new translation model. The simplest method is to merge the original data set and the newly added data set as a new training set to train the original model. However, in actual operation, the original data cannot be used due to user privacy, damage to the original data, etc., and retraining the model with a huge original data set will cause a huge waste of time and cost. Therefore, the training goal of the multilingual neural machine translation model is to adjust the model to adapt to the new language or improve the translation performance of the existing language through the newly added data set without accessing the original training data. However, the multilingual neural machine translation model will face the practical challenge of catastrophic forgetting in the process of continuous learning, which will cause the performance of the original translation model to decrease while learning the new translation direction. In order to solve this problem, the following methods are currently used: 1. The playback method uses the generation ability of the multilingual translation model to create pseudo data, merge the pseudo language data generated by the original model with the extended data of the new language, and use the merged data set to train the original model. However, this method has the disadvantages of large training data sets and long training time. 2. The adapter method, which involves adding new network parameters to the existing model to train a new language. The problem is that the introduction of new model parameters will increase the size of the model, which is not conducive to the simplification of the translation model. 3. The parameter isolation method, which involves allocating specific parameters in the model to translate different languages into specific directions. This method will cause the translation performance of the original language and the new language to decrease to varying degrees. Summary of the invention
[0003] The first object of the present invention is to provide a machine translation model training method based on parameter pruning and tuning, so that the machine translation model has good continuous learning ability.
[0004] The second object of the present invention is to provide a machine translation system based on parameter pruning and tuning.
[0005] To achieve the above object, the first technical solution adopted by the present invention is a machine translation model training method based on parameter pruning and tuning, comprising the following steps:
[0006] S01. Acquire training incremental data and process it to obtain an incremental data set; acquire the original machine translation model to be trained and tuned; acquire a pseudo target language data set, use special text containing target language information as input to the encoder, and use the generation capability of the original machine translation model to generate a pseudo target language data set in the original translation direction.
[0007] S02. Obtain a fine-tuning dataset, use the original machine translation model to translate the pseudo target language dataset into the original language dataset, merge the pseudo target language dataset and the original language dataset into a pseudo dataset, and then merge the pseudo dataset with the incremental dataset of step S01 into a fine-tuning dataset (x, y), where x represents the source language and y represents the target language; use the incremental dataset to fine-tune the original machine translation model to obtain an incremental model.
[0008] S03. Construct a pruned model, evaluate the importance of each parameter in the FFN layer of the original machine translation model of step S01 and the incremental model of step S02, find out the parameters in the FFN layer of the original machine translation model that are important for the original language translation, and the parameters in the FFN layer of the incremental model that are unimportant for the incremental language translation, replace the parameters in the FFN layer of the incremental model that are unimportant for the incremental language translation but important for the original language translation with the parameters in the FFN layer of the original machine translation model that are important for the original language translation, so as to obtain the pruned FFN layer, and add the pruned FFN layer to the original machine translation model to construct the pruned model.
[0009] S04. Use the fine-tuning dataset to construct a four-tuple fine-tuning dataset.
[0010] S05. Freeze the parameters of the original machine translation model in the pruned model described in step S03, and use the optimized CPO algorithm using the four-tuple fine-tuning dataset to fine-tune the FFN layer in the pruned model, so as to obtain a new model after parameter pruning and tuning.
[0011] Specifically, in step S01, the acquired training incremental data is data of new translation direction obtained from a public machine translation text database. When processing the training incremental data, the Moses toolkit is first used for word segmentation to divide the original sentence into words or sub-words, and then the BPE algorithm is used to pre-process the training incremental data after word segmentation to obtain a vocabulary and segmented sub-word data. By this method, an incremental data set can be obtained.
[0012] Specifically, when using the incremental data set to fine-tune the original machine translation model to obtain the incremental model in step S02, the following steps are specifically adopted: using the parallel statements of the incremental data set as the input of the source statement and the target statement of the original machine translation model respectively, fine-tuning the original machine translation model and updating the weight parameters therein to obtain the incremental model.
[0013] Specifically, in step S03, when evaluating the importance of each parameter in the FFN layer of the original machine translation model and the incremental model, a method based on dual importance is used for evaluation.
[0014] Specifically, when evaluating the importance of each parameter in the FFN layer of the original machine translation model, the absolute value method is used for evaluation. The calculation formula for the importance of the parameter in the i-th row and j-th column of the FFN layer of the original machine translation model is: ,in Indicates the weight of the i-th row and j-th column of the FFN layer of the original machine translation model. When evaluating the importance of each parameter in the FFN layer of the incremental model, the absolute value difference method is used for evaluation. The calculation formula for the importance of the parameter of the i-th row and j-th column of the FFN layer of the incremental model is: ,in , is the weight of the i-th row and j-th column of the FFN layer of the incremental model.
[0015] Specifically, in step S03, when the pruned FFN layer is added to the original machine translation model to construct the pruned model, the FFN layer of the original machine translation model and the pruned FFN layer are placed in parallel, and a linear layer is newly added after the two FFN layers. The outputs of the parallel FFN layer of the original machine translation model and the pruned FFN layer are combined as the input of the linear layer, and the parallel FFN layer of the original machine translation model, the pruned FFN layer and the linear layer are combined into the FFN layer of the pruned model, and the FFN layer of the pruned model replaces the FFN layer of the original machine translation model to obtain the pruned model.
[0016] Specifically, when the fine-tuning dataset is used to construct the four-tuple fine-tuning dataset in step S04, the following steps are specifically adopted: the large model GPT-4 and Google Translate are used to translate x in the fine-tuning dataset into target sentences tgt_sentence2 and tgt_sentence3 respectively; the model KIWI-XXL is used to score the tgt_sentence2, tgt_sentence3 sentences and the sentence y in the fine-tuning dataset respectively, and the sentences with high scores are named y_h, the sentences with medium scores are named y_m, and the sentences with low scores are named y_l, so as to obtain the four-tuple fine-tuning dataset (x, y_h, y_m, y_l).
[0017] Specifically, in step S05, the loss calculation formula of the optimized CPO algorithm is as follows:
[0018] L CPO-M = - E (x, y_h, y_m, y_l)~D [ log σ( β 1 log π θ (y_h|x) - β 2 log π θ (y_m|x) - β 3 log π θ (y_l|x))]- E (x, y_h)~D [log π θ (y_h|x)] ;in, is the model parameter, σ is the Sigmoid function, , , is a hyperparameter.
[0019] Specifically, in step S05, when the FFN layer of the pruned model is fine-tuned using the optimized CPO algorithm using the four-tuple fine-tuning dataset, the loss calculation formula of the optimized CPO algorithm is used as the loss function of the parameters of the FFN layer of the pruned model and the four-tuple fine-tuning dataset is used for fine-tuning calculation.
[0020] Specifically, after obtaining the new model after parameter pruning and tuning in step S05, the new model is tested using multiple benchmark test sets.
[0021] The second technical solution adopted by the present invention is: a machine translation model training system based on parameter pruning and tuning, including a preprocessing module, a machine translation model construction module, a training fine-tuning module and a translation module;
[0022] The preprocessing module is used to obtain an incremental data set and a multilingual translation original model, generate pseudo original language data based on the original model, merge the pseudo original language data set and the incremental data set into a fine-tuning data set, and use the fine-tuning data set to construct a four-tuple fine-tuning data set;
[0023] The machine translation model construction module is used to construct a machine translation model. The machine translation model adopts a Transformer architecture, including a position encoder, an input embedding layer, an encoder multi-head self-attention block, a feedforward neural network, a decoder multi-head self-attention block, and an output processing layer. The feedforward neural network includes an original model FFN layer, a pruned FFN layer, and a linear layer. The original model FFN layer and the pruned FFN layer are connected in parallel and then connected to a linear layer.
[0024] The training and fine-tuning module is used to input the source language sentences of the fine-tuning data set into the machine translation model to obtain the translated target language sentences, and train and fine-tune the machine translation model based on the translated target language sentences using the CPO-M optimization method to obtain a trained and fine-tuned machine translation model;
[0025] The translation module is used to translate source language sentences based on the trained and fine-tuned machine translation model.
[0026] The beneficial effects of the present invention are:
[0027] 1. The method of parameter pruning and model fine-tuning solves the problem of how the Transformer architecture machine translation model can perform continuous learning and catastrophic forgetting during the continuous learning process. While ensuring that the performance of the machine translation model in the original language translation direction does not decline, it can learn knowledge in the new language translation direction and ensure good translation performance in the new translation direction.
[0028] 2. The optimized CPO algorithm is used to fine-tune and optimize the FFN layer of the pruned model, so that the FFN layer of the pruned model can retain the knowledge of the original language while including new knowledge, which can effectively improve the model's continuous learning ability.
[0029] 3. Extensive testing using multiple benchmark test sets is performed to ensure the effectiveness of the new model after fine-tuning and training in both the original and new translation directions. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Attached Figure 1 It is an overall flow chart of the machine translation model training method based on parameter pruning and tuning in the embodiment.
[0031] Attached Figure 2 This is a flowchart of obtaining a new model after parameter pruning and tuning by using the machine translation model training method based on parameter pruning and tuning in the embodiment.
[0032] Attached Figure 3 This is a flow chart of fine-tuning the original model to obtain an incremental model in step S02 in the embodiment.
[0033] Attached Figure 4 This is a flow chart of obtaining the pruned FFN layer in step S03 in the embodiment.
[0034] Attached Figure 5 This is a flow chart of obtaining the pruning model in step S03 in the embodiment.
[0035] Attached Figure 6 A flowchart of a machine translation system based on parameter pruning and tuning is provided in an embodiment. DETAILED DESCRIPTION
[0036] Example 1, reference Figure 1-2 , a machine translation model training method based on parameter pruning and tuning, comprising the following steps:
[0037] S01, obtain training incremental data and process to obtain incremental data sets; obtain the original machine translation model to be tuned and trained; obtain a pseudo target language data set, use the special text containing the target language information as the input of the encoder, and use the generation ability of the original machine translation model to generate a pseudo target language data set in the original translation direction. In this embodiment, the original machine translation model to be tuned and trained can use the existing Transformer architecture multilingual machine translation model, or use the fairseq framework to train a Transformer architecture multilingual machine translation model. Among them, the acquired training incremental data is the data of the new translation direction obtained from the public machine translation text database. When processing the training incremental data, the Moses toolkit is first used for word segmentation processing to divide the original sentence into words or subwords, and then the BPE algorithm is used to pre-process the training incremental data after word segmentation processing, and the word list and the segmented subword data are obtained after processing. By this method, an incremental data set can be obtained.
[0038] S02. Obtain a fine-tuning dataset, use the original machine translation model to translate the pseudo target language dataset into the original language dataset, merge the pseudo target language dataset and the original language dataset into a pseudo dataset, and then merge the pseudo dataset with the incremental dataset of step S01 into a fine-tuning dataset (x, y), where x represents the source language and y represents the target language; use the incremental dataset to fine-tune the original machine translation model to obtain an incremental model. When using the incremental dataset to fine-tune the original machine translation model to obtain an incremental model, the following steps are specifically used: use the parallel statements of the incremental dataset as the source sentences, target sentences, and original sentences of the original machine translation model, respectively, and fine-tune the original machine translation model and update the weight parameters therein to obtain the incremental model. Reference Figure 3 The original machine translation model consists of five parts, namely, input embedding layer, position encoder, encoder, decoder and output processing layer. The source language x in the incremental data set is converted into a numerical sequence that can be processed by the machine through the input embedding layer, and then the position information of the input text is added to the numerical sequence through the position encoder to obtain the input sequence. The encoder can capture the long-distance dependencies and complex semantic information in the input sequence. The decoder combines the output of the encoder and the target language y in the incremental data set to predict the next word in the target sequence. In this way, the original machine translation model is fine-tuned and the weight parameters are updated to obtain the incremental model, so that the final output text is closer to the target language y in the incremental data set.
[0039] S03, build pruning model, refer to Figure 4 , respectively evaluate the importance of each parameter in the FFN layer of the original machine translation model of step S01 and the incremental model of step S02, find out the parameters in the FFN layer of the original machine translation model that are important to the original language translation, and the parameters in the FFN layer of the incremental model that are unimportant to the incremental language translation, replace the parameters in the FFN layer of the incremental model that are unimportant to the incremental language translation but important to the original language translation with the parameters in the FFN layer of the original machine translation model that are important to the original language translation, so as to obtain the pruned FFN layer, and add the pruned FFN layer to the original machine translation model to construct a pruned model.
[0040] When evaluating the importance of each parameter in the FFN layer of the original machine translation model and the incremental model, a method based on dual importance is used for evaluation. Specifically, when evaluating the importance of each parameter in the FFN layer of the original machine translation model, the absolute value method is used for evaluation. The calculation formula for the importance of the parameter in the i-th row and j-th column of the FFN layer of the original machine translation model is: ,in Indicates the weight of the i-th row and j-th column of the FFN layer of the original machine translation model. When evaluating the importance of each parameter in the FFN layer of the incremental model, the absolute value difference method is used for evaluation. The calculation formula for the importance of the parameter of the i-th row and j-th column of the FFN layer of the incremental model is: ,in , is the weight of the i-th row and j-th column of the FFN layer of the incremental model.
[0041] Reference Figure 5 When the pruned FFN layer is added to the original machine translation model to construct the pruned model, the FFN layer of the original machine translation model and the pruned FFN layer are placed in parallel, a linear layer is newly added after the two FFN layers, the outputs of the parallel FFN layer of the original machine translation model and the pruned FFN layer are combined as the input of the linear layer, the parallel FFN layer of the original machine translation model, the pruned FFN layer and the linear layer are combined into the FFN layer of the pruned model, and the FFN layer of the pruned model replaces the FFN layer of the original machine translation model to obtain the pruned model.
[0042] S04. Use the fine-tuning dataset to construct a four-tuple fine-tuning dataset, specifically using the following steps: use the large model GPT-4 and Google Translate to translate x in the fine-tuning dataset into target sentences tgt_sentence2 and tgt_sentence3 respectively; use the model KIWI-XXL to score the tgt_sentence2, tgt_sentence3 sentences and the sentence y in the fine-tuning dataset respectively, name the sentences with high scores y_h, the sentences with medium scores y_m, and the sentences with low scores y_l, and obtain the four-tuple fine-tuning dataset (x, y_h, y_m, y_l).
[0043] S05. Freeze the parameters of the original machine translation model in the pruned model described in step S04, and use the optimized CPO algorithm to fine-tune the FFN layer in the pruned model using the four-tuple fine-tuning dataset to obtain a new model after parameter pruning and tuning.
[0044] Among them, the loss calculation formula of the optimized CPO algorithm is as follows:
[0045] L CPO-M = - E (x, y_h, y_m, y_l)~D [ log σ( β 1 log π θ (y_h|x) - β 2 log π θ (y_m|x) - β 3 log π θ (y_l|x))]- E (x, y_h)~D [log π θ (y_h|x)] ;in, is the model parameter, σ is the Sigmoid function, , , When the FFN layer of the pruned model is fine-tuned using the optimized CPO algorithm using the quaternary fine-tuning dataset, the loss calculation formula of the optimized CPO algorithm is used as the loss function of the parameters of the FFN layer of the pruned model and the quaternary fine-tuning dataset is used for fine-tuning calculation.
[0046] In addition, after obtaining the new model after parameter pruning and tuning in step S05, the new model is tested using multiple benchmark test sets to ensure the effectiveness of the training.
[0047] In order to prove the effectiveness of the training method of the present application, the present embodiment sets the following experiment for verification. Specifically, the test sets of English-German, English-Chinese, and English-Romanian in the WMT2023 dataset are selected for testing. The original languages of the original machine translation model of the Transformer architecture are English-German and English-Chinese. English-Romanian is added as the target language. The datasets used in the experiment are shown in the following table:
[0048] Dataset Training set Validation set Test Set English - German 2.9G 3K 3K English-Chinese 6.9G 3K 3K English - Romanian 316M 3K 3K
[0049] During the experiment, in order to accurately evaluate the translation performance of the machine translation model, BLEU was used as an evaluation indicator. The BLEU indicator can accurately show the effect of translation. This experiment used the adapter method commonly used in the field of continuous learning and compared it with the training method of this application. At the same time, the machine translation model of the Transformer architecture obtained by training English-German, English-Chinese, and English-Romanian was added as a comparison benchmark, and the following experimental results were obtained:
[0050]
[0051] From the above experimental results, it can be seen that the translation performance of the machine translation model obtained by the training method of this embodiment can not only retain the translation performance of the original translation direction, but also have good learning ability for the newly added translation direction; while the model obtained by the adapter method has relatively weak learning ability in the new translation direction, and the translation ability in the original translation direction decreases. It can be seen that the training method of this embodiment has good learning ability and can effectively retain the performance of the original translation direction.
[0052] In order to prove the effectiveness of the training method and pruning model in this application, this embodiment sets up the following comparative experiments for verification, where KTNMT is a multilingual machine translation model based on knowledge transfer, and its approach is to replace the newly added FFN layer in the pruning model with the FFN layer in the translation model trained from scratch; DIMD is a machine translation model based on dual importance division, and its approach is to perform dual importance division on all parameters in the entire network, and the network retains parameters that are important to the original language and parameters that are important to the newly added language. The results obtained on the test set are as follows:
[0053]
[0054] From the above experimental results, it can be seen that since the original language of the KTNMT algorithm is translated through the translation path of the original model, the translation effect of the original language will not decrease, but the translation effect of the newly added language is the worst. This is due to the fact that the important parameters for the newly added language have changed significantly during the subsequent fine-tuning process. The DIMD model prunes the entire network, so its parameters retain a large number of parameters that are important for the translation of the newly added language, so it achieves the highest score in the translation of the newly added language, but it also causes it to have the lowest score in the translation of the two original languages. Using the pruned model in the training method of this embodiment, the highest score is achieved in the translation direction of the original language, and it is only 0.26 points lower than the DIMD model in the translation direction of the newly added language. Compared with the increase in the original translation direction, it is acceptable to be lower than the DIMD model in the translation direction of the newly added language. The above experiments prove the good performance of the pruned model in the training method of this embodiment.
[0055] In order to prove the effectiveness of the optimized CPO algorithm (CPO-M) in the training method of this application, this embodiment sets up the following ablation experiment for verification. The results are as follows:
[0056]
[0057] In the above experiment, different optimization methods are compared respectively. CPO optimization is to use only y_h and y_l when calculating the loss of CPO, without calculating the y_m part; CPO-M is to use y_h, y_m and y_l when calculating the loss of CPO. From the above experimental results, the CPO-M optimization algorithm has different degrees of improvement compared with the original CPO algorithm. The above experiment proves the good performance of CPO-M in the training method of this embodiment.
[0058] Of course, the above are only preferred embodiments of the present invention, and are not intended to limit the scope of use of the present invention. Therefore, any equivalent changes made to the principles of the present invention should be included in the protection scope of the present invention.
[0059] Embodiment 2
[0060] like Figure 6 As shown, an embodiment of the present invention provides a machine translation model training system based on parameter pruning and tuning, including a preprocessing module, a machine translation model building module, a training fine-tuning module and a translation module;
[0061] The preprocessing module is used to obtain an incremental data set and a multilingual translation original model, generate pseudo original language data based on the original model, merge the pseudo original language data set and the incremental data set into a fine-tuning data set, and use the fine-tuning data set to construct a four-tuple fine-tuning data set;
[0062] The machine translation model construction module is used to construct a machine translation model. The machine translation model adopts a Transformer architecture, including a position encoder, an input embedding layer, an encoder multi-head self-attention block, a feedforward neural network, a decoder multi-head self-attention block, and an output processing layer. The feedforward neural network includes an original model FFN layer, a pruned FFN layer, and a linear layer. The original model FFN layer and the pruned FFN layer are connected in parallel and then connected to a linear layer.
[0063] The training and fine-tuning module is used to input the source language sentences of the fine-tuning data set into the machine translation model to obtain the translated target language sentences, and train and fine-tune the machine translation model based on the translated target language sentences using the CPO-M optimization method to obtain a trained and fine-tuned machine translation model;
[0064] The translation module is used to translate source language sentences based on the trained and fine-tuned machine translation model.
Claims
1. A machine translation model training method based on parameter pruning and tuning, characterized in that: The steps include: S01, obtaining training incremental data and processing to obtain an incremental data set; obtaining an original machine translation model to be tuned and trained; obtaining a pseudo target language data set, using a special text containing target language information as an input to an encoder, and using the generation capability of the original machine translation model to generate a pseudo target language data set in the original translation direction; S02, obtaining a fine-tuning dataset, using the original machine translation model to translate the pseudo target language dataset into the original language dataset, merging the pseudo target language dataset and the original language dataset into a pseudo dataset, and then merging the pseudo dataset with the incremental dataset of step S01 into a fine-tuning dataset (x, y), where x represents the source language and y represents the target language; using the incremental dataset to fine-tune the original machine translation model to obtain an incremental model; S03, constructing a pruned model, respectively evaluating the importance of each parameter in the FFN layer of the original machine translation model of step S01 and the incremental model of step S02, finding out the parameters in the FFN layer of the original machine translation model that are important for the original language translation, and the parameters in the FFN layer of the incremental model that are unimportant for the incremental language translation, replacing the parameters in the FFN layer of the incremental model that are unimportant for the incremental language translation but important for the original language translation with the parameters in the FFN layer of the original machine translation model that are important for the original language translation, thereby obtaining a pruned FFN layer, and adding the pruned FFN layer to the original machine translation model to construct a pruned model; S04. Use the fine-tuning dataset to construct a four-tuple fine-tuning dataset; S05. Freeze the parameters of the original machine translation model in the pruned model described in step S03, and use the optimized CPO algorithm using the four-tuple fine-tuning dataset to fine-tune the FFN layer in the pruned model, so as to obtain a new model after parameter pruning and tuning.
2. The method for training a machine translation model based on parameter pruning and tuning according to claim 1, characterized in that: In step S01, the acquired training incremental data is data of new translation direction acquired from a public machine translation text database. When processing the training incremental data, the Moses toolkit is first used for word segmentation to divide the original sentence into words or subwords, and then the BPE algorithm is used to pre-process the training incremental data after word segmentation to obtain a vocabulary and segmented subword data. By this method, an incremental data set can be obtained.
3. The machine translation model training method based on parameter pruning and tuning according to claim 1, characterized in that: When the incremental data set is used to fine-tune the original machine translation model to obtain the incremental model in step S02, the following steps are specifically adopted: the parallel statements of the incremental data set are used as the input of the source statement and the target statement of the original machine translation model respectively, and the original machine translation model is fine-tuned and the weight parameters thereof are updated to obtain the incremental model.
4. The method for training a machine translation model based on parameter pruning and tuning according to claim 1, characterized in that: In the step S03, when evaluating the importance of each parameter in the FFN layer of the original machine translation model and the incremental model, a method based on dual importance is used for evaluation.
5. The method for training a machine translation model based on parameter pruning and tuning according to claim 4, characterized in that: When evaluating the importance of each parameter in the FFN layer of the original machine translation model, the absolute value method is used for evaluation. The calculation formula for the importance of the parameter in the i-th row and j-th column of the FFN layer of the original machine translation model is: ,in Indicates the weight of the i-th row and j-th column of the FFN layer of the original machine translation model. When evaluating the importance of each parameter in the FFN layer of the incremental model, the absolute value difference method is used for evaluation. The calculation formula for the importance of the parameter of the i-th row and j-th column of the FFN layer of the incremental model is: ,in , is the weight of the i-th row and j-th column of the FFN layer of the incremental model.
6. The method for training a machine translation model based on parameter pruning and tuning according to claim 1, characterized in that: In the step S03, when the pruned FFN layer is added to the original machine translation model to construct the pruned model, the FFN layer of the original machine translation model and the pruned FFN layer are placed in parallel, a linear layer is newly added after the two FFN layers, the outputs of the parallel FFN layer of the original machine translation model and the pruned FFN layer are combined as the input of the linear layer, the parallel FFN layer of the original machine translation model, the pruned FFN layer and the linear layer are combined into the FFN layer of the pruned model, and the FFN layer of the pruned model replaces the FFN layer of the original machine translation model to obtain the pruned model.
7. The machine translation model training method based on parameter pruning and tuning according to claim 1, characterized in that: When the fine-tuning dataset is used to construct the four-tuple fine-tuning dataset in step S04, the following steps are specifically adopted: the large model GPT-4 and Google Translate are used to translate x in the fine-tuning dataset into target sentences tgt_sentence2 and tgt_sentence3 respectively; the model KIWI-XXL is used to score the tgt_sentence2, tgt_sentence3 sentences and the sentence y in the fine-tuning dataset respectively, and the sentences with high scores are named y_h, the sentences with medium scores are named y_m, and the sentences with low scores are named y_l, so as to obtain the four-tuple fine-tuning dataset (x, y_h, y_m, y_l).
8. The method for training a machine translation model based on parameter pruning and tuning according to claim 7, characterized in that: In step S05, the loss calculation formula of the optimized CPO algorithm is as follows: ;in, is the model parameter, σ is the Sigmoid function, , , is a hyperparameter.
9. The method for training a machine translation model based on parameter pruning and tuning according to claim 8, characterized in that: In the step S05, when the FFN layer of the pruned model is fine-tuned using the optimized CPO algorithm using the quaternary fine-tuning dataset, the loss calculation formula of the optimized CPO algorithm is used as the loss function of the parameters of the FFN layer of the pruned model and the quaternary fine-tuning dataset is used for fine-tuning calculation.
10. The method for training a machine translation model based on parameter pruning and tuning according to claim 1, characterized in that: Step S05 obtains a new model after parameter pruning and tuning, and tests the new model on multiple benchmark test sets.
11. A machine translation model training system based on parameter pruning and tuning, characterized in that: It includes preprocessing module, machine translation model building module, training and fine-tuning module and translation module; The preprocessing module is used to obtain an incremental data set and a multilingual translation original model, generate pseudo original language data based on the original model, merge the pseudo original language data set and the incremental data set into a fine-tuning data set, and use the fine-tuning data set to construct a four-tuple fine-tuning data set; The machine translation model construction module is used to construct a machine translation model. The machine translation model adopts a Transformer architecture, including a position encoder, an input embedding layer, an encoder multi-head self-attention block, a feedforward neural network, a decoder multi-head self-attention block, and an output processing layer. The feedforward neural network includes an original model FFN layer, the pruned FFN layer described in step S03 described in any one of claims 1 to 10, and a linear layer. The original model FFN layer and the pruned FFN layer are arranged in parallel and then connected to one of the linear layers. The training and fine-tuning module is used to input the source language sentences of the fine-tuning data set into the machine translation model to obtain the translated target language sentences, and train and fine-tune the machine translation model based on the translated target language sentences using the CPO-M optimization method to obtain a trained and fine-tuned machine translation model; The translation module is used to translate source language sentences based on the trained and fine-tuned machine translation model.