Method for improving small language machine translation accuracy through knowledge fusion

Abstract

The invention provides a method for improving small language machine translation accuracy through knowledge fusion. The method belongs to the field of minority language translation, and comprises the following steps: firstly, selecting a large language having a relatively strong language genetic relationship with the minority language, and pre-training a large amount of open corpora; performing model training by using semi-supervised learning in combination with a large amount of unlabeled minority language data and a small part of labeled minority language data; after training is completed, neural network pruning is carried out, unimportant parameters are removed, and the model is more simplified; and finally, knowledge distillation is carried out, and the model efficiency is improved by enabling the small models to reproduce behaviors of the large models. According to the method, the accuracy of small language translation is improved, the complexity of the model is reduced, the efficiency of the model is improved, and the practical value is high.

Description

Technical Field

[0001] The invention belongs to the field of minority language translation, and more specifically relates to a method for improving the accuracy of minority language machine translation by utilizing knowledge fusion. Background Art

[0002] Driven by the trend of globalization, cross-language communication activities are becoming more frequent, and machine translation (MT) technology has received extensive attention and research. Among the existing machine translation technologies, especially the machine translation technology based on neural networks, it can not only significantly improve the translation quality, but also effectively handle large-scale and complex language pairs.

[0003] However, in the translation process from a large language to a small language, the performance of machine translation has significantly declined due to the relative scarcity of training corpora in small languages. This has brought great difficulties to tasks such as cross-language information retrieval, automatic summarization, and text classification involving small languages. At the same time, machine translation of small languages ​​also faces problems such as complex grammatical rules of input sentences, rich and varied meanings of words, flexible and diverse syntactic structures, and obvious emotional tendencies.

[0004] Therefore, studying how to improve the accuracy of machine translation of minority languages, especially using existing large language resources for knowledge transfer to promote the improvement of machine translation performance of minority languages, has become an important topic at present. In this context, a method to improve the accuracy of machine translation of minority languages ​​through semi-supervised learning and knowledge fusion is proposed.

[0005] In addition, current machine learning models are usually highly complex and have many parameters, which not only increases the pressure of calculation and storage, but may also cause problems such as overfitting, affecting the generalization ability of the model. Therefore, how to effectively reduce the complexity of the model, improve the efficiency of the model, and maintain or even improve the performance of the model is also an important direction of current research. In response to this problem, the present invention uses the technology of neural network pruning and knowledge distillation to effectively simplify and optimize the model, thereby improving the translation efficiency and accuracy of the model. Summary of the invention

[0006] The core technical problem to be solved by the present invention is how to improve the accuracy and efficiency of machine translation of small languages. In particular, in the face of the scarcity of training data for small languages, how to make full use of the corpus resources of large languages ​​to transfer knowledge and conduct effective model training, and how to effectively prune and optimize the model while maintaining or improving the accuracy of translation to improve the efficiency of translation. These are the main technical problems to be solved by the present invention.

[0007] In order to achieve the above object, the present invention is implemented by adopting the following technical scheme: the method comprises:

[0008] Select a large language that has a strong linguistic affinity with a small language, and use a large amount of open corpus of this large language to pre-train the machine translation model;

[0009] Using semi-supervised learning methods, a large amount of unlabeled minority language data and a small amount of labeled minority language data are combined to further train the model;

[0010] After the model training is completed, the neural network is pruned to remove unimportant parameters to make the model more streamlined;

[0011] After the pruned model, knowledge distillation is performed, and the small model is used to reproduce the behavior of the large model to further improve the efficiency of the model.

[0012] In one embodiment, the pre-training includes:

[0013] Select source language and target language: select a major language that has a strong linguistic affinity with a minority language as the source language, and the minority language as the target language;

[0014] Data preparation: Collect bilingual corpus, one part of which is sentence pairs of the source language and the target language, and the other part is single sentences of the source language;

[0015] Preprocessing: Perform preprocessing operations on the collected data, including cleaning, screening, word segmentation, and word embedding;

[0016] Pre-trained model: Use the Transformer model, with sentences in the source language as input and sentences in the target language as output for training.

[0017] In one embodiment, the further training includes:

[0018] Data preparation: Collect a large amount of unlabeled data in minority languages ​​and some labeled data in minority languages ​​to fine-tune the pre-trained model;

[0019] Model fine-tuning: Fine-tune the pre-trained model on the labeled data of a minority language, compare the model output with the labeled target sentences, calculate the loss, and upgrade the model.

[0020] In one embodiment, the neural network pruning includes:

[0021] Prune small weights in the neural network based on the absolute value of the weights, and select weights whose absolute value is less than a certain threshold for pruning;

[0022] Select neurons whose L2 norm of neuron output is less than a certain threshold for pruning;

[0023] The pruning process is as follows:

[0024] First, a threshold is set, and then the importance of each neuron or weight is calculated. For weight pruning, the importance is represented by the absolute value of the weight; for unit pruning, the importance is represented by the L2 norm of the neuron output;

[0025] Then, the neurons or weights whose importance is lower than the set threshold are pruned away, i.e., set to zero and fix this value in subsequent optimizations.

[0026] In one embodiment, the knowledge distillation includes:

[0027] Training a large teacher model: The teacher model is the large model before pruning, denoted as model A;

[0028] Train a small student model: Create a new model, denoted as model B; the network structure of model B is smaller than that of model A, that is, the number of network layers and neurons in each layer is relatively small;

[0029] Perform knowledge distillation: let model B learn from the behavior of model A, that is, let model B learn under the guidance of model A;

[0030] The steps of knowledge distillation are as follows:

[0031] First, the same set of input data is fed into model A and model B respectively to obtain two sets of outputs, also called predicted labels.

[0032] Then, define a loss function, which includes two parts: one is the gap between the predicted label of model B and the true label, and the other is the similarity between the predicted label of model B and the predicted label of model A. The specific formula is as follows:

[0033] Loss=H(y,T(B(x)))+α*D(B(x),A(x))

[0034] in:

[0035] H(y,T(B(x))) is the cross entropy loss between the predicted label B(x) of model B and the true label y. T is the temperature parameter, which is used to soften the probability distribution of the model output, that is, to smooth the probability value.

[0036] α is a weight coefficient that controls the influence of the difference between B(x) and A(x);

[0037] D is defined as the difference between the predicted labels of model A and model B;

[0038] A(x) and B(x) are the prediction results of model A and model B respectively.

[0039] In one embodiment, the temperature parameter T is used to adjust the "softness" of the model output probability distribution. When T is close to 0, the model tends to generate deterministic output, that is, the category with the highest output probability; when T is close to ∞, the model tends to generate a uniform probability distribution.

[0040] In one solution, during the knowledge distillation, iterative training is performed multiple times, that is, the output result of model B is used as the input of model A again to further improve the accuracy of model B.

[0041] In one plan, when choosing between major and minor languages, reference is made to linguistic affinities, and languages ​​with a common ancestor or languages ​​with similar grammatical structures and vocabulary expressions are selected.

[0042] Beneficial effects of the present invention:

[0043] 1. The present invention fully considers the language affinity between minority languages ​​and major languages, uses major languages ​​that have a strong language affinity with minority languages ​​for pre-training, fully explores and utilizes the corpus resources of major languages, and helps solve the problem of scarcity of minority language corpus.

[0044] 2. The accuracy of machine translation of minority languages ​​can be effectively improved through semi-supervised learning and knowledge fusion methods. Semi-supervised learning combines a large amount of unlabeled minority language data with a small amount of labeled minority language data for model training, while knowledge fusion uses the student model to reproduce the behavior of the large model, further improving the accuracy of translation.

[0045] 3. The present invention also introduces the strategies of neural network pruning and knowledge distillation, which effectively simplifies the complexity of the model, reduces the computational and storage burden of the model, and also improves the efficiency and accuracy of the model by reproducing the behavior of the large model with a small model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 This is a pre-training flow chart of the present invention;

[0048] Figure 3 This is a flow chart of the method for semi-supervised learning of the present invention;

[0049] Figure 4 This is a flow chart of the knowledge distillation of the present invention. DETAILED DESCRIPTION

[0050] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Typical embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0051] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0052] like Figure 1 As shown, the method for improving the accuracy of machine translation of small languages ​​by utilizing knowledge fusion includes:

[0053] like Figure 2 As shown, S1. Select a large language that has a strong linguistic affinity with the small language, and use a large amount of open corpus of this large language to pre-train the machine translation model.

[0054] S101. Select source language and target language: Select a major language that has a strong linguistic affinity with a minority language as the source language, and the minority language as the target language. When selecting major and minority languages, you can refer to linguistic affinity and choose languages ​​with a common ancestor, or languages ​​with similar grammatical structures and vocabulary expressions.

[0055] S102. Data preparation: Collect a large amount of bilingual corpus, one part of which is sentence pairs of the source language and the target language, and the other part is single sentences in the source language.

[0056] S103, preprocessing: preprocessing operations are performed on the collected data, including cleaning, screening, word segmentation, word embedding, etc.

[0057] S104, pre-trained model: Use the Transformer model or other models suitable for machine translation tasks, with sentences in the source language as input and sentences in the target language as output for training.

[0058] The goal of pre-training is to minimize the following loss function:

[0059]

[0060] Among them, x n is a sentence in the source language (major language), y n is a sentence in the target language (minority language), P represents the translation probability predicted by the model, L is the loss function, θ is the model parameter, and N is the number of training samples.

[0061] After the pre-training is completed, a model is obtained that can translate the source language (a major language) into the target language (a minor language).

[0062] like Figure 3 As shown in Figure 2, S2 uses a semi-supervised learning method to combine a large amount of unlabeled minority language data and a small part of labeled minority language data to further train the model.

[0063] In the process of applying the pre-trained model to minority languages, since there is often very little labeled data for minority languages, semi-supervised learning methods are used to transfer the language knowledge of the majority language to the minority language.

[0064] S201. Data preparation: Collect a large amount of unlabeled data in minority languages ​​and some labeled data in minority languages. These data will be used to fine-tune the pre-trained model.

[0065] S202, model fine-tuning: fine-tune the pre-trained model on the labeled data of the minority language. Compare the output of the model with the labeled target sentence, calculate the loss, and upgrade the model.

[0066] S203, semi-supervised learning, whose learning goal is to find a prediction function f with optimal parameters, which can give a good prediction for any input x given the labeled samples (Xl, Yl) and the unlabeled samples Xu. The mathematical description is:

[0067]

[0068] in, represents the input and output of labeled data, represents the input of unlabeled data, L represents the loss function for labeled data, H represents the entropy function for unlabeled data, |f 2 It is used to represent the complexity of the model, also called the regularization term, and λ and γ are the weights for balancing each term.

[0069] The loss function L for labeled data here is usually the cross entropy loss used by the pre-trained model above, and the entropy function H for unlabeled data hopes that the model will be as confident as possible in the prediction results of unlabeled data, that is, to minimize the uncertainty of the prediction results.

[0070] In this way, under the guidance of pre-trained knowledge of major languages ​​and labeled data of minor languages, the model can better understand the semantics of minor languages, thereby improving the accuracy of translation.

[0071] S3. After the model training is completed, the neural network is pruned to remove unimportant parameters to make the model more streamlined.

[0072] Neural network pruning is a model optimization strategy that aims to reduce the size and computation of the model without significantly reducing the performance of the model. The most common pruning methods are weight pruning and unit pruning.

[0073] Weight pruning: Pruning small weights in a neural network. This is usually done based on the absolute value of the weight, and weights with an absolute value less than a certain threshold are selected for pruning.

[0074] Unit pruning: Select neurons whose L2 norm of neuron output is less than a certain threshold for pruning.

[0075] The pruning process is as follows:

[0076] First, you need to set a threshold, and then calculate the importance of each neuron or weight. For weight pruning, the importance can be represented by the absolute value of the weight; for unit pruning, the importance can be represented by the L2 norm of the neuron output.

[0077] Then, the neurons or weights whose importance is lower than the set threshold are pruned, that is, they are set to zero and fixed in subsequent optimizations. However, in practice, multiple rounds of pruning may be required, each of which performs a fine-tuning and then recalculates the importance of each neuron or weight.

[0078] The importance of weight pruning can be defined as the absolute value of the weight:

[0079] Importance=|weight|

[0080] The importance of unit pruning can be defined as the L2 norm of the neuron output:

[0081] Importance=‖output‖2

[0082] After several rounds of pruning and fine-tuning, a more streamlined model can be obtained, with a greatly reduced number of parameters and relatively little performance loss.

[0083] like Figure 4 As shown in S4, the pruned model performs knowledge distillation and uses a small model to reproduce the behavior of the large model, further improving the efficiency of the model.

[0084] Knowledge distillation is a model compression technique that allows a small model (student model) to learn under the guidance of a large model (teacher model). This method can significantly reduce the size and computational complexity of the model while maintaining performance.

[0085] The detailed implementation process is as follows:

[0086] S401. Training a large teacher model: The teacher model here is the large model before pruning, denoted as model A.

[0087] S402, training a small student model: In this step, a new model is created, which is recorded as model B. The network structure of model B is smaller than that of model A, that is, the network layers and the number of neurons in each layer are relatively small.

[0088] S403, perform knowledge distillation: In this step, let model B learn from the behavior of model A, that is, let model B learn under the guidance of model A. This process uses a technology called knowledge distillation.

[0089] The steps of knowledge distillation are as follows:

[0090] First, the same set of input data is input into model A and model B respectively to obtain two sets of outputs (also called predicted labels).

[0091] Then, a loss function is defined, which includes two parts: one is the gap between the predicted label of model B and the true label (this is similar to conventional supervised learning), and the other is the similarity (or gap) between the predicted label of model B and the predicted label of model A. The specific formula is as follows:

[0092] Loss=H(y,T(B(x)))+α*D(B(x),A(x))

[0093] in:

[0094] H(y,T(B(x))) is the cross entropy loss between the predicted label B(x) of model B and the true label y. T is the temperature parameter, which is used to soften the probability distribution of the model output, that is, to smooth the probability value. α is a weight coefficient that controls the influence of the difference between B(x) and A(x). D is the difference between the predicted labels defined by model A and model B. A(x) and B(x) are the prediction results of model A and model B respectively. Minimizing this loss function can make model B reproduce the behavior of model A as much as possible.

[0095] By training model B to replicate the behavior of model A, the model size is compressed while minimizing the degradation of model performance, thereby improving the efficiency of the artificial neural network model.

[0096] The temperature parameter T is used to adjust the "softness" of the model output probability distribution. When T is close to 0, the model tends to generate deterministic output, that is, the category with the highest output probability; when T is close to ∞, the model tends to generate a uniform probability distribution.

[0097] After pruning the neural network, the pruned model is used for machine translation of small languages. The results are compared with the standard answers, and the model is updated according to the failure rate.

[0098] When performing knowledge distillation, iterative training can be performed multiple times, that is, the output results of model B are used as the input of model A again to further improve the accuracy of model B.

[0099] Embodiment 2:

[0100] The following is a specific embodiment of the method of the present invention:

[0101] Suppose there is a small language, such as Uzbek, and a large language that is related to it, such as Turkish.

[0102] Pre-training: First, we selected a large amount of open Turkish corpus, including 1 million sentences of Turkish-Uzbek bilingual data and about 20 million sentences of monolingual Turkish data. We performed pre-processing operations, including cleaning irrelevant information, segmenting sentences, converting them into word embeddings, and adding start and end markers to Uzbek. We then used this data to train the Transformer model, using Turkish sentences as input and Uzbek sentences as output.

[0103] Further training: Based on the pre-training, 5,000 labeled Uzbek sentences and 1 million unlabeled Uzbek sentences were collected to fine-tune the model. By comparing the model output with the labeled target sentences, the loss was calculated and the model was upgraded.

[0104] Neural network pruning: Then perform model pruning by setting a weight threshold of 0.01 and selecting weights whose absolute values ​​are smaller than the threshold for pruning, as well as an L2 norm threshold of 0.05 and selecting neurons whose L2 norm of neuron output is smaller than the threshold for pruning.

[0105] Knowledge distillation: After pruning, use a smaller but similar model B to distill the large model A, where model A is the large model before pruning. First, the same set of input data is input into model A and model B respectively to obtain two sets of outputs. Then, model B is trained according to the formula, where α is set to 0.6 and T is set to 2. Through iterative optimization, the optimized small model B is finally obtained.

[0106] Through the above steps, the Uzbek translation model has been trained and optimized. The test results show that this method can effectively improve the accuracy of machine translation of small languages. The BLEU score of the model is 8 percentage points higher than that of the original Transformer model, and the model parameters are reduced by about 60%.

[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0108] It should be understood that the detailed description of the technical solutions of the present invention by means of the preferred embodiments is illustrative rather than restrictive. A person skilled in the art may modify the technical solutions described in the embodiments, or replace some of the technical features by equivalents, based on reading the specification of the present invention; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving the accuracy of machine translation of a minority language by utilizing knowledge fusion, characterized in that: The method includes: Select a large language that has a strong linguistic affinity with a small language, and use a large amount of open corpus of this large language to pre-train the machine translation model; Using semi-supervised learning methods, a large amount of unlabeled minority language data and a small amount of labeled minority language data are combined to further train the model; After the model training is completed, the neural network is pruned to remove unimportant parameters to make the model more streamlined; After the pruned model, knowledge distillation is performed, and the small model is used to reproduce the behavior of the large model to further improve the efficiency of the model.

2. The method for improving the accuracy of machine translation of minority languages ​​by utilizing knowledge fusion according to claim 1, characterized in that: The pre-training includes: Select source language and target language: select a major language that has a strong linguistic affinity with a minority language as the source language, and the minority language as the target language; Data preparation: Collect bilingual corpus, one part of which is sentence pairs of the source language and the target language, and the other part is single sentences of the source language; Preprocessing: Perform preprocessing operations on the collected data, including cleaning, screening, word segmentation, and word embedding; Pre-trained model: Use the Transformer model, with sentences in the source language as input and sentences in the target language as output for training.

3. The method for improving the accuracy of machine translation of minority languages ​​by utilizing knowledge fusion according to claim 1, characterized in that: Said further training includes: Data preparation: Collect a large amount of unlabeled data in minority languages ​​and some labeled data in minority languages ​​to fine-tune the pre-trained model; Model fine-tuning: Fine-tune the pre-trained model on the labeled data of a minority language, compare the model output with the labeled target sentences, calculate the loss, and upgrade the model.

4. The method for improving the accuracy of machine translation of a minority language by utilizing knowledge fusion according to claim 1, characterized in that: The neural network pruning includes: Prune small weights in the neural network based on the absolute value of the weights, and select weights whose absolute value is less than a certain threshold for pruning; Select neurons whose L2 norm of neuron output is less than a certain threshold for pruning; The pruning process is as follows: First, a threshold is set, and then the importance of each neuron or weight is calculated. For weight pruning, the importance is represented by the absolute value of the weight; for unit pruning, the importance is represented by the L2 norm of the neuron output; Then, the neurons or weights whose importance is lower than the set threshold are pruned away, i.e., set to zero and fix this value in subsequent optimizations.

5. The method for improving the accuracy of machine translation of a minority language by utilizing knowledge fusion according to claim 1, characterized in that: The knowledge distillation includes: Training a large teacher model: The teacher model is the large model before pruning, denoted as model A; Train a small student model: Create a new model, denoted as model B; the network structure of model B is smaller than that of model A, that is, the number of network layers and neurons in each layer is relatively small; Perform knowledge distillation: let model B learn from the behavior of model A, that is, let model B learn under the guidance of model A; The steps of knowledge distillation are as follows: First, the same set of input data is fed into model A and model B respectively to obtain two sets of outputs, also called predicted labels. Then, define a loss function, which includes two parts: one is the gap between the predicted label of model B and the true label, and the other is the similarity between the predicted label of model B and the predicted label of model A. The specific formula is as follows: Loss=H(y,T(B(x)))+α*D(B(x),A(x)) in: H(y,T(B(x))) is the cross entropy loss between the predicted label B(x) of model B and the true label y. T is the temperature parameter, which is used to soften the probability distribution of the model output, that is, to smooth the probability value. α is a weight coefficient that controls the influence of the difference between B(x) and A(x); D is defined as the difference between the predicted labels of model A and model B; A(x) and B(x) are the prediction results of model A and model B respectively.

6. The method for improving the accuracy of machine translation of a minority language by utilizing knowledge fusion according to claim 5, characterized in that: The temperature parameter T is used to adjust the "softness" of the model output probability distribution. When T is close to 0, the model tends to generate deterministic output, that is, the category with the highest output probability; when T is close to ∞, the model tends to generate a uniform probability distribution.

7. The method for improving the accuracy of machine translation of a minority language by utilizing knowledge fusion according to claim 5, characterized in that: During the knowledge distillation, iterative training is performed multiple times, that is, the output result of model B is used as the input of model A again to further improve the accuracy of model B.

8. The method for improving the accuracy of machine translation of a minority language by utilizing knowledge fusion according to claim 1, characterized in that: When choosing between major and minor languages, refer to linguistic affinities and choose languages ​​with common ancestors, or languages ​​with similar grammatical structures and vocabulary expressions.

Images