Knowledge learning method and device of a machine translation model

Abstract

The application provides a knowledge learning method and device of a machine translation model, comprising: constructing a migration dataset; respectively using a machine translation teacher model and a machine translation student model to translate the original text of each sample of the migration dataset to obtain the first candidate word probability distribution and the second candidate word probability distribution of each sample of the migration dataset; determining the migration loss of each sample of the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample of the migration dataset; and realizing the knowledge learning of the machine translation student model to the machine translation teacher model based on the first candidate word probability distribution of each sample of the migration dataset and the migration loss. The application enables the machine translation student model to learn complementary knowledge from the machine translation teacher model through the migration loss, realizes the knowledge accumulation of the machine translation student model, and further improves the translation performance of the machine translation student model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine translation, and in particular to a knowledge learning method and device of a machine translation model. BACKGROUND

[0002] With the development of deep learning technology, a large number of machine translation models under the same language pair have emerged. These models are trained on different corpora, have different architectures, adapt to different scales, are deployed in different occasions, and perform well in different professional fields. Realizing knowledge sharing of multiple machine translation models under the same language pair will greatly improve the translation performance of machine translation models. However, research shows that the good performance of the model comes from the knowledge contained in it, which is obtained from the training data and gradually stored in the model parameters of the neural network during the training process, and it is difficult to express and explain explicitly. Knowledge transfer between models is very difficult.

[0003] At present, in some technologies, knowledge transfer between different models is realized in a way that does not expose data, and knowledge distillation is the most representative paradigm.

[0004] However, the knowledge distillation method usually assumes that there is an explicit and known relationship between the models, and knowledge can only be transferred from the model with good performance to the model with poor performance in one direction. This seriously restricts the scope of knowledge transfer and ignores the case where a model with poor performance can also provide complementary knowledge for a model with good performance. SUMMARY

[0005] The present application provides a knowledge learning method and device of a machine translation model to solve the technical problem that knowledge can only be transferred from a model with good performance to a model with poor performance in one direction in the prior art. Two machine translation models under a preset language pair regardless of performance are taken as a machine translation student model and a machine translation teacher model, respectively, complementary knowledge beneficial to the machine translation student model is identified from the machine translation teacher model, and the identified knowledge is migrated to the machine translation student model, realizing knowledge learning of the machine translation student model and improving the translation performance of the machine translation student model.

[0006] In a first aspect, the present application provides a knowledge learning method of a machine translation model, comprising:

[0007] constructing a migration dataset;

[0008] translating the original text of each sample of the migration dataset by using the machine translation teacher model to obtain the first candidate word probability distribution of each sample of the migration dataset;

[0009] translate the original text of each sample in the migration dataset by using the machine translation student model to obtain a second candidate word probability distribution of each sample in the migration dataset;

[0010] determine a migration loss of each sample in the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration dataset;

[0011] based on the first candidate word probability distribution and the migration loss of each sample in the migration dataset, realize the knowledge learning of the machine translation student model on the machine translation teacher model;

[0012] wherein the machine translation student model and the machine translation teacher model are both used to realize language translation under a preset language pair; each sample in the migration dataset is composed of original text segmentation and translation segmentation under a preset language pair, and has a unique standard candidate word probability distribution.

[0013] According to the knowledge learning method of the machine translation model, the determination of the migration loss of each sample in the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration dataset comprises:

[0014] calculate the cross entropy of the first candidate word probability distribution of each sample in the migration dataset to the standard candidate word probability distribution of each sample in the migration dataset, and record it as the first cross entropy of each sample in the migration dataset;

[0015] determine the proportion coefficient of each sample in the migration dataset based on the first cross entropy of each sample in the migration dataset and the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration dataset;

[0016] calculate the KL divergence of the first candidate word probability distribution of each sample in the migration dataset to the second candidate word probability distribution of each sample in the migration dataset, and record it as the first KL divergence of each sample in the migration dataset;

[0017] calculate the KL divergence of the first candidate word probability distribution of each sample in the migration dataset to the third candidate word probability distribution pre-stored in the knowledge repository, and record it as the second KL divergence of each sample in the migration dataset;

[0018] determine the migration loss of each sample in the migration dataset according to the first cross entropy, the proportion coefficient, the first KL divergence and the second KL divergence of each sample in the migration dataset;

[0019] The proportionality coefficient of each sample in the migration data set represents a learning direction of the machine translation student model to the "first candidate word probability distribution of each sample in the migration data set" transmitted by the machine translation teacher model.

[0020] The third candidate word probability distribution is a candidate word probability distribution obtained by using a machine translation teacher model learned by the machine translation student model to translate the original text of each sample in the migration data set.

[0021] According to the knowledge learning method of the machine translation model, the proportionality coefficient of each sample in the migration data set is determined based on the first cross entropy of each sample in the migration data set and the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set, and includes:

[0022] The cross entropy of the second candidate word probability distribution of each sample in the migration data set to the standard candidate word probability distribution of each sample in the migration data set is calculated and recorded as the second cross entropy of each sample in the migration data set.

[0023] The proportionality coefficient of each sample in the migration data set is determined based on the first cross entropy and the second cross entropy of each sample in the migration data set.

[0024] According to the knowledge learning method of the machine translation model, when the proportionality coefficient of each sample in the migration data set is -1, it represents that the learning direction of the machine translation student model to the "first candidate word probability distribution of each sample in the migration data set" transmitted by the machine translation teacher model is negative, and when the proportionality coefficient of each sample in the migration data set is 1, it represents that the learning direction of the machine translation student model to the "first candidate word probability distribution of each sample in the migration data set" transmitted by the machine translation teacher model is positive.

[0025] The proportionality coefficient of each sample in the migration data set is determined based on the first cross entropy and the second cross entropy of each sample in the migration data set, and includes:

[0026] If the second cross entropy of each sample in the migration data set is greater than the first cross entropy, the proportionality coefficient of each sample in the migration data set is -1.

[0027] If the second cross entropy of each sample in the migration data set is not greater than the first cross entropy, the proportionality coefficient of each sample in the migration data set is 1.

[0028] According to the knowledge learning method of the machine translation model, the migration loss of each sample in the migration data set is determined according to the first cross entropy, the proportion coefficient, the first KL divergence and the second KL divergence of each sample in the migration data set, and the migration loss of each sample in the migration data set is determined by adding the product of the proportion coefficient and the first KL divergence of each sample in the migration data set, the negative value of the first cross entropy of each sample in the migration data set and the second KL divergence of each sample in the migration data set.

[0029] The product of the proportion coefficient and the first KL divergence of each sample in the migration data set, the negative value of the first cross entropy of each sample in the migration data set and the second KL divergence of each sample in the migration data set are added as the migration loss of each sample in the migration data set.

[0030] According to the knowledge learning method of the machine translation model, the knowledge learning of the machine translation student model to the machine translation teacher model is realized based on the first candidate word probability distribution and the migration loss of each sample in the migration data set.

[0031] The network parameters in the machine translation student model are optimized by gradient descent based on the first candidate word probability distribution and the migration loss of each sample in the migration data set.

[0032] According to the knowledge learning method of the machine translation model, after the knowledge learning of the machine translation student model to the machine translation teacher model is realized, the third candidate word probability distribution stored in the knowledge repository is updated to the first candidate word probability distribution of each sample in the migration data set.

[0033] In the case that the knowledge repository does not pre-store the third candidate word probability distribution, the second KL divergence of each sample in the migration data set is set to 0.

[0034] In the case that the knowledge repository does not pre-store the third candidate word probability distribution, the second KL divergence of each sample in the migration data set is set to 0.

[0035] In the case that the knowledge repository does not pre-store the third candidate word probability distribution, the second KL divergence of each sample in the migration data set is set to 0.

[0036] The first translation module is configured to translate the original text of each sample in the migration data set by using the machine translation teacher model to obtain the first candidate word probability distribution of each sample in the migration data set.

[0037] The second translation module is configured to translate the original text of each sample in the migration data set by using the machine translation student model to obtain the second candidate word probability distribution of each sample in the migration data set.

[0038] The migration loss calculation module is configured to determine the migration loss of each sample in the migration data set according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set.

[0039] a knowledge learning module configured to implement knowledge learning of the machine translation student model on the machine translation teacher model based on the first candidate word probability distribution of each sample in the migration dataset and the migration loss;

[0040] The machine translation student model and the machine translation teacher model are configured to implement language translation in a preset language pair.

[0041] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the knowledge learning method of the machine translation model according to the first aspect.

[0042] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the knowledge learning method of the machine translation model according to the first aspect.

[0043] The present application provides a knowledge learning method and device of a machine translation model. First, a migration dataset is constructed, wherein each sample in the migration dataset is composed of original text segmentation and translation text segmentation in a preset language pair, and determines a unique standard candidate word probability distribution. Second, the original text of each sample in the migration dataset is translated by a machine translation teacher model to obtain a first candidate word probability distribution of each sample in the migration dataset, and the original text of each sample in the migration dataset is translated by a machine translation student model to obtain a second candidate word probability distribution of each sample in the migration dataset. The machine translation student model and the machine translation teacher model are configured to implement language translation in a preset language pair. Third, the first candidate word probability distribution, the second candidate word probability distribution, and the standard candidate word probability distribution of each sample in the migration dataset are used to effectively analyze the real-world knowledge contained in the machine translation teacher model and the machine translation student model, so as to explore the beneficial and complementary knowledge of the machine translation student model from the machine translation teacher model and reflect on the first candidate word probability distribution of each sample in the migration dataset and the migration loss. Finally, the detected knowledge is migrated from the machine translation teacher model to the machine translation student model to implement knowledge accumulation of the machine translation student model. The present application provides an effective method for the machine translation student model to learn knowledge from the machine translation teacher model, and the performance of the machine translation student model can be continuously improved through learning of different machine translation teacher models.

[0044] The application transmits knowledge among models in a knowledge distillation manner, does not need additional data and information, has wide application fields, low use cost, and thus has high practical value. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0046] Figure 1 is a flowchart of a knowledge learning method of a machine translation model provided by the present application;

[0047] Figure 2 is a structural schematic diagram of a knowledge learning device of a machine translation model provided by the present application;

[0048] Figure 3 is a structural schematic diagram of an electronic device for implementing a knowledge learning method of a machine translation model provided by the present application. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the present application.

[0050] The knowledge learning method and device of a machine translation model of the present application will be described below with reference to Figures 1-3

[0051] In a first aspect, the purpose of the present application is to enable a machine translation student model to learn beneficial and complementary knowledge from different machine translation teacher models, so as to complete the knowledge accumulation of the machine translation student model and improve the translation performance of the machine translation student model. The knowledge accumulation process of the machine translation student model is carried out in stages. The process of the machine translation student model learning knowledge from a machine translation teacher model on a migration data set is a knowledge accumulation stage. The present application provides a knowledge learning method of a machine translation model, which is described in detail for a knowledge accumulation stage, as shown in Figure 1 The method comprises the following steps.

[0052] ​S11, construct a migration dataset; each sample in the migration dataset is composed of original text segmentation and translation text segmentation under a preset language pair, which determines a unique standard candidate word probability distribution.

[0053] In order to save time and labor, the training data set of the machine translation student model is usually taken as the migration data set in the knowledge accumulation process.

[0054] It should be noted that the machine translation student model and the machine translation teacher model of the present application are trained on different corpora (for example, corpora belonging to legal, financial, literary, news and other industry fields respectively) and are used to realize language translation under a preset language pair (for example, English to Chinese, French to English, American to German, etc.).

[0055] S12, translate the original text of each sample in the migration data set by using the machine translation teacher model to obtain the first candidate word probability distribution of each sample in the migration data set;

[0056] That is, the original text of each sample in the migration data set is input into the machine translation teacher model to obtain the candidate word probability distribution (softmax probability distribution) containing the knowledge of the machine translation teacher model, which is the implicit knowledge corresponding to each sample in the migration data set in the machine translation teacher model. The candidate word probability distribution can be regarded as a probability distribution composed of each translation candidate word of the original text and its possible probability.

[0057] S13, translate the original text of each sample in the migration data set by using the machine translation student model to obtain the second candidate word probability distribution of each sample in the migration data set;

[0058] That is, the original text of each sample in the migration data set is input into the machine translation student model to obtain the candidate word probability distribution (softmax probability distribution) containing the knowledge of the machine translation student model, which is the implicit knowledge corresponding to each sample in the migration data set in the machine translation student model.

[0059] S14, determine the migration loss of each sample in the migration data set according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set; it can be imagined that in order to enable the machine translation student model to learn the knowledge beneficial to itself and complementary to the knowledge "first candidate word probability distribution of each sample in the migration data set" transferred from the machine translation teacher model, the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set are analyzed in depth, and the migration loss of each sample in the migration data set is constructed to control the learning direction and learning intensity.

[0060] S15, based on the first candidate word probability distribution of each sample of the migration data set and the migration loss, realizing the knowledge learning of the machine translation student model to the machine translation teacher model.

[0061] In the case of determining the migration loss, the gradient descent method is used to back-propagate the optimized machine translation student model to optimize the parameters of the machine translation student model and improve the performance of the machine translation student model.

[0062] In the application of the machine translation student model, the target translation text is segmented to obtain the corresponding segmentation sequence;

[0063] The segmentation sequence is input into the machine translation student model to obtain the candidate word probability distribution of each segmentation in the segmentation sequence;

[0064] Based on the candidate word probability distribution of each segmentation in the segmentation sequence, the translation text of the target translation text is obtained;

[0065] Among them, the target translation text is a text expressed in a source language under a preset language.

[0066] The knowledge learning method of the machine translation model provided by the application uses the migration loss of each sample in the migration data set and the first candidate word probability distribution of each sample in the migration data set obtained by using the machine translation teacher model to translate the original text of each sample in the migration data set to express the complementary knowledge that the machine translation student model can learn from the machine translation teacher model. The complementary knowledge is migrated to the machine translation student model to accumulate the knowledge in the machine translation student model and improve the translation performance of the machine translation student model.

[0067] On the basis of the above embodiments, as an optional embodiment, the migration loss of each sample in the migration data set is determined according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set, comprising:

[0068] The cross entropy of the first candidate word probability distribution of each sample in the migration data set to the standard candidate word probability distribution of each sample in the migration data set is calculated and recorded as the first cross entropy of each sample in the migration data set, which can be represented as c t ;

[0069] c t =∑x i log a i

[0070] Among them, x i represents the probability of the i-th candidate word in the standard candidate word probability distribution, and a irepresents the probability of the i-th candidate word in the first candidate word probability distribution.

[0071] In order to make the candidate word probability distribution output by the machine translation teacher model as close as possible to the standard candidate word probability distribution, for each sample, the cross entropy c between the first candidate word probability distribution and the standard candidate word probability distribution is calculated. t , and serves as the first part of the loss function.

[0072] Based on the first cross entropy of each sample in the migration data set and the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set, the proportion coefficient of each sample in the migration data set is determined; wherein the proportion coefficient of each sample in the migration data set represents the learning direction of the machine translation student model to the "first candidate word probability distribution of each sample in the migration data set" transmitted by the machine translation teacher model.

[0073] The present application explores the complementary knowledge beneficial to the machine translation student model from the machine translation teacher model, and therefore needs to analyze whether the knowledge "first candidate word probability distribution of each sample in the migration data set" transmitted by the machine translation teacher model plays a positive reference role (should be close) or a negative reference role (should be far away) on the machine translation student model. The present application uses the proportion coefficient i (only -1 and 1) to represent the positive reference role and the negative reference, and uses the first cross entropy of each sample in the migration data set and the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set to analyze the value of the proportion coefficient.

[0074] The KL divergence of the first candidate word probability distribution of each sample in the migration data set to the second candidate word probability distribution of each sample in the migration data set is calculated, and is recorded as the first KL divergence of each sample in the migration data set.

[0075] In order to calculate the similarity of the candidate word probability distributions output by the machine translation teacher model and the machine translation student model, for each sample, the KL divergence (k) of the candidate word probability distribution output by the machine translation teacher model to the candidate word probability distribution output by the machine translation student model is calculated for the learning of the machine translation student model.

[0076] It should be noted that the present application needs to use the proportion coefficient i to define the direction of the first KL divergence. If the proportion coefficient i = 1, the KL divergence should be positive, and the candidate word probability distribution output by the machine translation teacher model should be as close as possible to the candidate word probability distribution output by the machine translation student model; if the proportion coefficient i = -1, the KL divergence should be negative, and the candidate word probability distribution output by the machine translation teacher model should be as far away as possible from the candidate word probability distribution output by the machine translation student model. Therefore, ik is taken as the second part of the loss function.

[0077] The KL divergence of the first candidate word probability distribution of each sample in the migration data set to the third candidate word probability distribution pre-stored in the knowledge repository is calculated and recorded as the second KL divergence of each sample in the migration data set; wherein the third candidate word probability distribution is the candidate word probability distribution obtained by translating the original text of each sample in the migration data set by using the machine translation teacher model learned by the machine translation student model last time;

[0078] In order to avoid forgetting the old knowledge that has been learned, the candidate word probability distribution output by the machine translation teacher model needs to be close to the candidate word probability distribution output by the last machine translation teacher model, and the KL divergence s of the candidate word probability distribution output by the machine translation teacher model to the candidate word probability distribution output by the last machine translation teacher model is taken as the third part of the loss function.

[0079] According to the first cross entropy of each sample in the migration data set, the proportion coefficient, the first KL divergence and the second KL divergence, the migration loss of each sample in the migration data set is determined.

[0080] The present application provides an effective method for transferring knowledge between models, and ensures that the learned knowledge will not be forgotten, so as to continuously improve the performance of the model.

[0081] On the basis of the above embodiments, as an optional embodiment, the proportion coefficient of each sample in the migration data set is determined based on the first cross entropy of each sample in the migration data set, and the second candidate word probability distribution of each sample in the migration data set and the standard candidate word probability distribution, comprising:

[0082] The cross entropy of the second candidate word probability distribution of each sample in the migration data set to the standard candidate word probability distribution of each sample in the migration data set is calculated and recorded as the second cross entropy of each sample in the migration data set, which can be expressed as: s

[0083] c s =∑x i log b i ​

[0084] wherein b i represents the probability of the i-th candidate word in the second candidate word probability distribution.

[0085] Determine the proportionality coefficient of each sample in the migration dataset based on the first cross-entropy and the second cross-entropy of each sample in the migration dataset.

[0086] The cross-entropy is a function used to measure the difference between two probability distributions. In the present application, the standard candidate word probability distribution of each sample in the migration dataset is taken as a reference to determine the difference c t between the first candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration dataset, and the difference c s between the second candidate word probability distribution and the standard candidate word probability distribution. Then, the difference between c s and c t is used to measure whether the knowledge "the first candidate word probability distribution of each sample in the migration dataset" transferred by the machine translation teacher model is beneficial to the machine translation student model, which is simple and requires small amount of operation resources.

[0087] On the basis of the above embodiments, as an optional embodiment, when the proportionality coefficient of each sample in the migration dataset is -1, it represents that the learning direction of "the first candidate word probability distribution of each sample in the migration dataset" transferred by the machine translation teacher model to the machine translation student model is negative; when the proportionality coefficient of each sample in the migration dataset is 1, it represents that the learning direction of "the first candidate word probability distribution of each sample in the migration dataset" transferred by the machine translation teacher model to the machine translation student model is positive.

[0088] The determination of the proportionality coefficient of each sample in the migration dataset based on the first cross-entropy and the second cross-entropy of each sample in the migration dataset comprises:

[0089] If the second cross-entropy of each sample in the migration dataset is greater than the first cross-entropy, the proportionality coefficient of each sample in the migration dataset is -1.

[0090] If the second cross-entropy of each sample in the migration dataset is not greater than the first cross-entropy, the proportionality coefficient of each sample in the migration dataset is 1.

[0091] Compare the relative sizes of c s and c t . If c s > c tIf c s , the degree to which the candidate word probability distribution output by the machine translation student model approaches the standard candidate word probability distribution is greater than the degree to which the candidate word probability distribution output by the machine translation teacher model approaches the standard candidate word probability distribution, undoubtedly, the candidate word probability distribution output by the machine translation teacher model is non-complementary and harmful to the machine translation student model, and the machine translation student model should take it as a negative reference, the knowledge detection should set the proportional coefficient i = -1, representing that the two cross-entropy directions are opposite and should be far away from each other. Conversely, if c t , the degree to which the candidate word probability distribution output by the machine translation student model approaches the standard candidate word probability distribution is less than or equal to the degree to which the candidate word probability distribution output by the machine translation teacher model approaches the standard candidate word probability distribution, undoubtedly, the candidate word probability distribution output by the machine translation teacher model is complementary and beneficial to the machine translation student model, and the machine translation student model should take it as a positive reference, the knowledge detection should set the proportional coefficient i = 1, representing that the two cross-entropy directions are the same and should be close to each other.

[0092] The present application can determine the learning direction of the machine translation student model by simply comparing the relative sizes of c s and c t , promote the accumulation of beneficial knowledge and discard the integration of harmful knowledge.

[0093] On the basis of the above embodiments, as an optional embodiment, the method further comprises:

[0094] multiplying the proportional coefficient and the first KL divergence of each sample in the migration data set, adding the negative value of the first cross-entropy of each sample in the migration data set and the second KL divergence of each sample in the migration data set, and taking the sum as the migration loss of each sample in the migration data set.

[0095] The present application considers the negative cross-entropy of the candidate word probability distribution output by the machine translation teacher model to the standard candidate word probability distribution, the KL divergence between the candidate word probability distribution output by the machine translation teacher model and the candidate word probability distribution output by the machine translation student model, and the KL divergence between the candidate word probability distribution output by the machine translation teacher model and the third candidate word probability distribution, ensures that the migration loss enables the student model to learn beneficial knowledge without forgetting old knowledge, guarantees the machine translation student model to continuously accumulate the beneficial knowledge that is lacking, and further continuously improves the performance of the machine translation student model.

[0096] On the basis of each of the above embodiments, as an optional embodiment, the knowledge learning of the machine translation student model on the machine translation teacher model is implemented based on the first candidate word probability distribution and the migration loss of each sample of the migration data set, and the knowledge learning of the machine translation student model on the machine translation teacher model comprises the following steps:

[0097] The network parameters in the machine translation student model are optimized by gradient descent based on the first candidate word probability distribution and the migration loss of each sample of the migration data set.

[0098] Gradient descent is a common network parameter optimization method in deep learning, which will not be described here.

[0099] On the basis of each of the above embodiments, as an optional embodiment, after the knowledge learning of the machine translation student model on the machine translation teacher model is implemented, the third candidate word probability distribution stored in the knowledge repository is updated to the first candidate word probability distribution of each sample of the migration data set.

[0100] In the case that the third candidate word probability distribution is not pre-stored in the knowledge repository, the second KL divergence of each sample of the migration data set is set to 0.

[0101] Continual learning requires retaining knowledge in old data when facing continuously changing data to overcome catastrophic forgetting. Therefore, the present application sets up a knowledge repository to ensure that the machine translation student model continuously accumulates the knowledge that has been learned.

[0102] The knowledge learning device of the machine translation model provided by the present application is described below, and the knowledge learning device of the machine translation model described below can be mutually corresponding to the knowledge learning method of the machine translation model described above. Figure 2 An example of the structure diagram of the knowledge learning device of the machine translation model is shown in Figure 2 As shown, the device comprises:

[0103] The construction module 21 is configured to construct a migration data set.

[0104] The first translation module 22 is configured to translate the original text of each sample of the migration data set by using the machine translation teacher model to obtain the first candidate word probability distribution of each sample of the migration data set.

[0105] The second translation module 23 is configured to translate the original text of each sample of the migration data set by using the machine translation student model to obtain the second candidate word probability distribution of each sample of the migration data set.

[0106] The migration loss calculation module 24 is configured to determine a migration loss of each sample in the migration data set according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set.

[0107] The knowledge learning module 25 is configured to realize knowledge learning of the machine translation student model on the machine translation teacher model based on the first candidate word probability distribution and the migration loss of each sample in the migration data set.

[0108] The machine translation student model and the machine translation teacher model are both configured to realize language translation in a preset language pair.

[0109] The knowledge learning device of the machine translation model provided by the application uses the migration loss of each sample in the migration data set and the first candidate word probability distribution of each sample in the migration data set obtained by translating the original text of each sample in the migration data set by using the machine translation teacher model to express the complementary knowledge that the machine translation student model can learn from the machine translation teacher model, and migrates the complementary knowledge to the machine translation student model to accumulate knowledge in the machine translation student model and improve the translation performance of the machine translation student model.

[0110] Based on the above-mentioned embodiments, as an optional embodiment, the migration loss calculation module comprises:

[0111] The first cross-entropy calculation unit is configured to calculate the cross-entropy of the first candidate word probability distribution of each sample in the migration data set with respect to the standard candidate word probability distribution of each sample in the migration data set, and record the cross-entropy as the first cross-entropy of each sample in the migration data set.

[0112] The proportion coefficient calculation unit is configured to determine the proportion coefficient of each sample in the migration data set based on the first cross-entropy of each sample in the migration data set and the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration data set.

[0113] The first KL divergence calculation unit is configured to calculate the KL divergence of the first candidate word probability distribution of each sample in the migration data set with respect to the second candidate word probability distribution of each sample in the migration data set, and record the KL divergence as the first KL divergence of each sample in the migration data set.

[0114] a second KL divergence calculation unit configured to calculate a KL divergence of the first candidate word probability distribution of each sample in the migration dataset with respect to a third candidate word probability distribution pre-stored in the knowledge repository, and record the KL divergence as a second KL divergence of each sample in the migration dataset;

[0115] a migration loss determination unit configured to determine a migration loss of each sample in the migration dataset according to the first cross entropy, the scaling factor, the first KL divergence and the second KL divergence of each sample in the migration dataset;

[0116] The scaling factor of each sample in the migration dataset represents a learning direction of the machine translation student model with respect to the first candidate word probability distribution of each sample in the migration dataset transmitted by the machine translation teacher model.

[0117] The third candidate word probability distribution is a candidate word probability distribution obtained by translating the source text of each sample in the migration dataset by using a machine translation teacher model learned by the machine translation student model.

[0118] On the basis of the above embodiments, as an optional embodiment, the scaling factor calculation unit comprises:

[0119] a second cross entropy calculation sub-module configured to calculate a cross entropy of the second candidate word probability distribution of each sample in the migration dataset with respect to the standard candidate word probability distribution of each sample in the migration dataset, and record the cross entropy as a second cross entropy of each sample in the migration dataset;

[0120] a scaling factor determination sub-module configured to determine the scaling factor of each sample in the migration dataset based on the first cross entropy and the second cross entropy of each sample in the migration dataset.

[0121] On the basis of the above embodiments, as an optional embodiment, when the scaling factor of each sample in the migration dataset is -1, it represents that the learning direction of the machine translation student model with respect to the first candidate word probability distribution of each sample in the migration dataset transmitted by the machine translation teacher model is negative, and when the scaling factor of each sample in the migration dataset is 1, it represents that the learning direction of the machine translation student model with respect to the first candidate word probability distribution of each sample in the migration dataset transmitted by the machine translation teacher model is positive.

[0122] The scaling factor determination sub-module is specifically configured to:

[0123] If the second cross entropy of each sample in the migration dataset is greater than the first cross entropy, the scaling factor of each sample in the migration dataset is -1.

[0124] If the second cross-entropy of each sample in the migration dataset is not greater than the first cross-entropy, the scaling factor of each sample in the migration dataset is 1.

[0125] On the basis of the above embodiments, as an optional embodiment, the migration loss determination unit is specifically configured to:

[0126] The product of the scaling factor of each sample in the migration dataset and the first KL divergence, the negative value of the first cross-entropy of each sample in the migration dataset, and the second KL divergence of each sample in the migration dataset are added as the migration loss of each sample in the migration dataset.

[0127] On the basis of the above embodiments, as an optional embodiment, the knowledge learning module is specifically configured to:

[0128] Based on the first candidate word probability distribution of each sample in the migration dataset and the migration loss, the network parameters in the machine translation student model are optimized by gradient descent.

[0129] On the basis of the above embodiments, as an optional embodiment, the device further comprises an updating module configured to update the third candidate word probability distribution stored in the knowledge repository to the first candidate word probability distribution of each sample in the migration dataset after the knowledge learning of the machine translation student model on the machine translation teacher model is implemented.

[0130] The second KL divergence calculation unit is further configured to, in the case that the third candidate word probability distribution is not pre-stored in the knowledge repository, set the second KL divergence of each sample in the migration dataset to 0.

[0131] In a third aspect, Figure 3 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 3As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 complete mutual communication through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a knowledge learning method of a machine translation model, the method including: constructing a migration dataset; translating a source text of each sample of the migration dataset using a machine translation teacher model to obtain a first candidate word probability distribution of each sample of the migration dataset; translating the source text of each sample of the migration dataset using a machine translation student model to obtain a second candidate word probability distribution of each sample of the migration dataset; determining a migration loss of each sample of the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution, and a standard candidate word probability distribution of each sample of the migration dataset; and based on the first candidate word probability distribution and the migration loss of each sample of the migration dataset, realizing knowledge learning of the machine translation student model on the machine translation teacher model; wherein the machine translation student model and the machine translation teacher model are both used to realize language translation under a preset language pair; each sample in the migration dataset is composed of source text segmentation and translation segmentation under the preset language pair, and determines a unique standard candidate word probability distribution.

[0132] In addition, the logical instructions in the memory 330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0133] In a fourth aspect, the present application further provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and which, when executed by a processor, enables a computer to perform the knowledge learning method of the machine translation model provided by the above method, and the method comprises: constructing a migration dataset; translating the source text of each sample in the migration dataset by using a machine translation teacher model to obtain the first candidate word probability distribution of each sample in the migration dataset; translating the source text of each sample in the migration dataset by using a machine translation student model to obtain the second candidate word probability distribution of each sample in the migration dataset; determining the migration loss of each sample in the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration dataset; and based on the first candidate word probability distribution and the migration loss of each sample in the migration dataset, realizing the knowledge learning of the machine translation student model on the machine translation teacher model; wherein the machine translation student model and the machine translation teacher model are both used to realize language translation under a preset language pair; each sample in the migration dataset is composed of source text segmentation and translation text segmentation under the preset language pair, and determines a unique standard candidate word probability distribution.

[0134] In a fifth aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, realizes the knowledge learning method of the machine translation model provided by the above method, and the method comprises: constructing a migration dataset; translating the source text of each sample in the migration dataset by using a machine translation teacher model to obtain the first candidate word probability distribution of each sample in the migration dataset; translating the source text of each sample in the migration dataset by using a machine translation student model to obtain the second candidate word probability distribution of each sample in the migration dataset; determining the migration loss of each sample in the migration dataset according to the first candidate word probability distribution, the second candidate word probability distribution and the standard candidate word probability distribution of each sample in the migration dataset; and based on the first candidate word probability distribution and the migration loss of each sample in the migration dataset, realizing the knowledge learning of the machine translation student model on the machine translation teacher model; wherein the machine translation student model and the machine translation teacher model are both used to realize language translation under a preset language pair; each sample in the migration dataset is composed of source text segmentation and translation text segmentation under the preset language pair, and determines a unique standard candidate word probability distribution.

[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0136] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0137] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A knowledge learning method for a machine translation model, characterized in that, The method includes: Build a migration dataset; The original text of each sample in the transfer dataset is translated using a machine translation teacher model to obtain the probability distribution of the first candidate word for each sample in the transfer dataset; The original text of each sample in the transfer dataset is translated using a machine translation student model to obtain the probability distribution of the second candidate word for each sample in the transfer dataset; Based on the probability distributions of the first candidate word, the second candidate word, and the standard candidate word for each sample in the migration dataset, the migration loss for each sample in the migration dataset is determined. Based on the probability distribution of the first candidate word and the transfer loss of each sample in the transfer dataset, the machine translation student model learns knowledge from the machine translation teacher model. The machine translation student model and the machine translation teacher model are both used to achieve language translation under a preset language pair; each sample in the transfer dataset consists of word segmentation of the original text and word segmentation of the target text under the preset language pair, and determines a unique standard candidate word probability distribution.

2. The knowledge learning method for the machine translation model according to claim 1, characterized in that, The step of determining the migration loss for each sample in the migration dataset based on the probability distributions of the first candidate word, the second candidate word, and the standard candidate word for each sample includes: Calculate the cross-entropy of the probability distribution of the first candidate word of each sample in the migration dataset with respect to the probability distribution of the standard candidate word of each sample in the migration dataset, and denote it as the first cross-entropy of each sample in the migration dataset; Based on the first cross-entropy of each sample in the migration dataset and the second candidate word probability distribution and standard candidate word probability distribution of each sample in the migration dataset, the proportion coefficient of each sample in the migration dataset is determined; Calculate the KL divergence between the probability distribution of the first candidate word of each sample in the migration dataset and the probability distribution of the second candidate word of each sample in the migration dataset, and record it as the first KL divergence of each sample in the migration dataset; Calculate the KL divergence between the probability distribution of the first candidate word of each sample in the migration dataset and the probability distribution of the third candidate word pre-stored in the knowledge repository, and record it as the second KL divergence of each sample in the migration dataset. Based on the first cross-entropy, scaling factor, first KL divergence and second KL divergence of each sample in the migration dataset, determine the migration loss of each sample in the migration dataset; The proportion coefficient of each sample in the transfer dataset represents the learning direction of the machine translation student model on the "probability distribution of the first candidate word of each sample in the transfer dataset" passed by the machine translation teacher model. The third candidate word probability distribution is the candidate word probability distribution obtained by translating the original text of each sample in the transfer dataset using the machine translation teacher model learned from the machine translation student model.

3. The knowledge learning method for the machine translation model according to claim 2, characterized in that, The determination of the proportion coefficient for each sample in the migration dataset based on the first cross-entropy of each sample in the migration dataset, the second candidate word probability distribution, and the standard candidate word probability distribution of each sample in the migration dataset includes: Calculate the cross-entropy of the second candidate word probability distribution of each sample in the migration dataset with respect to the standard candidate word probability distribution of each sample in the migration dataset, and denote it as the second cross-entropy of each sample in the migration dataset; Based on the first cross-entropy and the second cross-entropy of each sample in the migration dataset, the proportion coefficient of each sample in the migration dataset is determined.

4. The knowledge learning method for the machine translation model according to claim 3, characterized in that, When the proportion coefficient of each sample in the transfer dataset is -1, it indicates that the learning direction of the machine translation student model on the "probability distribution of the first candidate word of each sample in the transfer dataset" passed by the machine translation teacher model is negative; when the proportion coefficient of each sample in the transfer dataset is 1, it indicates that the learning direction of the machine translation student model on the "probability distribution of the first candidate word of each sample in the transfer dataset" passed by the machine translation teacher model is positive. The determination of the proportion coefficient for each sample in the migration dataset based on the first cross-entropy and the second cross-entropy of each sample includes: If the second cross-entropy of each sample in the migration dataset is greater than the first cross-entropy, then the proportional coefficient of each sample in the migration dataset is -1. If the second cross-entropy of each sample in the migration dataset is not greater than the first cross-entropy, then the scaling factor of each sample in the migration dataset is 1.

5. The knowledge learning method for the machine translation model according to claim 2, characterized in that, The step of determining the migration loss for each sample in the migration dataset based on the first cross-entropy, scaling factor, first KL divergence, and second KL divergence of each sample includes: The product of the scaling factor and the first KL divergence of each sample in the migration dataset, the negative value of the first cross-entropy of each sample in the migration dataset, and the sum of the second KL divergence of each sample in the migration dataset are used as the migration loss of each sample in the migration dataset.

6. The knowledge learning method for the machine translation model according to any one of claims 1 to 5, characterized in that, The process of enabling the machine translation student model to learn knowledge from the machine translation teacher model based on the probability distribution of the first candidate word and the transfer loss of each sample in the transfer dataset includes: Based on the probability distribution of the first candidate word and the migration loss of each sample in the migration dataset, gradient descent optimization is performed on the network parameters in the machine translation student model.

7. The knowledge learning method for the machine translation model according to any one of claims 2 to 4, characterized in that, After the machine translation student model learns the knowledge of the machine translation teacher model, the probability distribution of the third candidate word stored in the knowledge repository is updated to the probability distribution of the first candidate word for each sample in the transfer dataset; In the absence of a pre-stored probability distribution of a third candidate word in the knowledge repository, the second KL divergence of each sample in the migration dataset is set to 0.

8. A knowledge learning device for a machine translation model, characterized in that, The device includes: The building block is used to construct the migration dataset; The first translation module is used to translate the original text of each sample in the transfer dataset using a machine translation teacher model, and to obtain the probability distribution of the first candidate word for each sample in the transfer dataset. The second translation module is used to translate the original text of each sample in the transfer dataset using a machine translation student model, and to obtain the probability distribution of the second candidate word for each sample in the transfer dataset. The migration loss calculation module is used to determine the migration loss of each sample in the migration dataset based on the probability distribution of the first candidate word, the probability distribution of the second candidate word, and the probability distribution of the standard candidate word for each sample in the migration dataset. The knowledge learning module is used to enable the machine translation student model to learn knowledge from the machine translation teacher model based on the probability distribution of the first candidate word and the transfer loss of each sample in the transfer dataset. The machine translation student model and the machine translation teacher model are both used to achieve language translation under a preset language pair; each sample in the transfer dataset consists of word segmentation of the original text and word segmentation of the target text under the preset language pair, and determines a unique standard candidate word probability distribution.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the knowledge learning method of the machine translation model as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the knowledge learning method of the machine translation model as described in any one of claims 1 to 7.

Images