A data distillation method, medium and device for machine translation

Abstract

The present invention belongs to the technical field of machine translation, and provides a data distillation method, medium and device for machine translation. The present invention introduces label distillation into the data distillation algorithm, which can make the label vector in the distilled data set contain more information; introduces the concept of attention label distillation into the data distillation algorithm, which greatly improves the performance of data distillation; compared with the existing technology, it greatly reduces the scale of training data, reduces the training time and computing power overhead.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine translation, and in particular, relates to a data distillation method, medium and device for machine translation. Background Art

[0002] Deep learning models can achieve very good performance in many fields, including computer vision and natural language processing, through large-scale neural networks and a large amount of training data. However, as the amount of data and model parameters increase, training a deep learning model requires a lot of training time and graphics card resources. In order to reduce the training cost, the data needs to be compressed. At present, the main data compression method is data selection, which mainly reduces the size of the original data through manual or clustering methods. However, such methods make it difficult for the compressed data to achieve model performance similar to the original data, and manual methods require a lot of labor costs.

[0003] Data distillation methods can distill and compress the original data set into a very small comprehensive data set, so that the model can be quickly trained and learned in the distilled data set, greatly shortening the model training time and computing power overhead, and keeping the model close to the original data. However, existing data distillation methods are only applied to simple text classification data sets and traditional text processing models. There are no related methods and research on applying data distillation methods to text generation tasks, such as machine translation. Summary of the invention

[0004] The purpose of the present invention is to provide a data distillation method for machine translation to solve the technical problems existing in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A data distillation method for machine translation, comprising the following steps:

[0007] Step S1: Initialize the distillation dataset parameters;

[0008] Step S2: setting training hyperparameters;

[0009] Step S3: Learn the distillation data set and determine whether the relationship between the current training batch s and the total batch S satisfies: s<S. If so, execute steps S4 to S7. If not, execute step S10. The total batch S is a set value.

[0010] Step S4: Initialize the parameters of the machine translation model ;

[0011] Step S5: Distilling the dataset Training a machine translation model in: Parameters of the machine translation model On the distilled dataset Gradient descent update is performed in , and the gradient descent update formula is as follows:

[0012]

[0013] in, is the loss function of the machine translation model, is the learning rate, t is the current number of gradient descents;

[0014] And determine whether the relationship between the current gradient descent times t and the preset gradient descent times T satisfies: t<T, if not, execute steps S6~S7, if yes, execute steps S8~S9;

[0015] Step S6: Calculate the Loss of the distilled dataset: , ; In the formula, represents the distilled attention label, It is distilled data Loss;

[0016] Step S7: Based on the optimization function, update the corresponding distillation target parameters: ,in, is an adjustable weight hyperparameter; and returns to step S3 to determine the relationship between the current training batch s and the total batch S;

[0017] Step S8: Obtain the tth batch of the distillation data set;

[0018] Step S9: Calculate the distillation label of the distillation dataset: Calculate the distillation attention label, the gradient of the learning rate, and update the model parameters: , and return to step S5 to determine the relationship between the current gradient descent times t and the preset gradient descent times T;

[0019] Step S10: Use the total batch S of original data to update and output the distilled dataset And the optimized learning rate :

[0020]

[0021]

[0022]

[0023] In the formula, S represents the total batch, Represents the learning rate of the distillation dataset.

[0024] Preferably, in step S1, the parameters of the distillation data set are initialized randomly, and the parameters are random vector data, which are randomized according to a normal distribution.

[0025] Preferably, the hyperparameter in step S2 is the learning rate used for training. Setting the training hyperparameter is to set the initial learning rate Initialize to .

[0026] Preferably, in step S5, when the machine translation model is in the distillation dataset After T gradient descents, the effect is close to the original data:

[0027]

[0028]

[0029] in, is a batch in the original data set, and the original data refers to the distilled data.

[0030] Preferably, the calculation method of the distilled attention label is as follows:

[0031] calculate The KL divergence between the self-attention probability value and the distilled attention label:

[0032]

[0033] in, is the probability distribution of the initial parameters of the machine translation model, is the kth attention head and hth layer of the distilled attention label, is the kth attention head and hth layer of the training model; Indicates the calculation of KL divergence;

[0034] The final distilled attention labels are obtained by optimizing the difference between the self-attention probability distribution and the distilled attention label distribution.

[0035] In order to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the data distillation method for machine translation.

[0036] In order to achieve the above-mentioned purpose, the present invention also provides a data distillation device for machine translation, comprising: a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the data distillation device for machine translation executes the data distillation method for machine translation.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention introduces label distillation into the data distillation algorithm, which can make the label vector in the distilled data set contain more information; introduces the concept of attention label distillation into the data distillation algorithm, which greatly improves the performance of data distillation; compared with the existing technology, it greatly reduces the scale of training data, and reduces the training time and computing power overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the process in Example 1 of the present invention.

[0040] Figure 2 This is a principle block diagram of Example 3 of the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to have a clearer understanding and knowledge of the present invention, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described below are only used to explain the present invention, and are convenient for understanding. The technical solutions provided by the present invention are not limited to the technical solutions provided by the following embodiments, and the technical solutions provided by the embodiments should not limit the protection scope of the present invention.

[0042] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0043] Example 1

[0044] like Figure 1 As shown, this embodiment provides a data distillation method for machine translation. This method compresses data to a length much smaller than the original data through the "soft label" and "distilled attention label" methods based on data distillation, so that the machine translation model can be trained and learned quickly, while achieving model performance similar to the original data set.

[0045] Assume the original training dataset: ,in, is the translation data pair of the original text and the translation data, and N is the length of the original data set. The purpose of the present invention is to compress the original data set D into a distilled data set: , where M is the length of the distilled dataset, and in particular M is much smaller than N.

[0046] The specific steps of this method are as follows:

[0047] Step S1: Initialize the parameters of the distillation data set; the parameters of the distillation data set are initialized randomly, and the parameters are random vector data, which are randomized according to the normal distribution.

[0048] Step S2: Set training hyperparameters; hyperparameters are the learning rate used in training. Setting training hyperparameters is to set the initial learning rate Initialize to .

[0049] Step S3: Learn the distilled data set and determine whether the relationship between the current training batch s and the total batch S satisfies: s<S. If so, execute steps S4~S7; if not, execute step S10; wherein the total batch S is a set value; this step is an outer loop step.

[0050] Step S4: Initialize the parameters of the machine translation model ;

[0051] Step S5: Parameters of the machine translation model On the distilled dataset Gradient descent update is performed in , and the gradient descent update formula is as follows:

[0052]

[0053] in, is the loss function of the machine translation model, is the learning rate, t is the current number of gradient descents;

[0054] And determine whether the relationship between the current gradient descent times t and the preset gradient descent times T satisfies: t<T, if not, execute steps S6~S7, if yes, execute steps S8~S9; this step to step S9 are inner loop steps;

[0055] The goal of this invention is to enable the model to achieve a similar effect to the original data after T gradient descents in the distilled data set:

[0056]

[0057]

[0058] in is a mini-batch in the original dataset. Therefore, the optimization problem of distilled data can be transformed into:

[0059]

[0060] in is the probability distribution of the initial parameters of the machine translation model. In order to make the distilled data and the original data set obtain similar model training performance, the above formula can be transformed into:

[0061] .

[0062] The label distillation method introduced in this example:

[0063] The traditional original data distillation method is limited to the input data distillation of the source data, and the label is a simple one-hot encoding. Usually the label is output by the probability distribution of the label at the last softmax layer of the model, and then the index with the maximum probability is selected as the output (prediction) of the model. However, usually a piece of data not only contains the label information, but also contains the feature information of other labels. For example, in the handwritten digital picture, picture 3 also contains some digital features of picture 8. The data distillation method that includes label distillation can make the distilled data contain more feature information. Therefore, the distilled data set with the original label as the target A distilled dataset that can be converted to a distilled label , and the gradient descent update formula for the distilled dataset is:

[0064] a

[0065] Step S6: Calculate the Loss of the distilled dataset: distill the attention label and the gradient of the learning rate: , ;middle, represents the distilled attention label, It is distilled data Loss;

[0066] This embodiment introduces the distilled attention label based on the transformer structure: In order to enable the distilled data set to obtain transfer learning capabilities in the transformer structure model, the present invention introduces the distilled attention mechanism label. After the attention mechanism label can be optimized and learned, when training and learning in the distilled data set, the training effect of the multi-head attention mechanism module is enhanced. Computational model The KL divergence between the self-attention probability value and the distilled attention label:

[0067]

[0068] in is the kth attention head and hth layer of the distilled attention label, and is the kth attention head and hth layer of the training model. The final distilled attention label is obtained by optimizing the difference between the self-attention probability distribution and the distilled attention label distribution. Due to the introduction of the distilled attention label, the optimization function of the original data distillation can be transformed into:

[0069]

[0070] in is a weight hyperparameter that can be adjusted.

[0071] Step S7: Based on the optimization function, update the corresponding distillation target parameters: ,in, is an adjustable weight hyperparameter; and returns to step S3 to determine the relationship between the current training batch s and the total batch S;

[0072] Step S8: Get the tth batch of the distilled dataset (data shard, training set);

[0073] Step S9: Calculate the distillation label of the distillation dataset: Calculate the distillation attention label, the gradient of the learning rate, and update the model parameters: , and return to step S5 to determine the relationship between the current gradient descent times t and the preset gradient descent times T;

[0074] Step S10: Use the total batch S of original data to update and output the distilled dataset And the optimized learning rate :

[0075] .

[0076] In the formula, S represents the total batch, Represents the learning rate of the distillation dataset.

[0077] The following is an example of the steps to implement the method provided in this embodiment:

[0078] Input: original machine translation dataset , the probability distribution of the initial parameters , the number of outer loops , the number of inner loops , the initial learning rate , distillation dataset learning rate , weight hyperparameter ;

[0079] 1: Randomly initialize the distillation dataset: ;

[0080] 2: Initialize learning rate: ;

[0081] 3: Outer loop (learning distilled dataset) :

[0082] 4: Initialize model parameters: ;

[0083] 5: Inner loop (training machine translation model on distilled dataset) ;

[0084] 6: Get The tth batch ;

[0085] 7: Calculation ;

[0086] 8: Calculation ;

[0087] 9: Update model parameters: ;

[0088] 10: End the inner loop;

[0089] 11: Update the distilled dataset using batches of original data:

[0090] .

[0091] Verification example:

[0092] Set the initial probability distribution is randomly distributed, and the length of distilled data is , and the original dataset length , the number of gradient descents is set to , weight hyperparameter .use The transformer model has 12 layers for encoders and 6 layers for decoders. The dropout is 0.1. By using the data distillation method of the present invention, the original The Chinese-English translation data was distilled and compressed to 1,000, which greatly reduced the size of the training data. The sub-gradient descent target also greatly reduces the training time and computing power. In traditional machine translation training, the original training data set In the distillation dataset of the present invention, it takes 112 hours of training to reach a Bleu value of 26.812 in the test set using a single A100 (40G) graphics card. In the distillation dataset of the present invention, it only takes 4 hours to reach a Bleu value of 25.967 in the same test set using a single A100 (40G) graphics card. At the cost of losing less than 1.0 Bleu value, the time or computing power cost of the training stage is greatly reduced.

[0093] Example 2

[0094] This embodiment provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the data distillation method for machine translation provided in Embodiment 1. A person skilled in the art can understand that all or part of the steps of implementing the method provided in Embodiment 1 can be completed by hardware related to the computer program, and the above-mentioned computer program can be stored in a computer-readable storage medium, and when the program is executed, the steps of the method provided in Embodiment 1 are executed; and the above-mentioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0095] Example 3

[0096] like Figure 2 As shown, this embodiment provides a data distillation device for machine translation, including: a processor and a memory; the memory is used to store computer programs; the processor is connected to the memory, and is used to execute the computer programs stored in the memory, so that the data distillation device for machine translation executes the data distillation method for machine translation provided in Example 1.

[0097] Specifically, the memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0098] Preferably, the processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0099] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A data distillation method for machine translation, characterized in that: The following steps are involved: Step S1: Initialize the distillation dataset parameters; Step S2: setting training hyperparameters; Step S3: Learn the distillation data set and determine whether the relationship between the current training batch s and the total batch S satisfies: s<S. If so, execute steps S4 to S7. If not, execute step S10. The total batch S is a set value. Step S4: Initialize the parameters of the machine translation model ; Step S5: Distilling the dataset Training a machine translation model in: Parameters of the machine translation model On the distilled dataset Gradient descent update is performed in , and the gradient descent update formula is as follows: in, is the loss function of the machine translation model, is the learning rate, t is the current number of gradient descents; And determine whether the relationship between the current gradient descent times t and the preset gradient descent times T satisfies: t<T, if not, execute steps S6~S7, if yes, execute steps S8~S9; Step S6: Calculate the Loss of the distilled dataset: , ; In the formula, represents the distilled attention label, It is distilled data Loss; Step S7: Based on the optimization function, update the corresponding distillation target parameters: ,in, is an adjustable weight hyperparameter; and returns to step S3 to determine the relationship between the current training batch s and the total batch S; Step S8: Obtain the tth batch of the distillation data set; Step S9: Calculate the distillation label of the distillation dataset: Calculate the distillation attention label, the gradient of the learning rate, and update the model parameters: , and return to step S5 to determine the relationship between the current gradient descent times t and the preset gradient descent times T; Step S10: Use the total batch S of original data to update and output the distilled dataset And the optimized learning rate : In the formula, S represents the total batch, Represents the learning rate of the distillation dataset; The calculation method of the distilled attention label is as follows: calculate The KL divergence between the self-attention probability value and the distilled attention label: in, is the probability distribution of the initial parameters of the machine translation model, is the kth attention head and hth layer of the distilled attention label, is the kth attention head and hth layer of the training model; Indicates the calculation of KL divergence; the final distilled attention label is obtained by optimizing the difference between the self-attention probability distribution and the distilled attention label distribution.

2. The data distillation method for machine translation according to claim 1, characterized in that: In the step S1, the parameters of the distillation data set are initialized randomly, and the parameters are random vector data, which are randomized according to a normal distribution.

3. The data distillation method for machine translation according to claim 2, characterized in that: The hyperparameter in step S2 is the learning rate used for training. Setting the training hyperparameter is to set the initial learning rate Initialize to .

4. The data distillation method for machine translation according to claim 3, characterized in that: In step S5, when the machine translation model is in the distillation dataset After T gradient descents, the effect is close to the original data: in, is a batch in the original data set, and the original data refers to the distilled data; Update formula for gradient descent.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the data distillation method for machine translation as described in any one of claims 1 to 4.

6. A data distillation device for machine translation, characterized in that: include: Processor and memory; The memory is used to store computer programs; the processor is connected to the memory and is used to execute the computer programs stored in the memory, so that the data distillation device for machine translation performs the data distillation method for machine translation as described in any one of claims 1 to 4.

Images