Distillation method, apparatus, and computer-readable storage medium

Abstract

The application discloses a distillation method, a device and a computer readable storage medium, wherein the distillation method comprises the following steps: inputting a source end sentence into an encoder of a teacher model to obtain a first low-level hidden layer, a first middle-level hidden layer and a first high-level hidden layer; inputting the source end sentence into an encoder of a student model to obtain a second low-level hidden layer, a second middle-level hidden layer and a second high-level hidden layer; distilling the second low-level hidden layer based on the first low-level hidden layer to obtain a first distillation loss, distilling the second middle-level hidden layer based on the first middle-level hidden layer to obtain a second distillation loss, and distilling the second high-level hidden layer based on the first high-level hidden layer to obtain a third distillation loss; and optimizing the encoder of the student model according to the first distillation loss, the second distillation loss and the third distillation loss. The application aims to improve the performance of the distilled model.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a distillation method, apparatus, and computer-readable storage medium. Background Technology

[0002] With the development of deep learning, Transformer neural networks have been widely used in natural language processing (such as machine translation and automatic question answering) due to their superior performance. Meanwhile, to meet the translation needs of daily communication, it is necessary to develop a neural machine translation engine that can be deployed on mobile devices, enabling it to quickly and accurately meet daily translation needs in offline environments. In some scenarios, it is necessary to compress the Transformer neural network model.

[0003] In related technologies, knowledge distillation is used to compress models. The hidden layer information output by the teacher model guides the student model to fit the teacher model, achieving the computational performance of the teacher model with a smaller student model. In other words, the student model after the distillation process is equivalent to a compressed teacher model. However, current knowledge distillation methods use the final output layer of the teacher model to distill the final output layer of the student model. The intermediate layers of the student model cannot replicate the performance of the intermediate layers of the teacher model, resulting in the student model not fully replicating the computational performance of the teacher model, and thus, the distilled model exhibits poor performance.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a distillation method, apparatus, and computer-readable storage medium, which aims to improve the performance of the distilled model.

[0006] To achieve the above objectives, the present invention provides a distillation method, the distillation method comprising:

[0007] Input the source statement into the encoder of the teacher model to obtain the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer;

[0008] The source statement is input into the encoder of the student model to obtain the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer.

[0009] A first distillation loss is obtained by distilling the second low-level hidden layer based on the first low-level hidden layer; a second distillation loss is obtained by distilling the second intermediate hidden layer based on the first intermediate hidden layer; and a third distillation loss is obtained by distilling the second advanced hidden layer based on the first advanced hidden layer.

[0010] The encoder of the student model is optimized based on the first distillation loss, the second distillation loss, and the third distillation loss.

[0011] Optionally, the steps of distilling the second low-level hidden layer based on the first low-level hidden layer to obtain a first distillation loss, distilling the second intermediate hidden layer based on the first intermediate hidden layer to obtain a second distillation loss, and distilling the second high-level hidden layer based on the first high-level hidden layer to obtain a third distillation loss include:

[0012] The first low-level hidden layer, the first intermediate hidden layer, and / or the first high-level hidden layer are concatenated and then input into a fully connected network to obtain fully connected mapping results.

[0013] Distill the second low-level hidden layer, the second intermediate-level hidden layer, and the second high-level hidden layer according to the fully connected mapping result.

[0014] Optionally, after the step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss, the method further includes:

[0015] The attention weight matrix of the decoder of the student model is decomposed into a low-rank value to obtain the decoder after low-rank decomposition.

[0016] The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

[0017] Optionally, the step of performing a low-rank decomposition on the attention weight matrix of the decoder of the student model to obtain the low-rank decomposed decoder includes:

[0018] Obtain the weight matrices of the multi-head self-attention and multi-head cross-attention of the decoder;

[0019] The weight matrix is ​​subjected to singular value decomposition to obtain the decoder after low-rank decomposition.

[0020] Optionally, after the step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss, the method further includes:

[0021] The parameter values ​​of the multi-head cross attention matrix of the decoder of the student model are determined based on the hidden layer of the last level in the first high-level hidden layer.

[0022] The decoder is trained based on the parameter values.

[0023] Optionally, after the step of training the decoder based on the parameter values, the method further includes:

[0024] The attention weight matrix of the decoder of the student model is decomposed into a low-rank value to obtain the decoder after low-rank decomposition.

[0025] The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

[0026] Optionally, the step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss includes:

[0027] Multiple auxiliary learning tasks are constructed based on the first distillation loss, the second distillation loss, and the third distillation loss;

[0028] The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

[0029] Optionally, the step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss includes:

[0030] Obtain the model loss of the student model;

[0031] Determine the root mean square error estimates of the first distillation loss, the second distillation loss, and the third distillation loss;

[0032] Multiple auxiliary learning tasks are constructed based on the model loss and the mean squared error estimate;

[0033] The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

[0034] In addition, to achieve the above objectives, the present invention also provides a distillation apparatus, the distillation apparatus including a memory, a processor, and a distillation program stored in the memory and executable on the processor, wherein the distillation program, when executed by the processor, implements the steps of the distillation method as described above.

[0035] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a distillation program, which, when executed by a processor, implements the steps of the distillation method as described above.

[0036] This invention proposes a distillation method, distillation apparatus, and computer-readable storage medium. First, a source statement is input into the encoder of a teacher model to obtain a first low-level hidden layer, a first intermediate-level hidden layer, and a first high-level hidden layer. Then, the source statement is input into the encoder of a student model to obtain a second low-level hidden layer, a second intermediate-level hidden layer, and a second high-level hidden layer. The second low-level hidden layer is distilled based on the first low-level hidden layer to obtain a first distillation loss. The second intermediate-level hidden layer is distilled based on the first intermediate-level hidden layer to obtain a second distillation loss. Finally, the second high-level hidden layer is distilled based on the first high-level hidden layer to obtain a third distillation loss. The encoder of the student model is then optimized based on the first distillation loss, the second distillation loss, and the third distillation loss. Based on the input from the same source statement, the low-level, mid-level, and high-level hidden layers of the teacher model's encoder are distilled using the encoder's low-level, mid-level, and high-level hidden layers. The low-level, mid-level, and high-level encoding layers of the student model's encoder can respectively fit the low-level, mid-level, and high-level encoding layers of the teacher model's encoder. This allows the intermediate layers of the student model to replicate the performance of the teacher model's intermediate layers, enabling the student model to further replicate the excellent computational performance of the teacher model, thereby improving the performance of the distilled student model. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention;

[0038] Figure 2 This is a schematic flowchart of an embodiment of the distillation method of the present invention;

[0039] Figure 3 This is a schematic flowchart of another embodiment of the distillation method of the present invention;

[0040] Figure 4 This is a diagram of the model optimization architecture involved in the embodiments of the present invention;

[0041] Figure 5 This is a diagram of the model inference architecture involved in an embodiment of the present invention;

[0042] Figure 6 This is a diagram of the encoder distillation architecture according to an embodiment of the present invention.

[0043] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0045] In related technologies, knowledge distillation uses the final output layer of the teacher model to distill the final output layer of the student model. The intermediate layers of the student model cannot replicate the performance of the intermediate layers of the teacher model, resulting in the student model not being able to fully replicate the computational performance of the teacher model, and the performance of the distilled model being poor.

[0046] To improve the performance of the model after distillation, embodiments of the present invention propose a distillation method, a distillation apparatus, and a computer-readable storage medium, wherein the main steps of the method include:

[0047] Input the source statement into the encoder of the teacher model to obtain the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer;

[0048] The source statement is input into the encoder of the student model to obtain the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer.

[0049] A first distillation loss is obtained by distilling the second low-level hidden layer based on the first low-level hidden layer; a second distillation loss is obtained by distilling the second intermediate hidden layer based on the first intermediate hidden layer; and a third distillation loss is obtained by distilling the second advanced hidden layer based on the first advanced hidden layer.

[0050] The encoder of the student model is optimized based on the first distillation loss, the second distillation loss, and the third distillation loss.

[0051] This allows the intermediate levels of the student model to replicate the performance of the intermediate levels of the teacher model, enabling the student model to further replicate the excellent computational performance of the teacher model, thereby improving the performance of the distilled student model.

[0052] The claims of this invention will be described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention.

[0054] In this embodiment of the invention, the terminal can be a distillation apparatus.

[0055] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a memory 1003, and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The memory 1003 may be a high-speed RAM or a stable, non-volatile memory, such as a disk drive. Optionally, the memory 1003 may also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1 As shown, the memory 1003, which serves as a computer storage medium, may include an operating system and a distillation program.

[0058] exist Figure 1 In the terminal shown, the processor 1001 can be used to call the distillation program stored in the memory 1003 and perform the following operations:

[0059] Input the source statement into the encoder of the teacher model to obtain the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer;

[0060] The source statement is input into the encoder of the student model to obtain the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer.

[0061] A first distillation loss is obtained by distilling the second low-level hidden layer based on the first low-level hidden layer; a second distillation loss is obtained by distilling the second intermediate hidden layer based on the first intermediate hidden layer; and a third distillation loss is obtained by distilling the second advanced hidden layer based on the first advanced hidden layer.

[0062] The encoder of the student model is optimized based on the first distillation loss, the second distillation loss, and the third distillation loss.

[0063] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0064] The first low-level hidden layer, the first intermediate hidden layer, and / or the first high-level hidden layer are concatenated and then input into a fully connected network to obtain fully connected mapping results.

[0065] Distill the second low-level hidden layer, the second intermediate-level hidden layer, and the second high-level hidden layer according to the fully connected mapping result.

[0066] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0067] The attention weight matrix of the decoder of the student model is decomposed into a low-rank value to obtain the decoder after low-rank decomposition.

[0068] The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

[0069] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0070] Obtain the weight matrices of the multi-head self-attention and multi-head cross-attention of the decoder;

[0071] The weight matrix is ​​subjected to singular value decomposition to obtain the decoder after low-rank decomposition.

[0072] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0073] The parameter values ​​of the multi-head cross attention matrix of the decoder of the student model are determined based on the hidden layer of the last level in the first high-level hidden layer.

[0074] The decoder is trained based on the parameter values.

[0075] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0076] The attention weight matrix of the decoder of the student model is decomposed into a low-rank value to obtain the decoder after low-rank decomposition.

[0077] The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

[0078] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0079] Multiple auxiliary learning tasks are constructed based on the first distillation loss, the second distillation loss, and the third distillation loss;

[0080] The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

[0081] Furthermore, the processor 1001 can call the distillation program stored in the memory 1003 and also perform the following operations:

[0082] Obtain the model loss of the student model;

[0083] Determine the root mean square error estimates of the first distillation loss, the second distillation loss, and the third distillation loss;

[0084] Multiple auxiliary learning tasks are constructed based on the model loss and the mean squared error estimate;

[0085] The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

[0086] With the development of deep learning, Transformer neural networks have been widely used in natural language processing (such as machine translation and automatic question answering) due to their superior performance. Meanwhile, to meet the translation needs of daily communication, it is necessary to develop a neural machine translation engine that can be deployed on mobile devices, enabling it to quickly and accurately meet daily translation needs in offline environments. The excellent translation performance of online neural machine translation engines relies on high-performance servers to provide computational support for a large number of parameters. However, when running Transformer neural network models on mobile portable devices, these devices are severely limited by hardware resources and battery life, making it impossible to provide high-performance computing services. Therefore, it is necessary to compress Transformer neural network models to enable them to still provide fast and accurate translation tasks on mobile devices with limited hardware resources. Thus, in some scenarios, it is necessary to compress Transformer neural network models.

[0087] In related technologies, knowledge distillation is also used to compress models. The hidden layer information output by the teacher model guides the student model to fit the teacher model, achieving the computational performance of the teacher model with a smaller student model. In other words, the student model after the distillation operation is equivalent to a compressed teacher model. However, current knowledge distillation methods use the final output layer of the teacher model to distill the final output layer of the student model. The intermediate layers of the student model cannot replicate the performance of the intermediate layers of the teacher model, resulting in the student model not fully replicating the computational performance of the teacher model, and thus the distilled model exhibits poor performance.

[0088] Therefore, the aforementioned defects exist in the relevant distillation methods. To address these defects, this invention proposes a distillation method that aims to improve the performance of the distilled student model by using the low-level, mid-level, and high-level hidden layers of the teacher model's encoder, through input from the same source statement, to distill these layers.

[0089] The following explanation, through specific exemplary solutions, clarifies the scope of protection claimed in the claims of this invention, so that those skilled in the art can better understand the scope of protection of the claims. It is understood that the following exemplary solutions do not limit the scope of protection of this invention, but are only used to explain this invention.

[0090] For example, refer to Figure 2 In one embodiment of the distillation method of the present invention, the distillation method includes the following steps:

[0091] Step S10: Input the source statement into the encoder of the teacher model to obtain the first low-level hidden layer, the first intermediate hidden layer and the first high-level hidden layer;

[0092] Step S20: Input the source statement into the encoder of the student model to obtain the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer;

[0093] In this embodiment, both the teacher model and the student model are machine translation neural network models that can process the input source text and output a translation in a specified language. However, the teacher model has higher computational performance and accuracy than the student model. The architecture of the teacher model and the student model is generally not limited, but for model compression, the architecture of the student model needs to be smaller than that of the teacher model. In this way, the distilled student model can replicate the performance of the teacher model and has the same translation ability as the teacher model. It can replace the teacher model to perform translation tasks, but its architecture is smaller than that of the teacher model. Thus, the distilled student model is equivalent to compressing the teacher model.

[0094] A complete translation model can perform translation tasks only if it has an encoder and a decoder. If the teacher model or the student model needs to complete the task independently, it needs to have an encoder and a decoder respectively. The steps to perform a translation task are: input the source text into the encoder of the translation model, after passing through multiple encoding layers of the encoder, output the last hidden layer, input the last hidden layer into the translation decoder, and obtain the translation.

[0095] In this embodiment, the main focus is on distilling the model's encoder. The encoder and decoder can be freely split and combined according to distillation requirements, or they can be trained independently, as described above. Figure 4 The teacher model and the student model can share the same encoder. In this embodiment, only one encoder for the teacher model and one encoder for the student model are needed to complete the distillation and optimize the encoder of the student model. The optimized encoder can be combined with the decoder of any translation model to achieve the overall optimization of the student model. The decoder is not needed in the distillation stage.

[0096] Reference Figure 4 The encoders of both the student and teacher models can each include multiple multi-head self-attention encoding layers. A complete encoder for either a student or teacher model includes positional encoding, word vector encoding, and multiple multi-head self-attention encoding layers. After the source sentence is input, it passes through word vector encoding and positional encoding before entering multiple encoding layers. These encoding layers are hierarchical, with each layer outputting a hidden layer. This output hidden layer is then input into the next level of encoding layer. For example, after the i-th encoding layer outputs a hidden layer, that hidden layer enters the i+1-th encoding layer for further processing. Therefore, the higher the level, the higher the level of the output hidden layer, and correspondingly, the more information is contained within the hidden layer.

[0097] Optionally, before step S10, the method further includes: dividing multiple encoding layers in the encoder of the teacher model into a first low-level compilation layer, a first intermediate compilation layer, and a first high-level compilation layer according to a first preset partitioning rule or the structural characteristics of the teacher model; and dividing multiple encoding layers in the encoder of the student model into a second low-level compilation layer, a second intermediate compilation layer, and a second high-level compilation layer according to a second preset partitioning rule or the structural characteristics of the student model. Step S10 includes: inputting source sentences from parallel corpora into the encoder of the teacher model, obtaining a first low-level hidden layer through the first low-level compilation layer, obtaining a first intermediate hidden layer through the first low-level hidden layer and the first intermediate compilation layer, and obtaining a first high-level hidden layer through the first intermediate hidden layer and the first high-level compilation layer. Step S20 includes: inputting source sentences from parallel corpora into the encoder of the student model, obtaining a second low-level hidden layer through the second low-level compilation layer, obtaining a second intermediate hidden layer through the second low-level hidden layer and the second intermediate compilation layer, and obtaining a second high-level hidden layer through the second intermediate hidden layer and the first high-level compilation layer.

[0098] In translation models, multiple coding layers can be divided into low-level, mid-level, and high-level coding layers based on the structural characteristics of the encoder or preset rules. Low-level coding layers can correspond to lexical layers, mid-level coding layers can correspond to syntactic layers, and high-level coding layers can correspond to semantic layers.

[0099] The low-level, mid-level, and high-level compilation layers each contain several compilation layers, but the number of compilation layers in each layer is different. For example, if the translation model includes six compilation layers, then the first encoding layer is the low-level compilation layer, the second and third encoding layers are the mid-level compilation layers, and the fourth, fifth, and sixth compilation layers are the high-level compilation layers. However, the number of layers in the low-level, mid-level, and high-level compilation layers increases gradually.

[0100] In the encoder of the teacher model, the hidden layer output by the low-level compilation layer is the first low-level hidden layer, the hidden layer output by the intermediate compilation layer is the first intermediate hidden layer, and the hidden layer output by the high-level compilation layer is the first high-level hidden layer; in the encoder of the student model, the hidden layer output by the low-level compilation layer is the second low-level hidden layer, the hidden layer output by the low-level compilation layer is the second low-level hidden layer, and the hidden layer output by the low-level compilation layer is the second low-level hidden layer.

[0101] Parallel corpora refer to bilingual / multilingual corpora consisting of source text and its parallel corresponding translation text. During translation, the text before translation is called the source statement, i.e., the source text in the parallel corpus, while the translated text is called the target statement. In this embodiment, the source statement is input into the compilers of the teacher model and the student model respectively. The teacher model and the student model process the student model based on the same source statement, respectively deriving the first low-level hidden layer, the first intermediate-level hidden layer, and the first high-level hidden layer, as well as the second low-level hidden layer, the second intermediate-level hidden layer, and the second high-level hidden layer.

[0102] Step S30: Distill the second low-level hidden layer based on the first low-level hidden layer to obtain a first distillation loss; distill the second intermediate hidden layer based on the first intermediate hidden layer to obtain a second distillation loss; and distill the second advanced hidden layer based on the first high-level hidden layer to obtain a third distillation loss.

[0103] In this embodiment, the hidden layers of the teacher model are distilled to the same level as the hidden layers of the student model. The number of coding layers in the teacher model and the student model may be different. For example, when compressing the model by distillation, the number of coding layers in the student model will be less than the number of coding layers in the teacher model. However, both the teacher model and the student model are translation models, and the processing ideas of translation models are consistent. Their multiple compilation layers can be divided into low-level hidden layers, intermediate hidden layers, and high-level hidden layers. Therefore, the intermediate layers in the teacher model can be distilled from the intermediate layers in the student model. The closer the output hidden layers of the same intermediate layer are, the higher the similarity between the two models.

[0104] Optionally, the first low-level hidden layer, the first intermediate hidden layer, and / or the first high-level hidden layer are concatenated and then input into a fully connected network to obtain fully connected mapping results; the second low-level hidden layer, the second intermediate hidden layer, and / or the second high-level hidden layer are distilled according to the fully connected mapping results.

[0105] The number of compiler layers in the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer is greater than or equal to one. Since the distillation method is one-to-one, it is necessary to select a representative compiler layer from multiple compiler layers of different levels as the main body of distillation. Optionally, different levels can be processed based on the last hidden layer. That is, based on the last hidden layer of the first low-level hidden layer, the last hidden layer of the second low-level hidden layer is distilled to obtain the first distillation loss. Based on the last hidden layer of the first intermediate hidden layer, the last hidden layer of the second intermediate hidden layer is distilled to obtain the second distillation loss. And based on the last hidden layer of the first high-level hidden layer, the last hidden layer of the second high-level hidden layer is distilled.

[0106] Alternatively, the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer with more than one level can be concatenated and then input into a fully connected network to obtain the fully connected mapping results corresponding to the first low-level hidden layer, the first intermediate hidden layer, and / or the first high-level hidden layer. The second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer can then be distilled based on the obtained fully connected mapping results.

[0107] Similarly, the number of layers in the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer may also be greater than 1. Therefore, the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer with a number of layers greater than 1 can be concatenated and then input into the fully connected network to obtain the fully connected mapping result corresponding to the second low-level hidden layer, the fully connected mapping result corresponding to the second intermediate hidden layer, and / or the fully connected mapping result corresponding to the second high-level hidden layer, and receive the distillation of the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer.

[0108] It should be noted that, in this embodiment, when the low, medium, and high hidden layers of the teacher model perform low, medium, and high model distillation on the student model, there is no need to uniformly limit the combination of hidden layers or fully connected mapping results selected for distillation at each level. For example, the first low-level hidden layer selects its last level hidden layer to distill the fully connected mapping hidden layer corresponding to the second low-level hidden layer, the first medium-level hidden layer selects its corresponding fully connected mapping hidden layer to distill the only hidden layer in the second medium-level hidden layer, and the first high-level hidden layer selects its corresponding fully connected mapping hidden layer to distill the fully connected mapping hidden layer corresponding to the second high-level hidden layer. However, in order to improve the consistency of distillation loss and the efficiency of distillation and multi-task assisted learning, the subject types used for distillation in different levels of hidden layers in the teacher model should be consistent, and the subject types used for distillation in different levels of hidden layers in the student model should be consistent. For example, regardless of the number of layers, the teacher model will concatenate the first low-level hidden layer, the first intermediate hidden layer, and the first high-level hidden layer and input them into the fully connected network to obtain the corresponding fully connected mapping results. Then, the last level of the second low-level hidden layer, the second intermediate hidden layer, and the second high-level hidden layer in the student model corresponding to the fully connected mapping results will be distilled.

[0109] Alternatively, since distillation is not interpretable enough, the distillation method is not stable enough. It is necessary to explore the architecture of the student model being distilled. In the process of compressing the machine translation model, the student model used to replace the teacher model is an encoder with three coding layers. In this way, the distilled model not only has fewer parameters, smaller architecture, and smaller size, but also improves the computing speed and achieves better computing performance.

[0110] Optionally, the hidden layers of each level of the teacher model are distilled to the student model, i.e., the distillation loss is obtained according to the distillation formula, which represents the difference between the output hidden layers of the teacher model and the output hidden layers of the student model.

[0111] In some alternative embodiments, refer to Figure 6 The encoder of the teacher model can include six multi-head self-attention encoding layers, and the encoder of the student model can include three multi-head self-attention encoding layers. The first distillation loss is obtained by distilling the first layer (second low-level hidden layer) of the student model using the concatenated fully connected mapping result of the first and second layers of the teacher model's encoding layers (first low-level hidden layers). The second distillation loss is obtained by distilling the second layer (second low-level hidden layer) of the student model using the concatenated fully connected mapping result of the third and fourth layers of the teacher model's encoding layers (first intermediate hidden layers). The third layer (second intermediate hidden layer) of the student model's encoding layers is distilled using the concatenated fully connected mapping result of the fifth and sixth layers of the teacher model's encoding layers (first high-level hidden layers). The specific calculation formula is as follows:

[0112]

[0113]

[0114] Where i represents the i-th layer of the encoder side of the student model. Let i represent the i-th hidden layer of the student model. Let F represent the j-th hidden layer of the teacher model, and let F denote the fully connected function. This represents the fully connected mapping result after concatenating the (2i-1)th and 2ith layers of the teacher model, where MSE represents the mean squared error function. This represents the loss from knowledge distillation at the i-th level.

[0115] Optionally, the distillation method provided in this embodiment can be applied to an encoder, which includes, as referenced Figure 4 The method employs positional encoding, word vector encoding, a three-layer multi-head self-attention encoding layer for the student model, a six-layer multi-head self-attention encoding layer for the teacher model, and three fully connected networks for distillation. The encoding layers from the encoders of both the teacher and student models are merged into a single encoder, sharing positional and word vector encoding. This encoder also includes fully connected networks to fuse hidden layers with multiple levels. When optimizing the student model using this method, the optimized student model's encoding layers, positional encoding, and word vector encoding are retained, thus saving space and reducing computational resource consumption.

[0116] Step S40: Optimize the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss.

[0117] In this embodiment, the distillation loss is the difference between the output of the teacher model and the output of the student model. The student model can continuously learn and optimize its algorithm based on the distillation loss, thereby achieving the goal of optimizing the student model and its encoder.

[0118] Optionally, multiple auxiliary learning tasks are constructed based on the first distillation loss, the second distillation loss, and the third distillation loss; the encoder of the student model is optimized based on the multiple auxiliary learning tasks.

[0119] The student model can be optimized sequentially or simultaneously based on multiple distillation losses. An auxiliary task can be constructed based on the first, second, and third distillation losses. The student model completes the auxiliary learning task, and upon completion, the optimization objective is achieved. Simultaneous optimization can reduce distillation time and improve optimization efficiency.

[0120] Optionally, the model loss of the student model is obtained; the mean squared error estimates of the first distillation loss, the second distillation loss, and the third distillation loss are determined; multiple auxiliary learning tasks are constructed based on the model loss and the mean squared error estimates; and the encoder of the student model is optimized based on the multiple auxiliary learning tasks.

[0121] Distillation loss is the loss determined by distillation between the teacher and student models, achieving optimization. The student model can also complete training tasks independently, reducing its own errors and improving accuracy. Specifically, a certain number of source and target sentences from a parallel corpus are input into the student model. The student model processes the source sentences to obtain its translation result, which is then compared with the target sentence to obtain the student model's model loss. Model loss represents the overall loss of the machine translation model and can also be used to optimize the student model, improving its accuracy and performance. The mean squared error estimates of the first, second, and third distillation losses are calculated. These mean squared error estimates and the model loss represent two dimensions of loss, used to construct multiple auxiliary learning tasks. The formula is as follows:

[0122]

[0123] Among them, L nmt It is the loss in neural machine translation. Let α represent the loss from knowledge distillation at the i-th level. iThese are the corresponding coefficients. Simultaneously optimizing the student model using both model loss and distillation loss can not only further improve the accuracy and performance of the student model, but also increase its optimization efficiency.

[0124] In the technical solution disclosed in this embodiment, the source statement is first input into the encoder of the teacher model to obtain a first low-level hidden layer, a first intermediate hidden layer, and a first high-level hidden layer; the source statement is then input into the encoder of the student model to obtain a second low-level hidden layer, a second intermediate hidden layer, and a second high-level hidden layer; the second low-level hidden layer is distilled based on the first low-level hidden layer to obtain a first distillation loss; the second intermediate hidden layer is distilled based on the first intermediate hidden layer to obtain a second distillation loss; and the second high-level hidden layer is distilled based on the first high-level hidden layer to obtain a third distillation loss; the encoder of the student model is optimized based on the first distillation loss, the second distillation loss, and the third distillation loss. Based on the same source statement input, the low-level, mid-level, and high-level hidden layers of the teacher model's encoder are distilled using the encoder's low-level, mid-level, and high-level hidden layers. Similarly, the low-level, mid-level, and high-level encoding layers of the student model's encoder can be fitted to the low-level, mid-level, and high-level encoding layers of the teacher model's encoder, respectively. This allows the intermediate layers of the student model's encoder to replicate the performance of the teacher model's intermediate layers, reducing the number of model parameters while maintaining the encoder's encoding performance. This enables the student model to further replicate the excellent computational performance of the teacher model, improving its generalization ability, enhancing the performance of the distilled student model, and strengthening its robustness.

[0125] Optionally, refer to Figure 3 Based on any of the above embodiments, in another embodiment of the distillation method of the present invention, the distillation method includes:

[0126] Step S50: Perform low-rank decomposition on the attention weight matrix of the decoder of the student model to obtain the decoder after low-rank decomposition.

[0127] In this embodiment, the student model includes an encoder and a decoder, as shown in the following example. Figure 4 The decoder mainly includes multi-head self-attention and multi-head cross-attention. Multi-head cross-attention, also known as multi-head cross-cross attention, can also include positional encoding and word encoding. Both types of attention have corresponding weight matrices. The weight matrices are decomposed into low-rank values, which reduces the number of model parameters and the amount of computation, while still ensuring the model's translation performance.

[0128] Optionally, the weight matrices of the multi-head self-attention and multi-head cross-attention of the decoder are obtained; singular value decomposition is performed on the weight matrices to obtain the decoder after low-rank decomposition.

[0129] A rank is set for different decoding layers in the decoder. Each decoding layer has multi-head self-attention and multi-head cross-attention. The weight matrix of the multi-head self-attention and multi-head cross-attention of each decoding layer is decomposed into a low-rank matrix that conforms to the rank value through singular value decomposition.

[0130] In one alternative implementation, the decoder comprises three decoding layers. The weight matrices of the multi-head self-attention instances and multi-head cross-attention instances in the first and second layers of the decoder are decomposed into multiple low-rank matrices using singular value decomposition (SVD), with each decomposition having a rank of 32. The weight matrices of the multi-head self-attention instances and multi-head cross-attention instances in the third layer of the decoder are also decomposed into multiple low-rank matrices using SVD, with each decomposition having a rank of 16. The formula is as follows:

[0131] W=UΣV T

[0132]

[0133] Where W is the weight matrix, U and V are unitary matrices, and the values ​​on the diagonal of Σ are singular values, u i It is in the U matrix based on λ i The corresponding left orthogonal vector, v i It is in the V matrix based on λ i The corresponding right orthogonal vectors, r denote the rank, λ1...λ r Arrange in descending order from largest to smallest.

[0134] Step S60: The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

[0135] In this embodiment, refer to Figure 4 The student model's encoder was optimized based on the first, second, and third distillation losses. The student model's decoder underwent low-rank decomposition. The distilled encoder and the low-rank decomposed decoder were retained, forming a complete neural translation model. This model is ultimately capable of performing machine translation inference tasks. (Refer to...) Figure 5 .

[0136] In an alternative embodiment, the parameter values ​​of the multi-head cross-attention matrix of the decoder of the student model are determined based on the last hidden layer in the first high-level hidden layer; the decoder is then trained based on the parameter values.

[0137] The encoder can be trained, and then the encoder of the distilled student model and the decoder of the trained student model can be combined to form a machine translation model.

[0138] Specifically, refer to Figure 4The target sentences of the parallel corpus are fed into the decoder of this network, but the decoder has not yet undergone low-rank decomposition. The decoder is trained based on the target sentences. Before training, the key and value values ​​of the multi-head cross-attention of the decoder are determined based on the last hidden layer in the first high-level hidden layer (i.e., the last layer of multiple output hidden layers obtained by the teacher model from the source sentences of the parallel corpus). The key and value values ​​are the aforementioned parameter values. The decoder is trained based on the parameter values ​​and the target sentences.

[0139] Optionally, the attention weight matrix of the decoder of the student model is decomposed into low-rank components to obtain a low-rank decomposed decoder; the combined distilled encoder of the student model and the low-rank decomposed decoder of the student model constitute a machine translation model. The trained encoder is further decomposed into low-rank components according to the above-described low-rank decomposition steps.

[0140] It is understandable that the distilled encoder can be combined with an untrained decoder without low-rank decomposition to form a machine translation model, or with a trained decoder without low-rank decomposition to form a machine translation model, or with an untrained decoder but with low-rank decomposition to form a machine translation model, or with a trained decoder with low-rank decomposition to form a machine translation model.

[0141] In the technical solution disclosed in this embodiment, the attention matrix of the decoder of the student model is decomposed into a low-rank value to obtain a low-rank decomposed decoder. The encoder of the student model after distillation and the decoder of the student model after low-rank decomposition are combined to form a machine translation model. In this way, by using low-rank decomposition for the attention weights in the decoder, the number of model parameters is reduced and the computational load of the model is decreased, while ensuring the translation performance of the model and improving the performance of the distilled model.

[0142] Furthermore, this invention also proposes a distillation apparatus, which includes a memory, a processor, and a distillation program stored in the memory and executable on the processor. When the distillation program is executed by the processor, it implements the steps of the distillation methods described in the above embodiments.

[0143] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a distillation program, which, when executed by a processor, implements the steps of the distillation method as described in the above embodiments.

[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0145] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause the distillation apparatus to execute the methods described in the various embodiments of the present invention.

[0147] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A distillation method, characterized in that, The distillation method includes: The source statement is input into the encoder of the teacher model to obtain the first low-level hidden layer, the first intermediate hidden layer and the first high-level hidden layer. The encoder of the teacher model includes six multi-head self-attention coding layers. The source statement is input into the encoder of the student model to obtain the second low-level hidden layer, the second intermediate hidden layer and the second high-level hidden layer. The encoder of the student model includes three multi-head self-attention coding layers. A first distillation loss is obtained by distilling the second low-level hidden layer based on the first low-level hidden layer; a second distillation loss is obtained by distilling the second intermediate hidden layer based on the first intermediate hidden layer; and a third distillation loss is obtained by distilling the second advanced hidden layer based on the first advanced hidden layer. The encoder of the student model is optimized based on the first distillation loss, the second distillation loss, and the third distillation loss; The steps of distilling the second low-level hidden layer based on the first low-level hidden layer to obtain a first distillation loss, distilling the second intermediate hidden layer based on the first intermediate hidden layer to obtain a second distillation loss, and distilling the second high-level hidden layer based on the first high-level hidden layer to obtain a third distillation loss include: The first distillation loss is obtained by distilling the first output hidden layer of the student model using the fully connected mapping result concatenated from the output hidden layers of the first and second coding layers of the teacher model. The second distillation loss is obtained by distilling the second output hidden layer of the student model using the fully connected mapping result concatenated from the output hidden layers of the third and fourth coding layers of the teacher model. The third distillation loss is obtained by distilling the third output hidden layer of the student model using the fully connected mapping result concatenated from the output hidden layers of the fifth and sixth coding layers of the teacher model. The step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss further includes: The source and target sentences from the parallel corpus are input into the student model. The student model processes the source sentences to obtain the translation result. The translation result is compared with the target sentence to obtain the model loss of the student model. The mean squared error estimates of the first distillation loss, the second distillation loss, and the third distillation loss are determined. Multiple auxiliary learning tasks are constructed based on the model loss and the mean squared error estimates. The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

2. The distillation method as described in claim 1, characterized in that, The steps of distilling the second low-level hidden layer based on the first low-level hidden layer to obtain a first distillation loss, distilling the second intermediate hidden layer based on the first intermediate hidden layer to obtain a second distillation loss, and distilling the second high-level hidden layer based on the first high-level hidden layer to obtain a third distillation loss include: The first low-level hidden layer, the first intermediate hidden layer, and / or the first high-level hidden layer are concatenated and then input into a fully connected network to obtain fully connected mapping results. Distill the second low-level hidden layer, the second intermediate-level hidden layer, and the second high-level hidden layer according to the fully connected mapping result.

3. The distillation method as described in claim 1, characterized in that, After the step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss, the method further includes: The attention weight matrix of the decoder of the student model is decomposed into a low-rank value to obtain the decoder after low-rank decomposition. The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

4. The distillation method as described in claim 3, characterized in that, The step of performing low-rank decomposition on the attention weight matrix of the decoder of the student model to obtain the low-rank decomposed decoder includes: Obtain the weight matrices of the multi-head self-attention and multi-head cross-attention of the decoder; The weight matrix is ​​subjected to singular value decomposition to obtain the decoder after low-rank decomposition.

5. The distillation method as described in claim 1, characterized in that, After the step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss, the method further includes: The parameter values ​​of the multi-head cross attention matrix of the decoder of the student model are determined based on the hidden layer of the last level in the first high-level hidden layer. The decoder is trained based on the parameter values.

6. The distillation method as described in claim 5, characterized in that, After the step of training the decoder based on the parameter values, the method further includes: The attention weight matrix of the decoder of the student model is decomposed into a low-rank value to obtain the decoder after low-rank decomposition. The encoder of the student model after combined distillation and the decoder of the student model after low-rank decomposition constitute a machine translation model.

7. The distillation method as described in claim 1, characterized in that, The step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss includes: Multiple auxiliary learning tasks are constructed based on the first distillation loss, the second distillation loss, and the third distillation loss; The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

8. The distillation method as described in claim 1, characterized in that, The step of optimizing the encoder of the student model based on the first distillation loss, the second distillation loss, and the third distillation loss includes: Obtain the model loss of the student model; Determine the root mean square error estimates of the first distillation loss, the second distillation loss, and the third distillation loss; Multiple auxiliary learning tasks are constructed based on the model loss and the mean squared error estimate; The encoder of the student model is optimized based on the multiple auxiliary learning tasks.

9. A distillation apparatus, characterized in that, The distillation apparatus includes: a memory, a processor, and a distillation program stored in the memory and executable on the processor, wherein the distillation program, when executed by the processor, implements the steps of the distillation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a distillation program, which, when executed by a processor, implements the steps of the distillation method as described in any one of claims 1 to 8.

Images