Student model training method and device based on gradient decoupling

Abstract

The invention provides a student model training method and device based on gradient decoupling, and the method comprises the steps: carrying out the prediction of a training sample based on a teacher model and a target model, calculating the difference loss, and enabling the target model to comprise a first branch network and a second branch network which are determined based on the network structure of an initial student model, a knowledge interaction module is arranged between the first branch network and the second branch network, and the difference loss comprises task loss and distillation loss; adjusting network parameters of the first branch network based on the gradient generated by the task loss; adjusting network parameters of the second branch network based on the gradient generated by the distillation loss; and determining a student model for reasoning based on the first branch network and the second branch network after parameter adjustment. According to the method, the mutual influence between gradients generated by task loss and distillation loss in the knowledge distillation process can be eliminated, so that the student model can learn more knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a student model training method and device based on gradient decoupling. BACKGROUND

[0002] In the past decade, artificial intelligence has made significant progress in various application fields, and the performance of trained models has become increasingly strong. However, the size of the model has also been growing, from the initial AlexNet-61M parameter amount to the current GPT3-175B parameter amount, which has brought serious challenges to the deployment and reasoning of the model on edge devices with limited computing and memory resources. In order to solve the above problems, model compression technology can be used to reduce the model parameter amount and speed up the reasoning speed while maintaining a certain accuracy. At present, model compression mainly includes knowledge distillation, pruning and quantization methods. Among them, knowledge distillation is a method of transferring knowledge from a large model (teacher model) to a lightweight model (student model). This technology mainly minimizes the difference between the soft predictions or intermediate features of the teacher model and the student model to transfer knowledge, thereby effectively improving the performance of the student model without introducing additional costs.

[0003] Knowledge distillation was first proposed by Hinton et al., which promotes knowledge transfer by minimizing the KL divergence (Kullback-Leibler Divergence) between the soft logical outputs of the teacher network (also known as the teacher model) and the student network (also known as the student model). After that, FitNet extended knowledge distillation by transferring deep features from the intermediate layers of the teacher network to the student network. Current knowledge distillation methods can generally include methods based on logical output, methods based on features, and methods based on relationships, etc. For example, methods based on logical output can include DKD (Decoupled Knowledge Distillation), MLKD (Multi-Layer Knowledge Distillation), etc., methods based on features can include FitNet (Hints for Thin Deep Nets), AT (Attention Transfer), Reviewkd (Reviewing Knowledge Distillation), etc., and methods based on relationships can include Relationalkd (Relational Knowledge Distillation), etc.

[0004] The loss function used in current knowledge distillation methods usually contains two parts: task loss and distillation loss wherein, may represent a task loss, may represent a distillation loss, by introducing a hyper-parameter to balance the size of the gradients generated by the two losses. However, experimental observations show that the gradients generated by the two losses not only differ in size, but also differ in direction. When the two gradients are directly added together, their difference in direction will result in a trade-off. The gradient can be regarded as a carrier for transmitting knowledge to the student model, and the mutual impairment of the information of the two gradients will hinder the student model from learning all the knowledge, thereby affecting its better convergence. Therefore, a better student model training method is needed to enable the student model to learn more knowledge. SUMMARY

[0005] In view of this, the embodiments of the present application provide a student model training method and device based on gradient decoupling, which can eliminate the mutual influence between the gradients generated by the task loss and the distillation loss in the knowledge distillation process, so that the student model can learn more knowledge and be more accurate.

[0006] According to a first aspect, a student model training method based on gradient decoupling is provided, comprising: calculating a difference loss based on the predictions of a teacher model and an above-mentioned target model on a training sample, wherein the target model comprises a first branch network and a second branch network determined based on the network structure of an initial student model, a knowledge interaction module is arranged between the first branch network and the second branch network for interacting with the output of a preset layer between the first branch network and the second branch network, and the difference loss comprises a task loss and a distillation loss; adjusting the network parameters of the first branch network based on the gradient generated by the task loss; adjusting the network parameters of the second branch network based on the gradient generated by the distillation loss; and determining an inference student model based on the first branch network and the second branch network after parameter adjustment.

[0007] According to a second aspect, a student model training apparatus based on gradient decoupling is provided, comprising: a computing unit configured to calculate a difference loss based on predictions of a teacher model and a target model on a training sample respectively, wherein the target model comprises a first branch network and a second branch network determined based on a network structure of an initial student model, a knowledge interaction module is arranged between the first branch network and the second branch network, and is used to interactively process outputs of a preset layer between the first branch network and the second branch network, and the difference loss comprises a task loss and a distillation loss; a first adjusting unit configured to adjust network parameters of the first branch network based on a gradient generated by the task loss; a second adjusting unit configured to adjust network parameters of the second branch network based on a gradient generated by the distillation loss; and a determining unit configured to determine an inference student model based on the first branch network and the second branch network after parameter adjustment.

[0008] According to a third aspect, a computer readable storage medium having stored thereon computer programs / instructions is provided, wherein the computer programs / instructions, when executed by a processor, implement the steps of the method according to any one of the first aspect.

[0009] According to a fourth aspect, another student model training apparatus based on gradient decoupling is provided, comprising a processor, a memory, and computer programs / instructions stored on the memory, wherein the processor is configured to execute the computer programs / instructions, and when the computer programs / instructions are executed, the apparatus implements the steps of the method according to any one of the first aspect.

[0010] The student model training method and apparatus based on gradient decoupling can train a target model, which can comprise a first branch network and a second branch network determined based on a network structure of an initial student model, and a knowledge interaction module arranged between the first branch network and the second branch network. Specifically, first, a difference loss can be calculated based on predictions of a teacher model and a target model on a training sample respectively, wherein the difference loss can comprise a task loss and a distillation loss. Then, network parameters of the first branch network can be adjusted based on a gradient generated by the task loss, and network parameters of the second branch network can be adjusted based on a gradient generated by the distillation loss. Finally, an inference student model can be determined based on the first branch network and the second branch network after parameter adjustment. Thus, the gradients generated by the task loss and the distillation loss can be decoupled during model training, and used to adjust network parameters of different branch networks respectively, and the inference student model can be determined based on the first branch network and the second branch network after parameter adjustment, so that the mutual influence between the gradients generated by the task loss and the distillation loss during knowledge distillation can be eliminated, and the inference student model can obtain more knowledge, thereby being more accurate.

[0011] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following detailed description and drawings in which

[0012] Those skilled in the art will appreciate that the objects and advantages of the application can be implemented in a variety of ways, and that the application can be implemented using other structures and methods than those specifically set forth in the description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description, serve to explain the principles of the application.

[0014] Figure 1 A flowchart of a student model training method based on gradient decoupling is shown according to an embodiment; Figure 2 A structural diagram showing the expansion from an initial student model to a target model is shown; Figure 3 A diagram showing the merging of the double-branch structure and knowledge interaction module of the target model into a single-branch linear learnable layer is shown; Figure 4 A schematic block diagram of a student model training device based on gradient decoupling is shown according to an embodiment. DETAILED DESCRIPTION

[0015] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the embodiments and drawings. Herein, the schematic embodiments of the present application and their descriptions are used to explain the present application, but not as a limitation to the present application.

[0016] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0017] It should be emphasized that the terms "comprises / comprising" when used in this specification, specify the presence of stated features, elements, steps or components, but do not preclude the presence or addition of one or more other features, elements, steps or components.

[0018] It should be noted that, if not specifically stated, the term "connected" herein can not only mean direct connection, but also mean indirect connection with an intermediate.

[0019] Embodiments of the present application will be described hereinafter with reference to the accompanying drawings. In the drawings, like reference numerals designate identical or similar parts, or identical or similar steps.

[0020] As mentioned before, knowledge distillation is a model compression method. In some existing knowledge distillation, the network parameters of the student model can be updated by the following formula: , wherein, may represent the parameters in the student network; may represent the learning rate; and may represent the task loss and the distillation loss, respectively; may represent the hyperparameter used to balance the two losses. Here, only balances the size of the two gradients, but cannot solve the conflict in their directions. Therefore, an effective method is needed to eliminate the mutual interference between the two gradients and retain their respective information. That is, a better student model training method is urgently needed to enable the student model to learn more knowledge.

[0021] Inspired by the view of neuroscience research that "different regions of the human brain are responsible for learning different types of knowledge", the embodiments of the present specification provide a student model training method based on gradient decoupling, aiming to enable the neural network to learn different types of knowledge with different parameters like the brain, which can eliminate the mutual influence between the gradients generated by the task loss and the distillation loss in the knowledge distillation process, so that the student model can learn more knowledge and be more accurate.

[0022] Please refer to Figure 1 , Figure 1 shows a flowchart of a student model training method based on gradient decoupling according to one embodiment. It can be understood that this method can be executed by any device, equipment, platform, device cluster with computing and processing capabilities. As Figure 1 shown, the student model training method based on gradient decoupling can include the following steps 101-104, specifically: Step 101, based on the predictions of the training samples by the teacher model and the target model respectively, calculate the difference loss.

[0023] In this embodiment, inspired by neuroscience research, different gradient updates can be used to update two different sets of model parameters in the student model to achieve gradient decoupling. Specifically, the initial student model can be expanded from a single-branch network to a double-branch network, which can be referred to as a target model, referred to as a first branch network (or a CE branch network) and a second branch network (or a KD branch network) respectively. During the training phase of the model, the target model can be trained, which can include a first branch network and a second branch network determined based on the network structure of the initial student model. For example, the network structure of the first branch network and the second branch network can be the same as or similar to (e.g., some layers are the same) the network structure of the initial student model. In the case where the network structure of the first branch network and the second branch network is the same as the network structure of the initial student model, the number of parameters updated by each gradient is the same as the number of parameters in the initial student model, thereby ensuring that it can learn two types of knowledge without reducing the ability of the student model.

[0024] In practice, knowledge distillation can be regarded as a form of multi-task learning. In multi-task learning, the interaction between losses enables the model to learn shared representations, thereby improving its representation ability. The decoupling method of "all or nothing" completely excludes the possibility of such interaction. Meanwhile, the creator of the present application found through a large number of experiments that randomly setting some layers of the student model to retain a single-branch structure (without gradient decoupling) and other layers to adopt a double-branch structure (with gradient decoupling) can improve the performance of the student model in some cases. This experiment shows that maintaining the interaction between the double branches in some layers of the student model helps to improve the accuracy. Therefore, a knowledge interaction module (KIM) can be set between the first branch network and the second branch network, which can be set in some layers as needed. The knowledge interaction module can model the interaction pattern between the first branch network and the second branch network in the current layer, thereby achieving dynamic and flexible decoupling. Specifically, the knowledge interaction module can be used to interactively process the outputs of the preset layers between the first branch network and the second branch network, for example, the knowledge interaction module can include some parameters, which can be used to calculate the outputs of the preset layers between the first branch network and the second branch network.

[0025] On this basis, the difference loss can be calculated based on the predictions of the training samples by the teacher model and the target model, where the difference loss can include a task loss and a distillation loss. Here, the parameter quantity of the teacher model is greater than that of the initial student model.

[0026] In some implementations, a linear learnable layer (e.g., a convolutional layer, a fully connected layer, etc.) and a normalization layer (e.g., a batch normalization layer) can be used as a basic unit for the dual-branch expansion. For example, assuming that the first branch network includes a first linear learnable layer and a first normalization layer, and the second branch network includes a second linear learnable layer and a second normalization layer, a knowledge interaction module can be arranged between the linear learnable layer and the normalization layer, and the knowledge interaction module is configured to interactively process the outputs of the first linear learnable layer and the second linear learnable layer.

[0027] As shown in Figure 2 , Figure 2 a structural diagram for expanding from an initial student model to a target model is shown. In the example shown in Figure 2 , (a) can represent the network structure of the initial student model, which is a single-branch model, and the initial student model can update all parameters through a task loss and a distillation loss . (b) can represent the network structure of the target model in the training stage, and the target model can include a first branch network Figure 2 (the left branch of the target model in Figure 2 ), a second branch network Figure 2 (the right branch of the target model in Figure 2 ), and a knowledge interaction module. In the training stage, the task gradient generated by the task loss and the distillation gradient generated by the distillation loss can be decoupled, and the parameters of each branch are updated by one kind of gradient. Specifically, the gradient generated by the task loss can be used to update the network parameters of the first branch network, and the gradient generated by the distillation loss can be used to update the network parameters of the second branch network. For ease of understanding, in the example shown in Figure 2 , the internal structure of the knowledge interaction module is also shown. For ease of understanding, in the example shown in Figure 2 , a solid arrow is used to represent forward propagation, and a dashed arrow is used to represent backward propagation. It can be understood that Figure 2 , in addition to the linear learnable layers, normalization layers, knowledge interaction modules, etc. shown, the original student model and the target model can also include more other network layers. For example, the original student model can also include multiple basic units composed of linear learnable layers and normalization layers, which can be used for dual-branch expansion. The expansion manner of these basic units can refer to the expansion manner shown in Figure 2 , which will not be described here.

[0028] based on Figure 2 The network structure of the target model shown, and the prediction of training samples based on the target model, can include the following steps 1) and 2), specifically: Step 1) The knowledge interaction module interacts with the outputs of the first linearly learnable layer and the second linearly learnable layer to obtain the processing result, and then forwards the processing result to the first normalization layer and the second normalization layer.

[0029] Step 2) The outputs of the first normalization layer and the second normalization layer are fused (for example, summed and then processed by an activation function) and then forwarded to the next layer of each branch network until the prediction layer of the target model outputs the prediction result.

[0030] In some implementations, the knowledge interaction module may include a parameter matrix. Based on this, step 1) above, where the knowledge interaction module interacts with the outputs of the first and second linearly learnable layers to obtain the processing result, may include: using the parameter matrix to linearly combine the activation values ​​of the first and second linearly learnable layers to obtain updated activation values.

[0031] For example, suppose the knowledge interaction module uses... express, The activation values ​​generated by the first and second linearly learnable layers can be respectively... Perform linear combinations to generate new activation values ​​respectively. Multiple linear combinations can effectively extract the shared representation of these two types of knowledge. Specific details are as follows: , , in, The formula for calculating it is as follows: , , in, These can represent learnable parameters of the Knowledge Interaction Module (KIM); It can represent a matrix Apply the softmax activation function along the first dimension; It can represent the identity matrix; It can be a hyperparameter threshold. In deep learning and machine learning, learnable parameters refer to parameters that the model can automatically adjust during training through backpropagation and optimization algorithms (such as stochastic gradient descent).

[0032] By the above formula, we ensure The following three objectives can be met: (1), The sum of each row of the matrix is 1, thereby ensuring that the size of the new activation value is similar to the original activation value in the above formula.

[0033] (2), In the formula, and are limited to the range of 0.5 to 1, which ensures that the new activation value of each branch is mainly dominated by its original activation value.

[0034] (3), Values less than are ignored to reduce the computational overhead in the training process. In practice, the size of can be set according to actual needs. For example, can be set to 0.05.

[0035] Next, the significance of different values of is explained, which can reflect the degree of interaction between the first branch network and the second branch network. Specifically, when all values of are equal to 0.5, it can indicate that contributes equally in each new activation value, and the interaction between the two branches reaches the highest level. When changes from 0.5 to 1, and drops from 0.5 to 0, it can indicate that the degree of interaction between the first branch network and the second branch network is decreasing. When is equal to 1, and is equal to 0, it can indicate that the two branches are independent of each other and there is no interaction.

[0036] Step 102, based on the gradient generated by the task loss, adjusting the network parameters of the first branch network.

[0037] Step 103, based on the gradient generated by the distillation loss, adjusting the network parameters of the second branch network.

[0038] In this embodiment, the network parameters of the first branch network and the second branch network can be updated by the following formula: wherein, and can represent the network parameters of the first branch network and the second branch network respectively, the parameters of the first linear learnable layer and the second linear learnable layer are , and the parameters of the first normalization layer and the second normalization layer are During the training phase, and Through respectively and The generated gradients are then updated. This training method eliminates the mutual influence between the two gradients and preserves the knowledge contained in each gradient.

[0039] In some implementations, the knowledge interaction module may also include learnable parameters. Since the knowledge interaction module involves both the first and second branch networks, the learnable parameters of the knowledge interaction module can be updated by combining the task loss and distillation loss. Specifically, the learnable parameters of the knowledge interaction module can be adjusted with the goal of minimizing the difference loss (including both task loss and distillation loss).

[0040] Step 104: Based on the first and second branch networks after parameter adjustment, determine the student model for inference.

[0041] In this embodiment, after adjusting (or updating) the network parameters of the first and second branch networks, a student model for inference can be determined based on the adjusted parameters of the first and second branch networks. This student model for inference can process the input and output a prediction result. As an example, a target model containing the first and second branch networks can be used as the student model for inference.

[0042] In practice, while gradient decoupling in a dual-branch target model improves performance, it inevitably increases inference overhead. To address this, the dual-branch structure and knowledge interaction module of the target model can be merged into a single-branch linear learnable layer while maintaining consistency in forward propagation results.

[0043] In some implementations, the network structures of the first branch network and the second branch network can be the same. Based on this, step 104 above can include the following (i) and (ii), specifically: (i) The first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module are merged into a single linearly learnable layer in the student model for reasoning. The parameters of this linearly learnable layer in the student model for reasoning can be determined based on the parameters of the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module.

[0044] like Figure 3 As shown, Figure 3 This diagram illustrates the merging of the target model's two-branch structure and knowledge interaction module into a single-branch linear learnable layer. Figure 3In the illustrated example, (c) can represent the model structure of the inference student model. Figure 3 In the illustrated example, the first linear learnable layer, the first normalization layer in the first branch network, the second linear learnable layer, the second normalization layer in the second branch network, and the knowledge interaction module can be merged into one linear learnable layer, thereby converting the double-branch structure in the target model into a single-branch structure, so that the inference student model can have a simplified single-branch linear learnable layer.

[0045] The specific way of merging is explained in detail below. In the double-branch structure composed of the first linear learnable layer, the first normalization layer, the second linear learnable layer, the second normalization layer, and the knowledge interaction module, the forward propagation calculation can be as follows: , , wherein, can represent the input; can represent the output of the double branch; can represent the parameters of the normalization layer.

[0046] First, the knowledge interaction module can be fused into the linear learnable layer of the double branch.

[0047] Specifically, the equivalent linear learnable layer and are used to replace and .

[0048] Second, since the normalization layer is frozen in the inference stage, its parameters are fixed, so the normalization layer can be fused into the linear learnable layer, and the double branch can be merged into a single branch. Specifically: wherein, can represent the weight of the equivalent single-branch linear learnable layer; can represent the bias. This simplification effectively reduces the number of floating point operations (FLOPs) required by the inference student model in the inference stage.

[0049] (II) The other layers in the first branch network and the second branch network are correspondingly merged to generate the inference student model.

[0050] Here, the other layers in the first branch network and the second branch network can refer to the layers in the first branch network and the second branch network other than the first linear learnable layer, the first normalization layer, the second linear learnable layer, and the second normalization layer. The other layers in the first branch network and the second branch network can be merged. For example, assuming that the first branch network can include a linear layer C1, which can correspond to a linear layer C2 in the second branch network, the linear layer C1 and the linear layer C2 can be fused into one linear layer, and the parameters of the linear layer can be determined based on the parameters of the linear layer C1 and the linear layer C2.

[0051] In some implementations, the student model training method based on gradient decoupling of the present embodiment can be applied to the training of various machine learning models, for example, can be applied to the training of an image classification model, a speech recognition model, a text translation model, etc. Thus, the inference student model can include at least one of the following models: an image classification model, a speech recognition model, a text translation model, etc.

[0052] For example, in the case of training an image classification model, the teacher model can be a model for performing image classification, and the training sample can include a sample image and a sample classification result corresponding to the sample image. The teacher model and the target model can respectively predict the sample image, and based on the predictions of the teacher model and the target model, the difference loss (including the task loss and the distillation loss) is calculated. For example, the task loss is calculated based on the prediction result of the target model and the sample classification result, and the distillation loss is calculated based on the prediction of the teacher model and the prediction of the target model. Then, the network parameters of the first branch network can be adjusted based on the gradient generated by the task loss, and the network parameters of the second branch network can be adjusted based on the gradient generated by the distillation loss. Finally, based on the first branch network and the second branch network after parameter adjustment, the inference student model is determined, which can be an image classification model.

[0053] For another example, in the case of training a speech recognition model, the teacher model can be a model for performing speech recognition, and the training sample can include a sample speech and a sample speech recognition result corresponding to the sample speech. Based on the predictions of the teacher model and the target model on the sample speech, the difference loss (including the task loss and the distillation loss) is calculated. For example, the task loss is calculated based on the prediction result of the target model and the sample speech recognition result, and the distillation loss is calculated based on the prediction of the teacher model and the prediction of the target model. Then, the network parameters of different branch networks are adjusted using the gradients generated by the two losses, respectively, and finally the inference student model is determined, which can be a speech recognition model.

[0054] For another example, in the case of training a text translation model, the teacher model can be a model for text translation, and the training sample can include a sample text to be translated and a sample translation result corresponding to the sample text to be translated. Based on the predictions of the sample text to be translated by the teacher model and the target model respectively, the difference loss (including the task loss and the distillation loss) is calculated. For example, the task loss is calculated based on the prediction result of the target model and the sample translation result, and the distillation loss is calculated based on the prediction of the teacher model and the prediction of the target model. Then, the gradients generated by the two losses are used to adjust the network parameters of the different branch networks respectively, and finally determine the inference student model, which can be a text translation model.

[0055] In review of the above process, in the embodiments of the present specification, the target model can be trained, which can include a first branch network and a second branch network determined based on the network structure of the initial student model, and a knowledge interaction module is arranged between the first branch network and the second branch network. Specifically, first, based on the predictions of the training sample by the teacher model and the target model respectively, the difference loss can be calculated, wherein the difference loss can include the task loss and the distillation loss. Then, based on the gradient generated by the task loss, the network parameters of the first branch network are adjusted, and based on the gradient generated by the distillation loss, the network parameters of the second branch network are adjusted. Then, based on the first branch network and the second branch network after parameter adjustment, the inference student model is determined. Thus, the gradients generated by the task loss and the distillation loss can be decoupled in the model training process, respectively used to adjust the network parameters of different branch networks, and the inference student model is determined based on the first branch network and the second branch network after parameter adjustment, so that the mutual influence between the gradients generated by the task loss and the distillation loss in the knowledge distillation process can be eliminated, and the inference student model can obtain more knowledge, thereby being more accurate.

[0056] According to another aspect, embodiments provide a gradient-decoupled student model training apparatus. The above-mentioned gradient-decoupled student model training apparatus can be deployed in any device, equipment, platform, device cluster with computing and processing capabilities.

[0057] Figure 4 A schematic block diagram of a gradient-decoupled student model training apparatus according to an embodiment is shown. As shown in the figure, the gradient-decoupled student model training apparatus can include a teacher model, a target model, a knowledge interaction module, a task loss calculation module, a distillation loss calculation module, a first branch network parameter adjustment module, a second branch network parameter adjustment module, and an inference student model determination module. Figure 4As shown, the student model training apparatus 400 based on gradient decoupling can include a calculation unit 401 configured to calculate a difference loss based on predictions of the training sample by the teacher model and the target model, wherein the target model includes a first branch network and a second branch network determined based on the network structure of the initial student model, and a knowledge interaction module is arranged between the first branch network and the second branch network to interact the output of a preset layer between the first branch network and the second branch network, and the difference loss includes a task loss and a distillation loss; a first adjustment unit 402 configured to adjust the network parameters of the first branch network based on the gradient generated by the task loss; a second adjustment unit 403 configured to adjust the network parameters of the second branch network based on the gradient generated by the distillation loss; and a determination unit 404 configured to determine an inference student model based on the first branch network and the second branch network after parameter adjustment.

[0058] In some implementations, the first branch network includes a first linear learnable layer and a first normalization layer, the second branch network includes a second linear learnable layer and a second normalization layer, and the knowledge interaction module is configured to interact the output of the first linear learnable layer and the output of the second linear learnable layer; and the calculation unit 401 is further configured to interact the output of the first linear learnable layer and the output of the second linear learnable layer by the knowledge interaction module to obtain a processing result, forward propagate the processing result to the first normalization layer and the second normalization layer, fuse the output of the first normalization layer and the output of the second normalization layer, and forward transmit the fused output to the next layer of each branch network until the prediction layer of the target model outputs a prediction result.

[0059] In some implementations, the network structure of the first branch network and the second branch network is the same; and the determination unit 404 is further configured to combine the first linear learnable layer, the first normalization layer, the second linear learnable layer, the second normalization layer, and the knowledge interaction module into one linear learnable layer in the inference student model, wherein the parameters of the linear learnable layer are determined based on the parameters of the first linear learnable layer, the first normalization layer, the second linear learnable layer, the second normalization layer, and the knowledge interaction module; and other layers in the first branch network and the second branch network are correspondingly combined to generate the inference student model, wherein the inference student model has a single-branch structure.

[0060] In some implementations, the knowledge interaction module includes a parameter matrix; and the interaction processing of the output of the first linear learnable layer and the output of the second linear learnable layer by the knowledge interaction module to obtain a processing result includes: performing linear combination on the activation value of the output of the first linear learnable layer and the activation value of the output of the second linear learnable layer by using the parameter matrix to obtain updated activation values.

[0061] In some implementations, the knowledge interaction module includes learnable parameters, and the apparatus further includes a third adjustment unit (not shown in the figure) configured to adjust the learnable parameters of the knowledge interaction module with the objective of minimizing the difference loss.

[0062] In some implementations, the network structure of the first branch network and the second branch network is the same as the network structure of the initial student model.

[0063] In some implementations, the inference student model includes at least one of the following models: an image classification model, a speech recognition model, and a text translation model.

[0064] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the aforementioned student model training method based on gradient decoupling. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0065] The embodiment of the present application further provides a computing device, which includes a processor, a memory, and a computer program / instruction stored on the memory, and the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the computing device implements the steps of the aforementioned student model training method based on gradient decoupling. Figure 1 The steps of the described method.

[0066] Those of ordinary skill in the art will appreciate that the various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or both. The particular implementation is dependent on the specific application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, for example, the hardware can comprise an electronic circuit, an Application Specific Integrated Circuit (ASIC), a suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the application are the program or code segments to perform a specific task. The program or code segments can be stored in a machine-readable medium, or transmitted by a carrier wave as data signals over a transmission medium or communication link.

[0067] It is to be understood that the application is not limited to the particular configurations and processes described herein and shown in the drawings. For simplicity, detailed descriptions of known methods and apparatuses are omitted so as not to obscure the disclosure. In the above-described embodiments, several specific steps are described and illustrated as examples. However, the method processes of the present application are not limited to the specific steps described and illustrated, and the order of the steps can be changed, or other steps can be added, or replaced, or eliminated, depending on the application.

[0068] In the present application, features described and / or illustrated with respect to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0069] The above description is only preferred embodiments of the present application, and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A student model training method based on gradient decoupling, comprising: calculating a difference loss based on predictions made by a teacher model and a target model on a training sample, wherein the target model comprises a first branch network and a second branch network determined based on a network structure of an initial student model, a knowledge interaction module is arranged between the first branch network and the second branch network, and the knowledge interaction module is configured to interactively process outputs of a preset layer between the first branch network and the second branch network, and the difference loss comprises a task loss and a distillation loss; adjusting network parameters of the first branch network based on gradients generated by the task loss; adjusting network parameters of the second branch network based on gradients generated by the distillation loss; determining an inference student model based on the first branch network and the second branch network after parameter adjustment.

2. The method of claim 1, wherein, The first branch network comprises a first linear learnable layer and a first normalization layer, the second branch network comprises a second linear learnable layer and a second normalization layer, and the knowledge interaction module is configured to interactively process outputs of the first linear learnable layer and the second linear learnable layer. And predicting the training sample based on the target model comprises: interactively processing, by the knowledge interaction module, the outputs of the first linear learnable layer and the outputs of the second linear learnable layer to obtain a processing result, and forward propagating the processing result to the first normalization layer and the second normalization layer; fusing the outputs of the first normalization layer and the outputs of the second normalization layer, and then forward transmitting the fused outputs to a next layer of each branch network until a prediction layer of the target model outputs a prediction result.

3. The method of claim 2, wherein, The network structures of the first branch network and the second branch network are the same, and determining the inference student model based on the first branch network and the second branch network after parameter adjustment comprises: merging the first linear learnable layer, the first normalization layer, the second linear learnable layer, the second normalization layer and the knowledge interaction module into one linear learnable layer in the inference student model, wherein parameters of the linear learnable layer are determined based on parameters of the first linear learnable layer, the first normalization layer, the second linear learnable layer, the second normalization layer and the knowledge interaction module; correspondingly merging other layers in the first branch network and the second branch network to generate the inference student model, wherein the inference student model has a single-branch structure.

4. The method of claim 2, wherein, The knowledge interaction module comprises a parameter matrix, and interactively processing, by the knowledge interaction module, the outputs of the first linear learnable layer and the outputs of the second linear learnable layer to obtain a processing result comprises: linearly combining, by the parameter matrix, activation values of the outputs of the first linear learnable layer and activation values of the outputs of the second linear learnable layer to obtain updated activation values.

5. The method of claim 1, wherein, The knowledge interaction module comprises learnable parameters, and the method further comprises: adjusting the learnable parameters of the knowledge interaction module to minimize the difference loss.

6. The method of claim 1, wherein, The network structures of the first branch network and the second branch network are the same as that of the initial student model.

7. The method of claim 1, wherein, The inference student model comprises at least one of the following models: an image classification model, a speech recognition model, and a text translation model.

8. The method of claim 4, wherein, The parameter matrix is: ， The calculation formula is as follows: ， ， wherein, denote learnable parameters of the knowledge interaction module; denote a matrix apply a softmax activation function along the first dimension; denote an identity matrix; denote a hyperparameter threshold.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the steps of the method of any one of claims 1-8. 10.A student model training apparatus based on gradient decoupling, comprising a processor, a memory and a computer program / instruction stored on the memory, wherein the computer program / instruction comprises the following steps of: The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the apparatus implements the steps of the method of any one of claims 1-8.

Images