Student model training method and device based on gradient decoupling

Abstract

The application provides a student model training method and device based on gradient decoupling, which comprises the following steps: calculating a difference loss based on the prediction of a training sample by a teacher model and a target model, wherein the target model comprises a first branch network and a second branch network determined based on the network structure of an initial student model, a knowledge interaction module is arranged between the first branch network and the second branch network, and the difference loss comprises a task loss and a distillation loss; adjusting the network parameters of the first branch network based on the gradient generated by the task loss; adjusting the network parameters of the second branch network based on the gradient generated by the distillation loss; and determining an inference student model based on the first branch network and the second branch network after parameter adjustment. The application can eliminate the mutual influence between the gradients generated by the task loss and the distillation loss in the knowledge distillation process, so that the student model can learn more knowledge.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a student model training method and apparatus based on gradient decoupling. Background Technology

[0002] Over the past decade, artificial intelligence has made significant progress in various application areas, and the performance of trained models has become increasingly powerful. However, the size of these models has also been growing continuously, from the parameter count of the early AlexNet-61M to that of today's GPT3-175B. This poses a significant challenge to deploying and inferring these models on edge devices with limited computing and memory resources. To address this issue, model compression techniques can be used to reduce the number of model parameters and accelerate inference speed while maintaining a certain level of accuracy. Currently, model compression mainly includes methods such as knowledge distillation, pruning, and quantization. Among these, knowledge distillation is a method for transferring knowledge from a large model (teacher model) to a lightweight model (student model). This technique mainly achieves knowledge transfer by minimizing the differences in soft predictions or intermediate features between the teacher and student models, thereby effectively improving the performance of the student model without introducing additional costs.

[0003] Knowledge distillation, initially proposed by Hinton et al., facilitates knowledge transfer by minimizing the Kullback-Leibler divergence between the soft logic outputs of the teacher network (also known as the teacher model) and the student network (also known as the student model). FitNet later extended knowledge distillation by transferring deep features from the intermediate layers of the teacher network to the student network. Current knowledge distillation methods can broadly categorize into logic output-based, feature-based, and relation-based methods. For example, logic output-based methods include DKD (Decoupled Knowledge Distillation) and MLKD (Multi-Layer Knowledge Distillation); feature-based methods include FitNet (Hints for Thin Deep Nets), AT (Attention Transfer), and ReviewKD (Reviewing Knowledge Distillation); and relation-based methods include RelationalkD (Relational Knowledge Distillation).

[0004] Current knowledge distillation methods typically use a loss function that consists of two parts: task loss and distillation loss. ,in, It can represent a loss of mission objectives. Distillation loss can be represented by introducing a hyperparameter. The goal is to balance the magnitudes of the gradients produced by these two loss methods. However, experimental observations show that the gradients produced by these two losses differ not only in magnitude but also in direction. When these two gradients are directly added, their difference in direction leads to a trade-off. Gradients can be seen as the carriers of knowledge transferred to the student model, and the mutual impairment of the two gradient information can hinder the student model from learning all the knowledge, thus affecting its ability to achieve better convergence. Therefore, a better student model training method is urgently needed to enable the student model to learn more knowledge. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a student model training method and apparatus based on gradient decoupling, which can eliminate the mutual influence between gradients generated by task loss and distillation loss during knowledge distillation, enabling the student model to learn more knowledge and thus become more accurate.

[0006] According to the first aspect, a gradient-decoupled student model training method is provided, comprising: calculating a difference loss based on predictions made by the teacher model and the target model on training samples respectively, wherein the target model includes a first branch network and a second branch network determined based on the network structure of the initial student model, and a knowledge interaction module is set between the first branch network and the second branch network for interactive processing of the outputs of a preset layer between the first branch network and the second branch network, and the difference loss includes a task loss and a distillation loss; adjusting the network parameters of the first branch network based on the gradient generated by the task loss; adjusting the network parameters of the second branch network based on the gradient generated by the distillation loss; and determining the student model for inference based on the first branch network and the second branch network after parameter adjustment.

[0007] According to the second aspect, a gradient-decoupled student model training device is provided, comprising: a computation unit configured to calculate a difference loss based on predictions made by a teacher model and a target model on training samples, wherein the target model includes a first branch network and a second branch network determined based on the network structure of an initial student model, and a knowledge interaction module is provided between the first branch network and the second branch network for interactive processing of the outputs of a preset layer between the first branch network and the second branch network, and the difference loss includes a task loss and a distillation loss; a first adjustment unit configured to adjust the network parameters of the first branch network based on the gradient generated by the task loss; a second adjustment unit configured to adjust the network parameters of the second branch network based on the gradient generated by the distillation loss; and a determination unit configured to determine a student model for inference based on the first branch network and the second branch network after parameter adjustment.

[0008] According to a third aspect, a computer-readable storage medium is provided, on which a computer program / instructions are stored, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of any of the methods described in the first aspect.

[0009] According to the fourth aspect, another student model training apparatus based on gradient decoupling is provided, including a processor, a memory, and a computer program / instructions stored in the memory, characterized in that the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the apparatus implements the steps of any of the methods described in the first aspect.

[0010] The present invention discloses a gradient-decoupled student model training method and apparatus for training a target model. The target model can include a first branch network and a second branch network determined based on the network structure of an initial student model. A knowledge interaction module is provided between the first and second branch networks. Specifically, firstly, a difference loss is calculated based on predictions made by the teacher model and the target model on training samples, where the difference loss can include task loss and distillation loss. Then, the network parameters of the first branch network are adjusted based on the gradient generated by the task loss, and the network parameters of the second branch network are adjusted based on the gradient generated by the distillation loss. Finally, the student model for inference is determined based on the parameter-adjusted first and second branch networks. Thus, the gradients generated by the task loss and distillation loss are decoupled during model training and used to adjust the network parameters of different branch networks respectively. The student model for inference is then determined based on the parameter-adjusted first and second branch networks, thereby eliminating the mutual influence between the gradients generated by the task loss and distillation loss during knowledge distillation. This allows the student model for inference to acquire more knowledge, resulting in greater accuracy.

[0011] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0012] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0013] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0014] Figure 1 A flowchart illustrating a gradient-decoupling-based student model training method according to one embodiment is shown. Figure 2 This diagram illustrates the structure of extending from the initial student model to the target model. Figure 3 A schematic diagram is shown showing the merging of the target model's two-branch structure and knowledge interaction module into a single-branch linear learnable layer; Figure 4 A schematic block diagram of a gradient-decoupled student model training apparatus according to one embodiment is shown. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0016] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0017] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0018] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0019] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0020] As mentioned earlier, knowledge distillation is a model compression method. In some existing knowledge distillation methods, the network parameters of the student model can be updated using the following formula: , in, It can represent parameters in a student network; It can represent the learning rate; and These can represent task loss and distillation loss, respectively; This can be represented by hyperparameters used to balance these two losses. Here, While it can balance the magnitudes of the two gradients, it cannot resolve their conflicting directions. Therefore, an effective method is needed to eliminate the mutual interference between the two gradients while preserving their respective information. In other words, there is an urgent need for a better student model training method to enable the student model to learn more knowledge.

[0021] Inspired by the neuroscience research that "different regions of the human brain are responsible for learning different types of knowledge," this specification provides a gradient-decoupled student model training method. This method aims to enable neural networks to learn different types of knowledge with different parameters, just like the brain. It can eliminate the mutual influence between gradients caused by task loss and distillation loss during knowledge distillation, allowing the student model to learn more knowledge and thus become more accurate.

[0022] Please see Figure 1 , Figure 1 A flowchart illustrating a gradient-decoupling-based student model training method according to one embodiment is shown. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 1 As shown, the student model training method based on gradient decoupling may include the following steps 101-104, specifically: Step 101: Calculate the difference loss based on the predictions made by the teacher model and the target model on the training samples respectively.

[0023] In this embodiment, inspired by neuroscience research, different gradients can be used to update two different sets of model parameters in the student model, achieving gradient decoupling. Specifically, the initial student model can be extended from a single-branch network to a dual-branch network, referred to as the first branch network (or CE branch network) and the second branch network (or KD branch network), which can be called the target model. During the model training phase, the target model can be trained. The target model can include the first and second branch networks determined based on the network structure of the initial student model. For example, the network structures of the first and second branch networks can be the same as or similar to the network structure of the initial student model (e.g., some layers are the same). When the network structures of the first and second branch networks are the same as the network structure of the initial student model, the number of parameters updated by each gradient is the same as the number of parameters in the initial student model, thereby ensuring that the student model can learn two types of knowledge without reducing its capabilities.

[0024] In practice, knowledge distillation can be viewed as a form of multi-task learning. In multi-task learning, the interaction between losses allows the model to learn shared representations, thereby improving its representational ability. An "all or nothing" decoupling approach completely eliminates the possibility of such interaction. Meanwhile, the inventors of this invention, through extensive experimentation, discovered that randomly setting some layers of the student model to retain a single-branch structure (without gradient decoupling), while other layers adopt a dual-branch structure (with gradient decoupling), can improve the performance of the student model in certain situations. This experiment demonstrates that maintaining the interaction between dual branches in some layers of the student model helps improve accuracy. To this end, a knowledge interaction module (KIM) can be set between the first and second branch networks. The KIM can be set in certain layers according to actual needs, and it can model the interaction pattern between the first and second branch networks in the current layer, thereby achieving dynamic and flexible decoupling. Specifically, the KIM can be used to interactively process the outputs of preset layers between the first and second branch networks. For example, the KIM can include parameters that can be used to calculate the outputs of preset layers between the first and second branch networks.

[0025] Based on this, the discrepancy loss can be calculated based on the predictions made by the teacher model and the target model on the training samples, respectively. The discrepancy loss can include task loss and distillation loss. Here, the teacher model has more parameters than the initial student model.

[0026] In some implementations, linearly learnable layers (e.g., convolutional layers, fully connected layers, etc.) and normalization layers (e.g., batch normalization layers) can be used as the basic units for dual-branch expansion. For example, assuming the first branch network may include a first linearly learnable layer and a first normalization layer, and the second branch network may include a second linearly learnable layer and a second normalization layer, a knowledge interaction module can be set between the linearly learnable layer and the normalization layer. This knowledge interaction module is used to interactively process the outputs of the first and second linearly learnable layers.

[0027] like Figure 2 As shown, Figure 2 A schematic diagram illustrating the structure from the initial student model to the target model is shown. Figure 2 In the example shown, (a) can represent the network structure of the initial student model, which is a single-branch model that can be determined by the task loss ( ) and distillation loss ( (a) Update all parameters. (b) Can represent the network structure of the target model during the training phase, and the target model may include the first branch network ( Figure 2 The left branch of the target model), the second branch network ( Figure 2 The right-hand branch of the target model and the knowledge interaction module can decouple the task gradient generated by the task loss and the distillation gradient generated by the distillation loss during the training phase, with the parameters of each branch being updated by one gradient. Specifically, the task loss ( The gradient generated can be used to update the network parameters of the first branch network, distillation loss ( The gradient generated can be used to update the network parameters of the second branch network. For ease of understanding, Figure 2 The example shown also illustrates the internal structure of the knowledge interaction module. For ease of understanding, in Figure 2 In the example shown, solid arrows represent forward propagation, and dashed arrows represent backward propagation. This is understandable. Figure 2 In addition to the linear learnable layers, normalization layers, and knowledge interaction modules shown, the original student model and the target model can also include more network layers. For example, the original student model can also include multiple basic units composed of linear learnable layers and normalization layers, which can be used for bi-branch expansion. The expansion methods of these single units can be referenced. Figure 2 The expansion methods shown will not be elaborated here.

[0028] based on Figure 2 The network structure of the target model shown, and the prediction of training samples based on the target model, can include the following steps 1) and 2), specifically: Step 1) The knowledge interaction module interacts with the outputs of the first linearly learnable layer and the second linearly learnable layer to obtain the processing result, and then forwards the processing result to the first normalization layer and the second normalization layer.

[0029] Step 2) The outputs of the first normalization layer and the second normalization layer are fused (for example, summed and then processed by an activation function) and then forwarded to the next layer of each branch network until the prediction layer of the target model outputs the prediction result.

[0030] In some implementations, the knowledge interaction module may include a parameter matrix. Based on this, step 1) above, where the knowledge interaction module interacts with the outputs of the first and second linearly learnable layers to obtain the processing result, may include: using the parameter matrix to linearly combine the activation values ​​of the first and second linearly learnable layers to obtain updated activation values.

[0031] For example, suppose the knowledge interaction module uses... express, The activation values ​​generated by the first and second linearly learnable layers can be respectively... Perform linear combinations to generate new activation values ​​respectively. Multiple linear combinations can effectively extract the shared representation of these two types of knowledge. Specific details are as follows: , , in, The formula for calculating it is as follows: , , in, These can represent learnable parameters of the Knowledge Interaction Module (KIM); It can represent a matrix Apply the softmax activation function along the first dimension; It can represent the identity matrix; It can be a hyperparameter threshold. In deep learning and machine learning, learnable parameters refer to parameters that the model can automatically adjust during training through backpropagation and optimization algorithms (such as stochastic gradient descent).

[0032] Through the above formula, we ensure It can satisfy the following three objectives: (1), The sum of each row of the matrix is ​​1, thus ensuring that the new activation value in the above formula is close to the original activation value.

[0033] (2), middle, and The value is limited to the range of 0.5 to 1, which ensures that the new activation value of each branch is mainly dominated by its original activation value.

[0034] (3), medium to small The value of is ignored to reduce computational overhead during training. In practice, it can be set according to actual needs. The size. For example, it can be... Set to 0.05.

[0035] Next, we will elaborate. The different values ​​of represent the degree of interaction between the first and second branch networks. Specifically, when When all values ​​are equal to 0.5, it can represent With equal contributions in each new activation value, the interaction between the two branches reaches its highest level. From 0.5 to 1, and A decrease from 0.5 to 0 indicates a reduction in the interaction between the first and second branch networks. It equals 1, and When the value is 0, it can indicate that the two branches are independent of each other and there is no interaction between them.

[0036] Step 102: Adjust the network parameters of the first branch network based on the gradient generated by the task loss.

[0037] Step 103: Adjust the network parameters of the second branch network based on the gradient generated by the distillation loss.

[0038] In this embodiment, the network parameters of the first branch network and the second branch network can be updated using the following formula: in, and The network parameters of the first branch network and the second branch network can be represented respectively. The parameters of the first linearly learnable layer and the second linearly learnable layer are respectively... The parameters of the first normalization layer and the second normalization layer are respectively During the training phase, and Through respectively and The generated gradients are then updated. This training method eliminates the mutual influence between the two gradients and preserves the knowledge contained in each gradient.

[0039] In some implementations, the knowledge interaction module may also include learnable parameters. Since the knowledge interaction module involves both the first and second branch networks, the learnable parameters of the knowledge interaction module can be updated by combining the task loss and distillation loss. Specifically, the learnable parameters of the knowledge interaction module can be adjusted with the goal of minimizing the difference loss (including both task loss and distillation loss).

[0040] Step 104: Based on the first and second branch networks after parameter adjustment, determine the student model for inference.

[0041] In this embodiment, after adjusting (or updating) the network parameters of the first and second branch networks, a student model for inference can be determined based on the adjusted parameters of the first and second branch networks. This student model for inference can process the input and output a prediction result. As an example, a target model containing the first and second branch networks can be used as the student model for inference.

[0042] In practice, while gradient decoupling in a dual-branch target model improves performance, it inevitably increases inference overhead. To address this, the dual-branch structure and knowledge interaction module of the target model can be merged into a single-branch linear learnable layer, while maintaining consistency in forward propagation results.

[0043] In some implementations, the network structures of the first branch network and the second branch network can be the same. Based on this, step 104 above can include the following (i) and (ii), specifically: (i) The first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module are merged into a single linearly learnable layer in the student model for reasoning. The parameters of this linearly learnable layer in the student model for reasoning can be determined based on the parameters of the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module.

[0044] like Figure 3 As shown, Figure 3 This diagram illustrates the merging of the target model's two-branch structure and knowledge interaction module into a single-branch linear learnable layer. Figure 3In the example shown, (c) can represent the model structure of the student model for reasoning. Figure 3 In the example shown, the first linearly learnable layer and the first normalized layer in the first branch network, the second linearly learnable layer and the second normalized layer in the second branch network, and the knowledge interaction module can be merged into a single linearly learnable layer. This transforms the two-branch structure in the target model into a single-branch structure, enabling the student model for reasoning to have a simplified single-branch linearly learnable layer.

[0045] The following explains in detail how the merging is performed. In the two-branch structure consisting of the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module, the forward propagation calculation can be performed as follows: , , in, It can represent input; It can represent the output of two branches; It can represent the parameters of the normalization layer.

[0046] First, the knowledge interaction module can be integrated into the two-branch linear learnable layer.

[0047] Specifically, using an equivalent linearly learnable layer and replace and .

[0048] Secondly, since the normalization layer is frozen during the inference phase and its parameters are fixed, it can be merged into the linearly learnable layer, and two branches can be combined into a single branch. Specifically: in, It can represent the weights of an equivalent single-branch linearly learnable layer; This can represent bias. This simplification effectively reduces the number of floating-point operations (FLOPs) required by the student model during the inference phase.

[0049] (ii) Merge the other layers in the first and second branch networks to generate a student model for inference.

[0050] Here, "other layers" in the first and second branch networks can refer to layers other than the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, and the second normalized layer. These other layers can be merged. For example, assuming the first branch network includes a linear layer C1, which corresponds to a linear layer C2 in the second branch network, then linear layers C1 and C2 can be merged into a single linear layer. The parameters of this linear layer can be determined based on the parameters of linear layers C1 and C2.

[0051] In some implementations, the gradient-decoupling-based student model training method of this embodiment can be applied to the training of various machine learning models, such as image classification models, speech recognition models, and text translation models. Therefore, the student model for inference can include at least one of the following models: image classification model, speech recognition model, and text translation model.

[0052] For example, in training an image classification model, the teacher model can be the model used for image classification, and the training samples can include sample images and corresponding sample classification results. The teacher model and the target model can each predict the sample images, and a difference loss (including task loss and distillation loss) is calculated based on their predictions. For example, the task loss is calculated based on the target model's prediction and the sample classification results, and the distillation loss is calculated based on the teacher model's prediction and the target model's prediction. Then, the network parameters of the first branch network can be adjusted based on the gradient generated by the task loss, and the network parameters of the second branch network can be adjusted based on the gradient generated by the distillation loss. Finally, based on the parameter-adjusted first and second branch networks, a learning model for inference is determined, which can be the image classification model.

[0053] For example, in training a speech recognition model, the teacher model can be the model used for speech recognition, and the training samples can include sample speech and the corresponding sample speech recognition results. Based on the predictions of the sample speech by the teacher model and the target model, respectively, difference losses (including task loss and distillation loss) are calculated. For example, the task loss is calculated based on the prediction results of the target model and the sample speech recognition results, and the distillation loss is calculated based on the predictions of the teacher model and the target model. Then, the gradients generated by the two losses are used to adjust the network parameters of different branches of the network, and finally, the student model for inference is determined, which can be the speech recognition model.

[0054] For example, in training a text translation model, the teacher model can be the model used for text translation, and the training samples can include sample text to be translated and the corresponding sample translation results. Based on the predictions of the teacher model and the target model for the sample text to be translated, respectively, difference losses (including task loss and distillation loss) are calculated. For example, the task loss is calculated based on the prediction results of the target model and the sample translation results, and the distillation loss is calculated based on the predictions of the teacher model and the target model. Then, the gradients generated by the two losses are used to adjust the network parameters of different branches of the network, and finally, the student model for inference is determined, which can be the text translation model.

[0055] Reviewing the above process, in the embodiments of this specification, a target model can be trained. This target model may include a first branch network and a second branch network determined based on the network structure of the initial student model. A knowledge interaction module is set between the first and second branch networks. Specifically, firstly, the difference loss can be calculated based on the predictions made by the teacher model and the target model on the training samples, where the difference loss may include task loss and distillation loss. Then, the network parameters of the first branch network can be adjusted based on the gradient generated by the task loss, and the network parameters of the second branch network can be adjusted based on the gradient generated by the distillation loss. Then, based on the parameter-adjusted first and second branch networks, the student model for inference is determined. Thus, the gradients generated by the task loss and distillation loss can be decoupled during model training and used to adjust the network parameters of different branch networks respectively. The student model for inference is determined based on the parameter-adjusted first and second branch networks, thereby eliminating the mutual influence between the gradients generated by the task loss and distillation loss during knowledge distillation, allowing the student model for inference to acquire more knowledge and thus become more accurate.

[0056] According to another embodiment, a student model training apparatus based on gradient decoupling is provided. This gradient decoupling-based student model training apparatus can be deployed in any device, equipment, platform, or device cluster with computing and processing capabilities.

[0057] Figure 4 A schematic block diagram of a gradient-decoupling-based student model training apparatus according to one embodiment is shown. Figure 4As shown, the gradient-decoupled student model training device 400 may include: a computing unit 401 configured to calculate a difference loss based on the predictions made by the teacher model and the target model on the training samples, wherein the target model includes a first branch network and a second branch network determined based on the network structure of the initial student model, and a knowledge interaction module is provided between the first branch network and the second branch network for interactive processing of the outputs of a preset layer between the first branch network and the second branch network, and the difference loss includes task loss and distillation loss; a first adjustment unit 402 configured to adjust the network parameters of the first branch network based on the gradient generated by the task loss; a second adjustment unit 403 configured to adjust the network parameters of the second branch network based on the gradient generated by the distillation loss; and a determination unit 404 configured to determine the student model for inference based on the first branch network and the second branch network after parameter adjustment.

[0058] In some implementations, the first branch network includes a first linearly learnable layer and a first normalization layer, the second branch network includes a second linearly learnable layer and a second normalization layer, the knowledge interaction module is used to interactively process the outputs of the first linearly learnable layer and the second linearly learnable layer; and the computing unit 401 is further configured to: the knowledge interaction module interactively processes the outputs of the first linearly learnable layer and the second linearly learnable layer to obtain a processing result, and propagates the processing result forward to the first normalization layer and the second normalization layer; after fusing the outputs of the first normalization layer and the second normalization layer, they are respectively forward transmitted to the next layer of each branch network, until the prediction layer of the target model outputs the prediction result.

[0059] In some implementations, the first branch network and the second branch network have the same network structure; and the determining unit 404 is further configured to merge the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module into a linearly learnable layer in the student model for reasoning, wherein the parameters of the linearly learnable layer are determined based on the parameters of the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module; and to merge the other layers in the first branch network and the second branch network to generate the student model for reasoning, wherein the student model for reasoning has a single-branch structure.

[0060] In some implementations, the knowledge interaction module includes a parameter matrix; and the interaction processing of the outputs of the first linearly learnable layer and the second linearly learnable layer by the knowledge interaction module to obtain the processing result includes: using the parameter matrix to linearly combine the activation values ​​of the first linearly learnable layer and the second linearly learnable layer to obtain updated activation values.

[0061] In some implementations, the knowledge interaction module includes learnable parameters, and the device further includes a third adjustment unit (not shown in the figure), configured to adjust the learnable parameters of the knowledge interaction module with the goal of minimizing the difference loss.

[0062] In some implementations, the network structures of the first branch network and the second branch network are the same as the network structure of the initial student model.

[0063] In some implementations, the student model used for the above inference includes at least one of the following models: image classification model, speech recognition model, and text translation model.

[0064] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned gradient-based decoupling student model training method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0065] This invention also provides a computing device, including a processor, a memory, and a computer program / instructions stored in the memory, characterized in that the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the computing device implements the following: Figure 1 The steps of the method described.

[0066] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0067] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0068] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A student model training method based on gradient decoupling, comprising: Based on predictions of training samples by the teacher model and the target model respectively, a difference loss is calculated. The target model includes a first branch network and a second branch network determined based on the network structure of the initial student model. The first and second branch networks have identical network structures. A knowledge interaction module is provided between the first and second branch networks to interactively process the outputs of preset layers between them. The difference loss includes task loss and distillation loss. The first branch network includes a first linearly learnable layer and a first normalization layer, and the second branch network includes a second linearly learnable layer and a second normalization layer. The knowledge interaction module is used to interactively process the outputs of the first and second linearly learnable layers. Predicting the training samples based on the target model includes: the knowledge interaction module interactively processes the outputs of the first and second linearly learnable layers to obtain a processing result; the processing result is forward-propagated to the first and second normalization layers; the outputs of the first and second normalization layers are fused and then forward-propagated to the next layer of each branch network until the prediction layer of the target model outputs the prediction result. Based on the gradient generated by the task loss, adjust the network parameters of the first branch network; Based on the gradient generated by the distillation loss, adjust the network parameters of the second branch network; The first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module are merged into a linearly learnable layer in the student model for reasoning. The parameters of the linearly learnable layer are determined based on the parameters of the first linearly learnable layer, the first normalized layer, the second linearly learnable layer, the second normalized layer, and the knowledge interaction module. The other layers in the first branch network and the second branch network are merged to generate a student model for inference. The student model for inference has a single-branch structure and includes at least one of the following models: an image classification model, a speech recognition model, and a text translation model.

2. The method according to claim 1, wherein, The knowledge interaction module includes a parameter matrix; and the step of the knowledge interaction module interacting with the outputs of the first linearly learnable layer and the second linearly learnable layer to obtain a processing result includes: The activation values ​​output by the first linearly learnable layer and the activation values ​​output by the second linearly learnable layer are linearly combined using the parameter matrix to obtain updated activation values.

3. The method according to claim 1, wherein, The knowledge interaction module includes learnable parameters, and the method further includes: With the goal of minimizing the difference loss, the learnable parameters of the knowledge interaction module are adjusted.

4. The method according to claim 1, wherein, The network structures of the first branch network and the second branch network are the same as the network structure of the initial student model.

5. The method according to claim 2, wherein, The parameter matrix is: ， The calculation formula is as follows: ， ， in, Indicates the learnable parameters of the knowledge interaction module; Represents a matrix Apply the softmax activation function along the first dimension; Represents the identity matrix; This represents a hyperparameter threshold.

6. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 5.

7. A student model training device based on gradient decoupling, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 5.

Images