Multi-teacher knowledge distillation method and system
By introducing a transition model and scheduler adjustment, the multi-teacher knowledge distillation method solves the problem that student models have difficulty learning knowledge from teacher models, achieving model lightweighting and performance improvement. In particular, in image super-resolution tasks, it dynamically adjusts learning weights to optimize knowledge transfer in key regions.
Patent Information
- Application Number
- CN202510779727.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-11
- Publication Date
- 2025-11-07
Smart Images
Figure CN120911544A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of deep learning, and particularly relates to a multi-teacher knowledge distillation method and system. BACKGROUND
[0002] In recent years, deep learning has become the cornerstone of the tremendous success of artificial intelligence, covering applications in multiple fields such as computer vision and reinforcement learning. With the support of the latest technologies such as residual connection and batch normalization, it is easy to train deep models with thousands of layers on powerful GPU or TPU clusters. These large deep neural networks perform well in processing large-scale data, so they are widely used in practical applications.
[0003] However, deploying these huge deep models on mobile devices and embedded systems faces huge challenges in terms of computing power and memory limitations. To solve this problem, scholars have proposed the method of knowledge distillation, which is a common model compression method in the industry. The student model learns from the pre-trained teacher model, so that the student model can approach the performance of the teacher model as much as possible while maintaining lightweight.
[0004] The teacher model is usually a large, complex and outstanding model, which is trained on large-scale data and can learn rich features and complex patterns. The student model is a smaller and more concise model, and the goal is to train it to replicate the performance of the teacher model as much as possible. During training, the teacher model generates prediction results for the same input data, which reflect high-level relationship reasoning and spatial understanding. The student model tries to imitate the behavior of the teacher model, for example, during training, it not only minimizes its own prediction error for unlabeled data, but also adjusts its learning goal according to the soft targets provided by the teacher model, that is, to make its output as close to the teacher model's output as possible. In this way, the student model can learn the decision-making process and feature representation of the teacher model, thereby achieving model miniaturization and acceleration while maintaining high accuracy. In addition, knowledge distillation can also be performed through intermediate layer information. In this case, the intermediate layer feature representations of the teacher model and the student model are extracted and aligned through specially designed intermediate layer mapping rules and alignment modules, achieving knowledge alignment and loss calculation between the teacher model and the student model based on the intermediate layer. This method allows the student model to learn the key feature information of the intermediate layer of the teacher model, thereby better understanding the internal rules of the teacher model.
[0005] However, when there is a large difference between the teacher model and the student model, the student model has difficulty effectively learning these knowledge. SUMMARY
[0006] Therefore, the present application aims to provide a multi-teacher knowledge distillation method and system, so that the student model first learns the knowledge of the transition model, and then gradually learns the knowledge of the teacher model.
[0007] In a first aspect, the present disclosure provides a multi-teacher knowledge distillation method, comprising:
[0008] A transition model is constructed according to a student model and a trained teacher model, and a corresponding relationship of feature layers of the teacher model, the student model and the transition model is established;
[0009] The training sample is input to the teacher model to output a prediction result;
[0010] The transition model is trained using the training sample, wherein the knowledge of the teacher model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layers of the transition model and the teacher model and the difference between the prediction results of the transition model and the teacher model;
[0011] The student model is trained using the training sample, wherein the knowledge of the teacher model and the transition model is transferred to the student model by minimizing the difference between the feature information of the corresponding feature layers of the student model, the teacher model and the transition model and the difference between the prediction results of the student model, the teacher model and the transition model;
[0012] The training of the student model is adjusted using a scheduler, wherein the student model is gradually switched from learning from the transition model to learning from the teacher model.
[0013] In some embodiments, further comprising: using the scheduler to adjust the proportion of offline distillation and online distillation of the current period.
[0014] In some embodiments, the loss function of the transition model is represented as:
[0015] Tloss = Weight1 * loss1 + Weight3 * loss3 + Weight7 * loss7;
[0016] The loss function of the student model is represented as:
[0017] Sloss = Weight2 * loss2 + Weight4 * loss4 + Weight5 * loss5 + Weight6 * loss6 + Weight8 * loss8;
[0018] Wherein, Weight1 to Weight8 represent weights, loss1 is the loss of the corresponding feature layer of the teacher model and the transition model, loss2 is the loss of the corresponding feature layer of the learning model and the teacher model, loss6 is the loss of the corresponding feature layer of the learning model and the transition model, loss3 is the output loss of the teacher model and the transition model, loss4 is the output loss of the learning model and the transition model, loss5 is the output loss of the learning model and the teacher model, loss7 is the loss between the output result of the transition model and the real data, and loss8 is the loss between the model result of the student model and the real data.
[0019] The scheduler controls the student model to gradually switch from learning from the transition model to learning from the teacher model by adjusting the weights in the loss function.
[0020] In some embodiments, the teacher model, the transition model and the student model are all super-resolution models, and the adjusting the student model to gradually switch from learning from the transition model to learning from the teacher model comprises:
[0021] Taking the feature maps of the corresponding convolution layers of the teacher model, the transition model and the student model as inputs, the correlation degree between the pixel points in each feature map is calculated through a self-attention mechanism to generate an attention weight matrix;
[0022] The similarity of the attention weight matrices of the teacher model, the transition model and the student model is calculated, and the feature similarity of the student model and the teacher model, the transition model in each region is obtained;
[0023] According to the feature similarity of the student model and the teacher model, the transition model in each region, the learning degree of the student model in each region to the transition model and the teacher model is adjusted.
[0024] In some embodiments, for regions with high similarity, the learning weight of the student model to the transition model in the region is reduced, and the learning weight to the teacher model is increased.
[0025] In some embodiments, the cosine similarity or the Euclidean distance is used to calculate the similarity of the attention weight matrices of the teacher model, the transition model and the student model.
[0026] In some embodiments, further comprising: segmenting a low-resolution image sample using a pre-trained semantic segmentation model to obtain different semantic regions; and
[0027] Importance of different semantic regions is combined into similarity calculation of each region to assign higher learning weight to key semantic regions.
[0028] In some embodiments, for each pixel p, the weight of the loss function is calculated according to the following formula: The weight of the error between the student model and the transition model in the loss function is: =(1+α⋅(1− ))⋅(1+β⋅M(p)); The weight of the error between the student model and the teacher model in the loss function is: =(1+α⋅(1− ))⋅(1+β⋅M(p)); Wherein, represents the region similarity matrix of the feature map output by the student model and the transition model at the i-th layer, represents the region similarity matrix of the feature map output by the student model and the teacher model at the i-th layer, M represents the mask matrix output by the segmentation model, and a and β are weight parameters adjusted by the scheduler, a represents a similarity weight coefficient, and β represents a segmentation mask weight coefficient.
[0029] In some embodiments, the training samples are divided into multiple batches, and after the end of each batch, the scheduler adjusts the weights according to the feature similarity and loss value of the corresponding convolutional layer of the current batch.
[0030] In a second aspect, the embodiments of the present disclosure provide a multi-teacher knowledge distillation system, comprising:
[0031] A teacher model prediction module configured to input a training sample into the teacher model to output a prediction result;
[0032] A transition model training module configured to train a transition model using the training sample, wherein the knowledge of the teacher model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layers of the transition model and the teacher model and the difference between the prediction results of the transition model and the teacher model;
[0033] A student model training module configured to train a student model using the training sample, wherein the knowledge of the teacher model and the transition model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layers of the student model, the teacher model and the transition model and the difference between the prediction results of the student model, the teacher model and the transition model, wherein the correspondence relationship of the feature layers of the teacher model, the transition model and the student model is set when the model is constructed;
[0034] The scheduler is configured to adjust the learning degree of the student model to the transition model and the teacher model in the student model training module.
[0035] Optionally, the teacher model, the transition model and the student model are all super-resolution models, and the corresponding feature layer is a corresponding convolution layer.
[0036] In a third aspect, the present disclosure provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the multi-teacher knowledge distillation method.
[0037] In a fourth aspect, the present disclosure provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-teacher knowledge distillation method when executing the computer program.
[0038] In a fifth aspect, the present disclosure provides an artificial intelligence chip for implementing the multi-teacher knowledge distillation method.
[0039] The innovation of the present application is that a transition model is added, and the network lightweight task is realized by combining the weight adjustment of the scheduler. The scheduler adjustment can not only adjust the learning degree of the student model from the transition model and the teacher model, but also control the gradual transition from learning from the transition model to learning from the teacher model. In the image super-resolution task, the scheduler can also control the student model to learn from one area to another area by comparing the similarity of the feature maps of each area, and control the different learning degrees of the student model to the key area and the non-key area.
[0040] It should be noted that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS
[0041] The above and other objects, features and advantages of the present application will become more apparent from the following description of the embodiments of the present application, taken in conjunction with the accompanying drawings, in which:
[0042] Figure 1 is a flowchart of a multi-teacher knowledge distillation method provided by the present disclosure;
[0043] Figure 2 A schematic diagram of a multi-teacher knowledge distillation scheme for an image super-resolution task is given.
[0044] Figure 3 A functional block diagram of a multi-teacher knowledge distillation system provided by the present disclosure is given. DETAILED DESCRIPTION
[0045] The application will be described in greater detail with reference to the accompanying drawings. Like elements in the various drawings are denoted by like reference numerals for consistency. Various portions of the drawings have not been drawn to scale for the sake of clarity. Further, some known portions can not be shown.
[0046] Multi-teacher knowledge distillation utilizes the knowledge of multiple teacher models to improve the performance of a student model. Compared with single-teacher knowledge distillation, the advantage of multi-teacher knowledge distillation lies in that each teacher network model can provide a unique perspective and knowledge, so that the student model learns different knowledge and thus improves the generalization ability of the student network model. Therefore, the embodiment of the present disclosure proposes a multi-teacher knowledge distillation method, Figure 1 The basic flowchart of the method is given.
[0047] In step S101, a transition model is constructed according to the student model and the trained teacher model, and a corresponding relationship of the feature layers of the teacher model, the student model and the transition model is established. In this step, the network structure of the student model is first constructed according to the power consumption requirement and the network structure of the teacher model, then the network structure of the transition model is constructed according to the network structures of the teacher model and the student model, and the level corresponding relationship, especially the feature layer corresponding relationship, of the teacher model, the transition model and the student model is determined. In actual operation, the intermediate value of the depth and level of the teacher model and the student model is taken as the depth and level of the transition model.
[0048] In step S102, the training sample is input to the teacher model to output a prediction result.
[0049] In step S103, the transition model is trained using the training sample, wherein the knowledge of the teacher model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layers of the transition model and the teacher model and the difference between the prediction results of the transition model and the teacher model.
[0050] In step S104, the student model is trained using the training sample, wherein the knowledge of the teacher model and the transition model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layers of the student model, the teacher model and the transition model and the difference between the prediction results of the student model, the teacher model and the transition model.
[0051] In step S105, the training of the student model is adjusted using a scheduler, wherein the student model is gradually switched from learning from the transition model to learning from the teacher model.
[0052] The embodiment is based on the idea of multi-teacher knowledge distillation, and proposes a transition model between the teacher model and the student model. The distillation efficiency of the large model is improved by adjusting the student model to gradually switch from learning from the transition model to learning from the teacher model through the scheduler. Preferably, the transition model and the student model are both online distillation, and the online distillation ratio of the transition model and the student model in the current period is adjusted by using the scheduler to solve the limitation of the learning ability caused by the offline distillation of the single teacher model. In addition, the student model can learn the information of the transition model and the teacher model in different dimensions through the prediction result and the feature information extracted by the feature layer, fully utilize the diversity and richness of different teacher networks to obtain more comprehensive knowledge transmission.
[0053] In practical applications, the distillation loss is usually combined with the task loss to form a total loss function. Therefore, in the above embodiment, the total loss function of the transition model can be:
[0054] Tloss = Weight1*loss1 + Weight3*loss3 + Weight7*loss7 Formula (1);
[0055] At the same time, the total loss function of the student model is defined as:
[0056] Sloss = Weight2*loss2 + Weight4*loss4 + Weight5*loss5 + Weight6*loss6+ Weight8*loss8 Formula (2);
[0057] Wherein, Weight1 to Weight8 represent the weights, which can be normalized, loss1 is the loss of the corresponding feature layer of the teacher model and the transition model, loss2 is the loss of the corresponding feature layer of the learning model and the teacher model, loss6 is the loss of the corresponding feature layer of the learning model and the transition model, loss3 is the output loss of the teacher model and the transition model, loss4 is the output loss of the learning model and the transition model, loss5 is the output loss of the learning model and the teacher model, loss7 is the loss between the output result of the transition model and the real data, and loss8 is the loss between the model result of the student model and the real data. Accordingly, the scheduler can control the student model to gradually switch from learning from the transition model to learning from the teacher model by adjusting the weights in the loss function, for example, as loss5 / loss8 increases and loss4 / loss5 decreases, it can be considered that the student model has learned sufficient knowledge from the transition model, at this time we can adjust the weight of the distillation loss to increase the learning proportion of the teacher model and reduce the learning proportion of the transition model to realize dynamic adjustment and improve the learning ability of the student model.
[0058] The multi-teacher knowledge distillation method of the embodiment can be used for an image super-resolution task. The image super-resolution is to use the strong learning ability of a deep neural network to restore a low-resolution (LR) image to a high-resolution (HR) image while retaining the details and texture information of the image as much as possible. Figure 2 A schematic diagram of a multi-teacher knowledge distillation scheme for an image super-resolution task is given.
[0059] As shown in Figure 2 During the training of the super-resolution task, an attention mechanism is introduced. The importance of different regions of the image is calculated through the attention mechanism, and the learning weights of the transition model, the teacher model and different regions are dynamically adjusted in combination with the knowledge matching degree of the regions and the transition model and the teacher model.
[0060] First, the feature maps of the student model, the transition model and the teacher model are processed by using the attention module 205. Specifically, when the teacher model, the transition model and the student model are constructed, the layer correspondence relationship between the multiple convolution layers included in the teacher model, the transition model and the student model is set, and then the attention module 205 is set after the corresponding convolution layers of the three models. The feature maps output by the corresponding convolution layers of the three models are respectively taken as the inputs of the three attention modules 205, and the three attention modules 205 respectively calculate the correlation degree between the pixel points in each feature map output by the corresponding convolution layers in the three models by using the self-attention mechanism, thereby generating an attention weight matrix. The attention weight matrix can reflect the importance difference of different regions in the image in feature expression. Subsequently, the similarity between the attention weight matrices output by the student model and the teacher model in each regional feature is calculated by the two similarity calculation modules 204, and the similarity between the attention weight matrices output by the student model and the transition model in each regional feature is calculated, and the calculation results are output to the dispatcher 206. The dispatcher 206 determines whether the student model has learned sufficient knowledge from each region according to the similarity of the student model and the teacher model and the transition model in each regional feature. If the student model has learned sufficient knowledge from the transition model in a certain region, the weight of learning from the transition model in the region can be appropriately reduced, and the weight of learning from the teacher model can be increased until sufficient knowledge is learned from the teacher model in the region. At the same time, if the student model has not learned sufficient knowledge from the transition model in a certain region, the weight of learning from the transition model in the region is maintained or increased to promote the student model to further learn the knowledge of the transition model in the region.
[0061] More specifically, Figure 2 The flow of the attention module 205 in the above embodiment can include the following steps S1-S5.
[0062] Step S1, input feature map is passed through 1x1 convolution to generate Query, Key and Value feature maps respectively;
[0063] Step S2, the matrix product of Query and Key is calculated to obtain the correlation score (i.e. attention score) between pixel points.
[0064] Step S3, Softmax operation is performed on the correlation score to normalize the score into an attention weight matrix between 0 and 1.
[0065] Step S4, the attention weight matrix is multiplied with the Value feature map to obtain a weighted feature map.
[0066] Step S5, cosine similarity or Euclidean distance is used to compare the closeness of the weighted feature maps of the student model and the transition model and the teacher model at the corresponding convolution layer.
[0067] As an optional way, Figure 2 The super-resolution task training process shown in the figure can also introduce a semantic segmentation module 207, and the semantic information output by the semantic segmentation module 207 is provided to the scheduler, and the scheduler enhances the learning weight of the key area in the image according to the semantic information. Specifically, the pre-trained semantic segmentation model segments the super-resolution image, divides the image into different semantic areas (such as people, buildings, natural landscapes, etc.), and obtains the masks of different semantic areas (such as people, buildings, and backgrounds); then, the scheduler combines the similarity calculation results of each area with the mask of the semantic area to distinguish the learning state of different semantic areas, and allocates higher learning weights to key semantic areas.
[0068] In Figure 2 During the super-resolution task training process shown in the figure, the scheduler 206 can realize the gradual transition from learning from the transition model to learning from the teacher model and the proportion of online distillation and offline distillation at the current time by adjusting the corresponding weight in the loss function.
[0069] The following exemplary illustratesThe loss function in the scheduler 206 in Figure 2
[0070] For each pixel p, the scheduler 206 calculates the weight w(p) of the loss function according to the following formula:
[0071] The weight of the error between the student model and the transition model in the loss function is: =(1+α⋅(1− ))⋅(1+β⋅M(p)) (1); The weight of the error between the student model and the teacher model in the loss function is: =(1+α⋅(1− ))⋅(1+β⋅M(p)) (2; in, This represents the region similarity matrix of the feature maps output by the attention module 205 in layer i between the student model and the transition model (pixel level, range [0,1], larger values indicate higher similarity). M represents the region similarity matrix (pixel level, range [0,1]) of the feature maps output by the student model and the teacher model in the i-th layer attention module 205. M represents the mask matrix (pixel level, 0 represents background, 1 represents foreground and other regions of interest) output by the segmentation model. α and β are weight parameters adjusted by the scheduler. α represents the similarity weight coefficient (controls the influence of similarity on weight, α>0), and β represents the weight coefficient of the segmentation mask (controls the weight priority of key regions, β≥1).
[0072] Regarding the similarity weight of the transition model, if the student model and the transition model have low similarity at pixel p (i.e. (small), then 1− Larger sizes and heavier weights force student models to focus on learning that region.
[0073] Regarding the weights of the segmentation mask, if M(p)=1 (critical region), then the weights are multiplied by (1+β) to highlight the loss contribution of critical regions such as the foreground (β can be set to amplify by 2-5 times).
[0074] Regarding the teacher model similarity weight, if the student model and the teacher model have low similarity at pixel p ( (Small), increased weight, strengthens the distillation signal of the teacher model.
[0075] When using formulas (1) and (2) above to calculate the loss function of the intermediate layer, the similarity matrix corresponding to the attention module 205 needs to be used to achieve size correspondence; as for the mask matrix, it can be obtained by bicubic interpolation algorithm to the corresponding feature map size.
[0076] As an alternative, when designing the loss function, multi-scale / multi-level weight fusion can also be considered: if the model has multiple attention modules (e.g., shallow, medium, and deep layers), the global similarity matrix can be obtained by averaging or weighting the similarity matrices of each layer using the following formula. and Substituting this into the weight formula above, where N is the number of 205 layers in the attention module, this approach is suitable for scenarios requiring the integration of multiple layers of feature similarity: (3); (4).
[0077] Then, the pixel-level weight matrix w(p) is converted into a weight matrix W with the same dimension as the affinity matrix using FAKD loss (Feature Augmented Knowledge Distillation loss), including: 1. Flattening (w(p) into a one-dimensional vector .
[0078] 2. Calculate the outer product to represent the importance weight of the association between positions i and j in the feature map: .
[0079] The loss function of the student model is designed as follows: Student loss = loss2 weighted + loss4 weighted + loss5 weighted + loss6 weighted + loss8 weighted
[0080]
[0081]
[0082]
[0083] (5)
[0084] wherein loss2 weighted is the FAKD weighted loss between the student model and the transition model, loss4 weighted is the weighted L1 loss between the student model and the transition model, loss5 weighted is the weighted L1 loss between the student model and the teacher model, loss6 weighted is the FAKD weighted loss between the student model and the teacher model, and loss8 weighted is the weighted L1 loss between the student model and the real data, , , is the feature affinity matrix of the corresponding feature layer of the student model, the transition model and the teacher model, and is the affinity matrix level weight (dimension (HW x HW)) generated based on the attention similarity and the segmentation mask, and is the pixel-level weight (dimension HxW) generated based on the attention similarity and the segmentation mask), which is obtained through formulas (1) and (2), is the error of the output data between the student model and the transition model at pixel p, is an error of output data at the pixel p between the student model and the transition model, is an error of output data at the pixel p between the student model and the real data.
[0085] In addition, the training samples can also be divided into multiple batches, and after the end of each batch, the scheduler adjusts the weights according to the similarity and loss value of the feature map of the corresponding convolution layer of the current batch.
[0086] Correspondingly, the embodiment of the disclosure provides a multi-teacher knowledge distillation system, which specifically comprises the following modules. Figure 3 As shown in the figure, specifically comprises the following modules.
[0087] The teacher model prediction module 301 is configured to input the training sample into the teacher model to output a prediction result.
[0088] The transition model training module 302 is configured to train the transition model using the training sample, wherein the knowledge of the teacher model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layer of the teacher model and the transition model and the difference between the prediction results of the teacher model and the transition model.
[0089] The student model training module 303 is configured to train the student model using the training sample, wherein the knowledge of the teacher model and the transition model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layer of the teacher model and the transition model and the difference between the prediction results of the teacher model and the transition model, and wherein the correspondence between the feature layers of the teacher model, the transition model and the student model is set during model construction.
[0090] The scheduler 304 is configured to adjust the learning degree of the student model to the transition model and the teacher model in the student model training module.
[0091] Correspondingly, the embodiment of the disclosure also provides a computer readable storage medium, which stores one or more computer instructions, and the one or more computer instructions implement the functions of the steps or modules in the above embodiments when executed.
[0092] Correspondingly, the embodiment of the disclosure also provides a computing device, which includes a processor and a memory, and the memory stores one or more computer instructions executable by the processor, and the one or more computer instructions implement the functions of the steps or modules in the above embodiments when executed.
[0093] Correspondingly, the embodiment of the disclosure can also be implemented as an artificial intelligence chip, and the above-mentioned methods and modules are completed by the microprocessor, flash memory and other hardware in the chip combined with specific software programs.
[0094] In summary, various embodiments of the present disclosure add a transition model, and combine online distillation and offline distillation through a scheduler to adjust weights to achieve network lightweight tasks. The scheduler adjustment can not only adjust the learning degree of the student model from the transition model and the teacher model, but also control the gradual transition from learning from the transition model to learning from the teacher model. In the image super-resolution task, the scheduler can also control the student model to learn from one area to another area by comparing the similarity of the feature maps of each area, and control the different learning degrees of the student model for key areas and non-key areas.
[0095] It should be understood that the methods, systems, computer-readable storage media, and computing devices of the embodiments of the present disclosure are all based on the same thought, and can be referred to each other.
[0096] Although the embodiments of the present application are disclosed as above with preferred embodiments, they are not intended to limit the claims, and any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application, therefore the protection scope of the present application should be limited by the scope defined by the claims of the present application.
[0097] The above is only the preferred embodiment of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A multi-teacher knowledge distillation method, comprising: building a transition model according to a student model and a trained teacher model, and establishing a correspondence relationship of feature layers of the teacher model, the student model and the transition model; inputting a training sample to the teacher model to output a prediction result; training the transition model using the training sample, wherein knowledge of the teacher model is transferred to the transition model by minimizing a difference between feature information output by corresponding feature layers of the transition model and the teacher model and a difference between prediction results of the transition model and the teacher model; training the student model using the training sample, wherein knowledge of the teacher model and the transition model is transferred to the student model by minimizing a difference between feature information of corresponding feature layers of the student model, the teacher model and the transition model and a difference between prediction results of the student model, the teacher model and the transition model; adjusting training of the student model using a scheduler, wherein the student model is gradually switched from learning from the transition model to learning from the teacher model.
2. The multi-teacher knowledge distillation method of claim 1, further comprising: adjusting a proportion of offline distillation and online distillation of a current period using the scheduler.
3. The multi-teacher knowledge distillation method of claim 1, wherein, a loss function of the transition model is represented as: Tloss = Weight1*loss1 + Weight3*loss3 + Weight7*loss7; a loss function of the student model is represented as: Sloss = Weight2*loss2 + Weight4*loss4 + Weight5*loss5 + Weight6*loss6 + Weight8*loss8; wherein Weight1 to Weight8 represent weights, loss1 is a loss of corresponding feature layers of the teacher model and the transition model, loss2 is a loss of corresponding feature layers of the learning model and the teacher model, loss6 is a loss of corresponding feature layers of the learning model and the transition model, loss3 is an output loss of the teacher model and the transition model, loss4 is an output loss of the learning model and the transition model, loss5 is an output loss of the learning model and the teacher model, loss7 is a loss between an output result of the transition model and real data, and loss8 is a loss between a model result of the student model and real data, then the scheduler controls the student model to be gradually switched from learning from the transition model to learning from the teacher model by adjusting the weights in the loss functions.
4. The multi-teacher knowledge distillation method of claim 1, wherein, the teacher model, the transition model and the student model are all super-resolution models, and the adjusting of the student model to be gradually switched from learning from the transition model to learning from the teacher model comprises: taking feature maps of corresponding convolution layers of the teacher model, the transition model and the student model as inputs, calculating a correlation degree between pixel points in each feature map through a self-attention mechanism to generate an attention weight matrix. The similarity of the attention weight matrices of the teacher model, the transition model and the student model is calculated, and the feature similarity of the student model with the teacher model and the transition model in each region is obtained; According to the feature similarity of the student model with the teacher model and the transition model in each region, the learning degree of the student model in each region to the transition model and the teacher model is adjusted.
5. The multi-teacher knowledge distillation method of claim 4, wherein, For regions with high similarity, the learning weight of the student model in the region to the transition model is reduced, and the learning weight to the teacher model is increased.
6. The multi-teacher knowledge distillation method of claim 5, wherein, The similarity of the attention weight matrices of the teacher model, the transition model and the student model is calculated using cosine similarity or Euclidean distance.
7. The multi-teacher knowledge distillation method of claim 5, further comprising: A pre-trained semantic segmentation model is used to segment low-resolution image samples to obtain different semantic regions. And The importance of different semantic regions is combined into the similarity calculation of each region to assign higher learning weights to key semantic regions.
8. The multi-teacher knowledge distillation method of claim 7, wherein, For each pixel p, the weight of the loss function is calculated according to the following formula: The error between the student model and the transition model has a weight in the loss function of: = (1 + a · (1 - p) · (1 - M(p))) · (1 + b · M(p)); )) · (1 + b · M(p)); a weight of an error of the student model and the teacher model in a loss function is: = (1 + a · (1 - p) ) · (1 + β · M(p)); wherein, represents the region similarity matrix of the feature maps output by the student model and the transition model at the i-th layer, represents the region similarity matrix of the feature maps output by the student model and the teacher model at the i-th layer, M represents the mask matrix output by the segmentation model, and α and β are weight parameters adjusted by the scheduler, wherein α represents a similarity weight coefficient, and β represents a weight coefficient of the segmentation mask.
9. The multi-teacher knowledge distillation method of claim 1, wherein, The training samples are divided into multiple batches, and after the end of each batch, the scheduler adjusts the weight according to the feature similarity and loss value of the corresponding convolution layer of the current batch.
10. A multi-teacher knowledge distillation system, comprising: a teacher model prediction module for inputting a training sample to the teacher model to output a prediction result; a transition model training module for training a transition model using the training sample, wherein the knowledge of the teacher model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layer of the transition model and the teacher model and the difference between the prediction results of the transition model and the teacher model; a student model training module for training a student model using the training sample, wherein the knowledge of the teacher model and the transition model is transferred to the transition model by minimizing the difference between the feature information output by the corresponding feature layer of the student model, the teacher model and the transition model and the difference between the prediction results of the student model, the teacher model and the transition model, wherein the correspondence between the feature layers of the teacher model, the transition model and the student model is set during model construction; a scheduler for adjusting the learning degree of the student model to the transition model and the teacher model in the student model training module.
11. A computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the multi-teacher knowledge distillation method of any one of claims 1 to 9.
12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the multi-teacher knowledge distillation method of any one of claims 1 to 9 when executing the computer program.
13. An artificial intelligence chip for implementing the multi-teacher knowledge distillation method of any one of claims 1 to 9.