Knowledge distillation method, electronic device, and storage medium

Abstract

The present application discloses a knowledge distillation method, an electronic device, and a storage medium, wherein the knowledge distillation method includes: using a student model and a teacher model to perform prediction processing on a first training sample, wherein the prediction processing process includes multiple processing stages, and different processing stages are performed by different network layers in the student model and the teacher model; obtaining the contribution index of the output data of the teacher model in each processing stage to the training of the student model; based on the contribution index corresponding to each processing stage, selecting at least one processing stage as the target stage; using the output data of the teacher model and the student model in each target stage, adjusting the parameters of the student model. The above scheme can improve the effect of knowledge distillation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a knowledge distillation method, electronic equipment, and storage medium. Background Art

[0002] Knowledge distillation is a training method for model compression of neural networks. It is usually used to design lightweight neural networks or high-performance neural networks for embedded devices, and is widely used in various visual tasks. Knowledge distillation usually requires pre-training a large neural network (called the teacher model) with a large number of parameters or a slow running speed but high model performance indicators. The data features or data labels learned by the teacher model are used to guide a small neural network (called the student model) with a small number of parameters or a fast running speed but poor model performance, so that the student model can achieve better performance.

[0003] At present, in each round of training of existing knowledge distillation, the teacher model uses a fixed preset number of data features or labels to guide the training of the student model. Since the network scale of the teacher model is usually much larger than that of the student model, the features learned by the teacher model cannot be effectively learned by the student model through distillation, resulting in a decrease in the final performance of the student model. In other words, the model capacity gap between the teacher model and the student model causes the teacher model to be unable to distill a student model with better performance. In view of this, how to improve the effect of knowledge distillation has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a knowledge distillation method, an electronic device, and a storage medium.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: a knowledge distillation method, comprising: using a student model and a teacher model to perform prediction processing on a first training sample respectively, wherein the prediction processing process includes multiple processing stages, and different processing stages are executed by different network layers in the student model and the teacher model; obtaining the contribution index of the output data of the teacher model in each processing stage to the training of the student model; based on the contribution index corresponding to each processing stage, selecting at least one processing stage as the target stage; and adjusting the parameters of the student model using the output data of the teacher model and the student model in each target stage.

[0006] According to one embodiment of the present application, the output data of each processing stage includes at least one of the following: intermediate layer features of at least one intermediate processing stage and prediction results of the final processing stage, wherein the resolutions of the intermediate layer features of different intermediate processing stages are different.

[0007] According to an embodiment of the present application, the intermediate layer features of the intermediate processing stage are the first intermediate layer features outputted by the intermediate processing stage, or at least one preset type of second intermediate layer features extracted from the first intermediate layer features.

[0008] According to one embodiment of the present application, in the case of at least one preset type of second intermediate layer feature obtained by extracting the first intermediate layer feature, in each intermediate processing stage, each preset type corresponds to a contribution indicator, and the contribution indicator corresponding to each preset type is obtained based on the second intermediate layer feature of the preset type.

[0009] According to one implementation mode of the present application, the contribution index of the output data of the teacher model at each processing stage to the training of the student model is obtained, including: for each processing stage, using the difference between the output data of the teacher model and the student model at the processing stage, respectively, to obtain the first optimization target of the student model at the processing stage; based on the first optimization target of each processing stage, respectively, determining the first parameter gradient corresponding to each processing stage; respectively using the first parameter gradient corresponding to each processing stage, respectively, to obtain the contribution index corresponding to each processing stage.

[0010] According to one embodiment of the present application, before respectively using the first parameter gradient corresponding to each processing stage to obtain the contribution index corresponding to each processing stage, the method also includes: using the difference between the prediction result of the student model and the label result of the first training sample to obtain the second optimization target of the student model; based on the second optimization target, determining the second parameter gradient of the student model; respectively using the first parameter gradient corresponding to each processing stage to obtain the contribution index corresponding to each processing stage, including: for each processing stage, obtaining the similarity between the second parameter gradient and the first parameter gradient corresponding to the processing stage as the contribution index corresponding to the processing stage.

[0011] According to an embodiment of the present application, the similarity between the second parameter gradient and the first parameter gradient is determined based on the angle between the second parameter gradient and the first parameter gradient, wherein the angle is negatively correlated with the similarity.

[0012] According to an embodiment of the present application, based on the contribution index corresponding to each processing stage, at least one processing stage is selected as the target stage, including: selecting a processing stage whose contribution index meets a preset condition as the target stage.

[0013] According to one embodiment of the present application, the output data of the teacher model and the student model at each target stage are used to adjust the parameters of the student model, including: adjusting the parameters of the student model using the first optimization target corresponding to the student model at each target stage and the second optimization target of the student model; wherein the first optimization target corresponding to each target stage is obtained by using the output data of the teacher model and the student model at each target stage, and the second optimization target is obtained by using the prediction result of the learning model and the label result of the first training sample.

[0014] According to one implementation mode of the present application, the parameters of the student model are adjusted using the first optimization target corresponding to the student model at each target stage and the second optimization target of the student model, including: weighting the first optimization target and the second optimization target to obtain a third optimization target; obtaining a third parameter gradient based on the third optimization target; and adjusting the parameters of the student model using the third parameter gradient.

[0015] According to an embodiment of the present application, before respectively using the student model and the teacher model to perform prediction processing on the first training sample, the method further includes: training the teacher model using the second training sample.

[0016] To solve the above technical problems, another technical solution adopted in this application is: an electronic device, including a processor, a memory and a communication circuit, the memory and the communication circuit are coupled to the processor; the memory stores program instructions, and the processor is used to execute the program instructions to implement the knowledge distillation method in the above solution.

[0017] In order to solve the above technical problems, another technical solution adopted in this application is: a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the knowledge distillation method in the above solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0019] Figure 1 It is a flow chart of an embodiment of the knowledge distillation method of the present application;

[0020] Figure 2 It is a schematic diagram of a framework of an embodiment of a student model and a teacher model;

[0021] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step S12;

[0022] Figure 4 It is a flow chart of another embodiment of the knowledge distillation method of the present application;

[0023] Figure 5 This is a process diagram of an embodiment of the knowledge distillation method of the present application;

[0024] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0025] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0027] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the knowledge distillation method of the present application. Specifically, it may include the following steps:

[0028] Step S11: Use the student model and the teacher model to perform prediction processing on the first training sample respectively.

[0029] In the disclosed embodiment, the prediction processing process includes multiple processing stages, and different processing stages are performed by different network layers in the student model and the teacher model. Specifically, the number of processing stages included in the prediction processing process is not limited here. For example, the prediction processing process may include 1 processing stage, and the student model and the teacher model may include 1 network layer correspondingly, or the prediction processing process may include 2 processing stages, and the student model and the teacher model may include 2 network layers correspondingly, or the prediction processing process may include 3 processing stages, and the student model and the teacher model may include 3 network layers correspondingly, which is not limited here.

[0030] In one implementation scenario, the resolutions of features output by different network layers are different. For example, the resolution of features output by the first network layer may be w×h / 2, the resolution of features output by the second network layer may be w×h / 4, the resolution of features output by the third network layer may be w×h / 8, and so on. Examples are not given one by one here.

[0031] In one implementation scenario, the resolution of the output features of the i-th network layer in the student model can be the same as the resolution of the output features of the i-th network layer in the teacher model. Figure 2 , Figure 2 Schematic diagram of the framework of an embodiment of a student model and a teacher model. Figure 2 As shown, both the student model and the teacher model have 4 network layers. For the convenience of description, the resolution of the output feature of the i-th network layer in the student model can be recorded as R i s , the feature resolution of the output of the i-th network layer in the teacher model can be recorded as R i t , and R i s Can be equal to R i t , where i∈{1,2,3,4}, i can be an integer between 1 and 4. Other cases can be deduced by analogy and will not be described here. It should be noted that the neural network contained in each network layer can be set according to the actual application needs and is not limited here. For example, for the teacher model, in order to improve its model performance, the network layer can include complex networks such as residual blocks, while for the student model, in order to simplify its model structure, the network layer can include a small number of lightweight networks such as convolutional layers, which are not limited here.

[0032] In an implementation scenario, the first training sample can be set according to actual application needs. For example, taking the target detection scenario as an example, the first training sample can be a sample image of the target area marked with the target object, or, taking the target segmentation scenario as an example, the first training sample can be a sample image marked with the edge contour of the target object. Other scenarios can be deduced by analogy, and examples are not given one by one here. It should be noted that the target object can also be set according to actual application needs. For example, in home monitoring applications, the target object can include but is not limited to: people, cats, dogs, and other pets, or, in smart door lock applications, the target object can include but is not limited to: biometric features such as human faces and fingerprints, or, in smart refrigerator applications, the target object can include but is not limited to: vegetables, fish, shrimp and other ingredients. Other applications can be deduced by analogy, and examples are not given one by one here.

[0033] In an implementation scenario, in order to improve the effect of knowledge distillation, before the prediction process, the teacher model can be trained using the second training sample. Specifically, a second training sample of the target task can be obtained, and the target task can include but is not limited to: target detection task, target segmentation task, etc., which are not limited here. In addition, the second training sample can be annotated with a label result (such as a target area, edge contour, etc.). On this basis, the second training sample can be predicted using the teacher model to obtain a prediction result, and the parameters of the teacher model can be adjusted based on the difference between the prediction result and the label result. Specifically, the parameters can be adjusted using optimization methods such as gradient descent. For the specific process, please refer to the technical details of optimization methods such as gradient descent, which will not be repeated here. In the above method, before the first training sample is predicted and processed by the student model and the teacher model respectively, the teacher model is first trained using the second training sample, which helps to improve the performance of the teacher model. On this basis, knowledge distillation is performed, which is conducive to improving the effect of knowledge distillation.

[0034] Step S12: Obtain the contribution index of the output data of the teacher model at each processing stage to the training of the student model.

[0035] In an implementation scenario, the contribution index may indicate the degree of benefit of each processing stage of the teacher model to the training of the student model. For example, the greater the contribution index of a processing stage of the teacher model to the training of the student model, the higher the benefit of the processing stage to the training of the student model. Conversely, the smaller the contribution index of a processing stage of the teacher model to the training of the student model, the lower the benefit of the processing stage to the training of the student model.

[0036] In an implementation scenario, the output data of each processing stage may include at least one of the following: intermediate layer features of at least one intermediate processing stage and the prediction results of the final processing stage, and as mentioned above, the resolutions of the intermediate layer features of different intermediate processing stages are different. In the above method, since the output data of each processing stage includes at least one of the intermediate layer features of at least one intermediate processing stage and the prediction results of the final processing stage, the contribution index can be evaluated from different processing stages such as the intermediate processing stage and the final processing stage, which is conducive to evaluating the contribution index corresponding to each processing stage as comprehensively as possible, and helps to improve the effect of knowledge distillation.

[0037] In a specific implementation scenario, the output data of each processing stage may include at least one intermediate layer feature of the intermediate processing stage. For example, the output data of each processing stage may include one intermediate layer feature of the intermediate processing stage, two intermediate layer features of the intermediate processing stage, or three intermediate layer features of the intermediate processing stage. This is not limited here. In addition, when the prediction processing process includes multiple intermediate processing stages, it is not limited to which one or several intermediate layer features of the intermediate processing stage are included in the output data of each processing stage. Please refer to Figure 2 , the prediction processing process includes three intermediate processing stages, corresponding to the 1st to 3rd network layers respectively, then the output data of each processing stage may include the intermediate layer features output by the 1st network layer, the intermediate layer features output by the 2nd network layer, and the intermediate layer features output by the 3rd network layer, or, the output data of each processing stage may include the intermediate layer features output by the 1st network layer and the intermediate layer features output by the 2nd network layer, or, the output data of each processing stage may only include the intermediate layer features output by the 1st network layer, or, the output data of each processing stage may only include the intermediate layer features output by the 2nd network layer, or, the output data of each processing stage may only include the intermediate layer features output by the 3rd network layer, and so on, and no examples are given here one by one.

[0038] In a specific implementation scenario, the output data of each processing stage may include the prediction result of the final processing stage. For example, taking the target detection scenario as an example, the prediction result may include the predicted area of ​​the target object in the first training sample, or taking the target segmentation scenario as an example, the prediction result may include the predicted contour of the target object in the first training sample. Other scenarios can be deduced by analogy, and examples are not given one by one here. Please refer to Figure 2 The prediction process can include one final processing stage, corresponding to the fourth network layer. Other network structures can be deduced in the same way, and no examples are given here one by one.

[0039] In a specific implementation scenario, the intermediate layer feature of the intermediate processing stage may be the first intermediate layer feature outputted by the intermediate processing stage. Figure 2 As mentioned above, the prediction process includes three intermediate processing stages, corresponding to the first to third network layers respectively. The intermediate layer features of the first intermediate processing stage can be the first intermediate layer features output by the first network layer, the intermediate layer features of the second intermediate processing stage can be the first intermediate layer features output by the second network layer, and the intermediate layer features of the third intermediate processing stage can be the first intermediate layer features output by the third network layer. In other network structures, this can be deduced by analogy, and no examples are given here one by one.

[0040] In a specific implementation scenario, the intermediate layer features in the intermediate processing stage can also be at least one second intermediate layer feature of a preset type extracted from the first intermediate layer features. Specifically, the at least one preset type may include but is not limited to: texture features, position features, etc., which are not limited here. Taking the example that at least one preset type includes texture features and position features, the first intermediate layer features can be subjected to texture feature extraction and position feature extraction, respectively, to obtain second intermediate layer features about texture features and second intermediate layer features about position features. It should be noted that texture features can reflect the visual characteristics of homogeneous phenomena in an image, and reflect the surface tissue structure arrangement properties of slowly changing or periodically changing surfaces. In addition, position features can reflect the position properties of target objects in an image.

[0041] In an implementation scenario, as described above, each processing stage may include at least one of an intermediate processing stage and a final processing stage, and the intermediate layer features of the intermediate processing stage may include the first intermediate layer features output by the intermediate processing stage, or at least one second intermediate layer feature of a preset type extracted from the first intermediate layer features. On this basis, corresponding contribution indicators may be calculated for different processing stages and different intermediate layer features.

[0042] In a specific implementation scenario, for the final processing stage, the contribution index of the final processing stage of the teacher model to the training of the student model can be analyzed based on the difference between the prediction results of the teacher model and the student model in the final processing stage. The specific calculation process can be referred to the following public embodiments, which will not be described here.

[0043] In a specific implementation scenario, for the intermediate processing stage, the contribution index of the intermediate processing stage of the teacher model to the training of the student model can be analyzed based on the difference between the first intermediate layer features of the teacher model and the student model in the intermediate processing stage. The specific calculation process can be referred to the following public embodiments, which will not be described here.

[0044] In a specific implementation scenario, for the intermediate processing stage, the contribution index of at least one preset type of the intermediate processing stage of the teacher model to the training of the student model can also be analyzed based on the difference between the second intermediate layer features of at least one preset type of the teacher model and the student model in the intermediate processing stage. That is to say, in the case where at least one preset type of second intermediate layer feature is extracted from the first intermediate layer feature, in the intermediate processing stage, each preset type corresponds to a contribution index, and the contribution index corresponding to each preset type is obtained based on the second intermediate layer feature of the preset type. For example, for the i-th intermediate processing stage, the contribution index of the position feature of the i-th intermediate processing stage of the teacher model to the training of the student model can be obtained based on the difference between the second intermediate layer features of the position feature of the teacher model and the student model in the i-th intermediate processing stage. At the same time, the contribution index of the texture feature of the i-th intermediate processing stage of the teacher model to the training of the student model can be obtained based on the difference between the second intermediate layer features of the texture feature of the teacher model and the student model in the i-th intermediate processing stage.

[0045] Step S13: Based on the contribution index corresponding to each processing stage, select at least one processing stage as the target stage.

[0046] In an implementation scenario, a processing stage whose contribution index meets the preset conditions can be selected as the target stage. The above method selects the target stage according to whether the contribution index meets the preset conditions, so as to adjust the parameters of the student model according to the output data of the teacher model and the student model in the target stage, which can help eliminate the influence of the processing stage whose contribution index does not meet the preset conditions on the training of the student model, and help improve the effect of knowledge distillation.

[0047] In a specific implementation scenario, as mentioned above, the contribution index can represent the degree of benefit of each processing stage of the teacher model to the training of the student model, and the contribution index is positively correlated with the degree of benefit. In this case, the preset condition can be set to that the contribution index is greater than the preset index threshold, that is, the processing stage with a greater degree of benefit can be selected as the target stage.

[0048] In a specific implementation scenario, for the intermediate processing results, if the contribution index of the intermediate processing stage of the teacher model to the training of the student model is analyzed based on the difference between the first intermediate layer features of the teacher model and the student model in the intermediate processing stage, then for the intermediate processing stage, it is possible to directly detect whether the contribution index of the intermediate processing stage meets the preset conditions. If so, the intermediate processing stage is used as the target stage, otherwise the intermediate processing stage may not be selected.

[0049] In a specific implementation scenario, for the intermediate processing stage, if the contribution index of at least one preset type of the intermediate processing stage of the teacher model to the training of the student model is analyzed based on the difference between the second intermediate layer features of at least one preset type of the teacher model and the student model in the intermediate processing stage, then for the intermediate processing stage, it is possible to detect whether the contribution indicators corresponding to various preset types meet the preset conditions. If there is at least one contribution indicator corresponding to the preset type that meets the preset conditions, the intermediate processing stage can be selected as the target stage. Otherwise, if no contribution indicator corresponding to the preset type meets the preset conditions, the intermediate processing stage may not be selected.

[0050] In a specific implementation scenario, for the final processing stage, as mentioned above, the contribution index of the final processing stage of the teacher model to the training of the student model can be analyzed based on the difference between the prediction results of the teacher model and the student model in the final processing stage. For the final processing stage, it is possible to detect whether the contribution index of the final processing stage meets the preset conditions. If so, the final processing stage is used as the target stage, otherwise the final processing stage may not be selected.

[0051] In an implementation scenario, all processing stages can also be used as target stages, and the reference weight of each processing stage can be determined according to the contribution index of each processing stage. For example, the contribution index can represent the degree of benefit of each processing stage of the teacher model to the training of the student model, and the contribution index is positively correlated with the benefit. Then the reference weight of the processing stage can be positively correlated with its contribution index, that is, the larger the contribution index of the processing stage, the larger the reference weight of the processing stage, and vice versa. The smaller the contribution index of the processing stage, the smaller the reference weight of the processing stage. In addition, when the contribution index of the processing stage is a negative number, the reference weight of the processing stage can be set to a preset value (such as 0, 0.001, 0.005, etc.). Therefore, by setting the reference weight according to the contribution index, the processing stage with a lower degree of benefit can be suppressed, and the processing stage with a higher degree of benefit can be enhanced, which is conducive to improving the knowledge distillation effect.

[0052] In a specific implementation scenario, illustratively, in order to reduce the computational complexity of the reference weight, the contribution index may be directly used as the reference weight. In addition, if the contribution index of the processing stage is a negative number, the reference weight of the processing stage may be reset to 0.

[0053] In a specific implementation scenario, for the intermediate processing stage, if the contribution index of at least one preset type of the intermediate processing stage of the teacher model to the training of the student model is analyzed based on the difference between the second intermediate layer features of at least one preset type of the teacher model in the intermediate processing stage, then for the intermediate processing stage, the reference weight corresponding to each preset type of the intermediate processing stage can be obtained according to the contribution index corresponding to each preset type of the intermediate processing stage. For example, taking the preset types including texture features and position features as an example, for the i-th intermediate processing stage, the contribution index corresponding to its texture features and the contribution index corresponding to the position features can be calculated, and on this basis, the contribution index corresponding to the texture features can be directly used as the reference weight of the texture features of the i-th intermediate processing stage, and the contribution index corresponding to the position features can be directly used as the reference weight of the position features of the i-th intermediate processing stage. Other situations can be deduced by analogy, and examples are not given one by one here.

[0054] In a specific implementation scenario, for the intermediate processing stage, if the contribution index of the intermediate processing stage of the teacher model to the training of the student model is analyzed based on the difference between the first intermediate layer features of the teacher model and the student model in the intermediate processing stage, then for the intermediate processing stage, the reference weight of the intermediate processing stage can be obtained according to the contribution index of the intermediate processing stage. For example, for the i-th intermediate processing stage, its contribution index can be directly used as the reference weight of the i-th intermediate processing stage. Other situations can be deduced by analogy, and examples are not given one by one here.

[0055] In a specific implementation scenario, for the final processing stage, as mentioned above, the contribution index of the final processing stage of the teacher model to the training of the student model can be analyzed based on the difference between the prediction results of the teacher model and the student model in the final processing stage, and the reference weight of the final processing stage can be obtained according to the contribution index of the final processing stage. For example, the contribution index of the final processing stage can be directly used as the reference weight of the final processing stage. Other situations can be deduced by analogy, and no examples are given here one by one.

[0056] Step S14: Using the output data of the teacher model and the student model at each target stage, adjust the parameters of the student model.

[0057] Specifically, the first optimization target corresponding to the student model at each target stage and the second optimization target of the student model can be used to adjust the network parameters of the student model, and the first optimization target corresponding to each target stage is obtained by using the output data of the teacher model and the student model at each target stage, respectively, and the second optimization target is obtained by using the test results of the student model and the label results of the first training sample. In the above method, when adjusting the parameters of the student model, not only the first optimization target of the student model at each target stage is paid attention to, but also the second optimization target of the student model. Therefore, the two can be used to jointly constrain the student model to learn the knowledge transferred by the teacher model, which is conducive to improving the knowledge distillation effect.

[0058] In an implementation scenario, as described above, a processing stage whose contribution index meets preset conditions can be selected as the target stage, and for the intermediate processing stage, the contribution index of the intermediate processing stage for each preset type can be obtained based on the second intermediate layer characteristics of at least one preset type, and the target stage can be selected on this basis, or the contribution index of the intermediate processing stage can be obtained based on the first intermediate layer characteristics output by the intermediate processing stage, and the target stage can be selected on this basis. For different processing methods of the contribution index, different calculation methods can be used to obtain the first optimization target of each target stage.

[0059] In a specific implementation scenario, if the target stage is the final processing stage, the first optimization target of the final processing stage can be obtained based on the difference between the prediction results of the teacher model and the student model in the final processing stage. Specifically, functions such as KL divergence (i.e., Kullback-Leibler Divergence) can be used to measure the difference between the prediction results to obtain the first optimization target of the final processing stage. It should be noted that, taking target detection as an example, the prediction result may include the predicted area of ​​the target object, or, taking target segmentation as an example, the prediction result may include the predicted contour of the target object (such as the probability value of each pixel belonging to the target object), and other situations can be deduced by analogy, and examples are not given one by one here.

[0060] In a specific implementation scenario, if the target stage is an intermediate processing stage, and the contribution index is calculated based on the first intermediate layer features outputted from the intermediate processing stage, the first optimization target of the intermediate processing stage can be obtained based on the difference between the first intermediate layer features of the teacher model and the student model at the intermediate processing stage. Specifically, a function such as the L2 norm can be used to measure the difference between the first intermediate layer features to obtain the first optimization target of the intermediate processing stage.

[0061] In a specific implementation scenario, if the target stage is an intermediate processing stage, and the contribution index is calculated based on at least one preset type of second intermediate layer feature extracted from the first intermediate layer feature, that is, each preset type has a corresponding contribution index. In this case, the preset type whose contribution index meets the preset conditions can be selected as the target type of the intermediate processing stage, and then the first optimization target of the intermediate processing layer is obtained based on the difference between the second intermediate layer features of the target type of the teacher model and the student model in the intermediate processing stage. Specifically, a function such as the L2 norm can be used to measure the difference between the second intermediate layer features of the target type to obtain the first optimization target of the intermediate processing stage. Taking at least one preset type including position features and texture features as an example, the target stage is the i-th intermediate processing stage, that is, for the i-th intermediate processing stage, the position features and texture features respectively correspond to contribution indicators. If the contribution index corresponding to the position feature meets the preset conditions, the position feature can be used as the target type. On this basis, the first optimization target of the i-th intermediate processing stage can be obtained based on the difference between the second intermediate layer features of the position feature of the teacher model and the student model in the i-th intermediate processing stage.

[0062] In a specific implementation scenario, functions such as cross entropy loss, dice loss, etc. can be used to measure the difference between the prediction result of the student model and the label result of the first training sample to obtain the second optimization target. It should be noted that, taking target detection as an example, the label result may include the target area of ​​the target object marked by the first training sample, and the prediction result may include the predicted area of ​​the target object, or, taking target segmentation as an example, the label result may include the edge contour of the target object marked by the first training sample (such as the type of each pixel point, such as 0-1 label can be used to mark it as belonging to the target object, or not belonging to the target object), and the prediction result may include the predicted contour of the target object (such as the probability value of each pixel point belonging to the target object), and other situations can be deduced by analogy, and examples are not given one by one here.

[0063] In a specific implementation scenario, the first optimization target corresponding to the student model at each target stage and the second optimization target of the student model can be weighted to obtain a third optimization target, and based on the third optimization target, a third parameter voxel is obtained, and the parameters of the student model are adjusted using the third parameter gradient. For the specific process, please refer to the technical details of the model optimization method such as gradient descent, which will not be repeated here. It should be noted that the weights of the first optimization target and the second optimization target can be set according to the application needs. For example, the weights of each optimization target can be set to the same, or the weights can be set according to the processing stages corresponding to each optimization target, such as setting a smaller weight for the processing stage at the front position and setting a larger weight for the processing stage at the back position, which is not limited here. In the above method, by weighting the first optimization target and the second optimization target, a third optimization target is obtained, and based on the third optimization target, a third parameter gradient is obtained, and the parameters of the student model are adjusted using the third parameter gradient, which is conducive to adjusting the parameters of the student model. When the parameters of the student model are adjusted, the first optimization target and the second optimization target are referred to with emphasis by weighting, which is conducive to improving the knowledge distillation effect.

[0064] In an implementation scenario, as described above, all processing stages can also be used as target stages, and the reference weight of each processing stage can be determined based on the contribution index of each processing stage. For the intermediate processing stage, the contribution index of the intermediate processing stage for each preset type can be obtained based on the second intermediate layer characteristics of at least one preset type, or the contribution index of the intermediate processing stage can be obtained based on the first intermediate layer characteristics output by the intermediate processing stage. For different processing methods of the contribution index, different calculation methods can be used to obtain the first optimization target of each target stage.

[0065] In a specific implementation scenario, if the target stage is the final processing stage, the first optimization target of the final processing stage can be obtained based on the difference between the prediction results of the teacher model and the student model in the final processing stage. For details, please refer to the above-mentioned related description, which will not be repeated here.

[0066] In a specific implementation scenario, if the target stage is the intermediate processing stage, and the contribution index is calculated based on the first intermediate layer features output by the intermediate processing stage, the first optimization target of the intermediate processing stage can be obtained based on the difference between the first intermediate layer features of the teacher model and the student model in the intermediate processing stage. For details, please refer to the above-mentioned related description, which will not be repeated here.

[0067] In a specific implementation scenario, if the target stage is an intermediate processing stage, and the contribution index is calculated based on at least one preset type of second intermediate layer feature extracted from the first intermediate layer feature, that is, each preset type corresponds to a contribution index. In this case, for each preset type, the first optimization target corresponding to the preset type of the intermediate processing layer can be obtained based on the difference between the second intermediate layer features of the preset type of the teacher model and the student model in the intermediate processing stage. Specifically, as mentioned above, the L2 norm can be used to measure the difference to obtain the first optimization target. Taking at least one preset type including position features and texture features as an example, the target stage is the i-th intermediate processing stage, that is, for the i-th intermediate processing stage, the first optimization target of the i-th intermediate processing layer with respect to the position feature can be obtained based on the difference between the second intermediate features of the teacher model and the student model in the i-th intermediate processing stage with respect to the position feature, and the first optimization target of the i-th intermediate processing layer with respect to the texture feature can be obtained based on the difference between the second intermediate features of the teacher model and the student model in the i-th intermediate processing stage with respect to the texture feature.

[0068] In a specific implementation scenario, a function such as cross entropy loss or dice loss can be used to measure the difference between the prediction result of the student model and the label result of the first training sample to obtain the second optimization goal. For details, please refer to the above related description, which will not be repeated here.

[0069] In a specific scenario, as mentioned above, all processing stages can be taken as target stages, and the reference weights of each processing stage can be determined according to the contribution index of each processing stage. On this basis, the weighted weight of the second optimization target can be obtained, and the first optimization target and the second optimization target can be weighted (e.g., weighted average) using the reference weight and the weighted weight to obtain the third optimization target, and based on the third optimization target, the third parameter gradient is obtained, and the parameters of the student model are adjusted using the third parameter gradient. For details, please refer to the above-mentioned related description, which will not be repeated here. It should be noted that the weighted weight of the second optimization target can be set according to the value range of the reference weight. For example, when the value range of the reference weight is between 0 and 1, the weighted weight can also be set to between 0 and 1, or to an order of magnitude close to the reference weight, so as to avoid excessive emphasis on the first optimization target or excessive emphasis on the second optimization target due to the weighted weight being too small or too large relative to the reference weight, thereby affecting the knowledge distillation effect.

[0070] The above scheme uses the student model and the teacher model to perform prediction processing on the first training sample respectively, and the prediction processing process includes multiple processing stages. Different processing stages are executed by different network layers in the student model and the teacher model, and the contribution index of the output data of the teacher model in each processing stage to the training of the student model is obtained. Based on the contribution index corresponding to each processing stage, at least one processing stage is selected as the target stage. On this basis, the output data of the teacher model and the student model in each target stage are used to adjust the parameters of the student model. That is, in the process of knowledge distillation, the contribution index of each processing stage to the training of the student model is measured, and the target stage is focused on according to the contribution index. Therefore, it can greatly alleviate the impact of the model capacity gap on knowledge distillation, which is conducive to improving the effect of knowledge distillation.

[0071] See also Figure 3 , Figure 3 yes Figure 1 The flowchart of step S12 in the embodiment of FIG. 1 includes the following steps:

[0072] Step S31: For each processing stage, the difference between the output data of the teacher model and the student model in the processing stage is used to obtain the first optimization target of the student model in the processing stage.

[0073] Specifically, as mentioned above, the prediction processing process includes an intermediate processing stage and a final processing stage. On this basis, the output data of the intermediate processing stage may include intermediate layer features, and the output data of the final processing stage may include prediction results. The intermediate layer features of the intermediate processing stage may include the first intermediate layer features output by the intermediate processing stage, or at least one preset type of second intermediate layer features extracted from the first intermediate layer features. For different processing stages, the first optimization target can be calculated in different ways.

[0074] In one implementation scenario, for the final processing stage, the first optimization target of the final processing stage can be obtained based on the difference between the prediction results of the teacher model and the student model in the final processing stage. For details, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0075] In an implementation scenario, for the intermediate processing stage, if its output data includes the first intermediate layer features output by the intermediate processing stage, the first optimization target of the intermediate processing stage can be obtained based on the difference between the first intermediate layer features of the teacher model and the student model in the intermediate processing stage. For details, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0076] In an implementation scenario, for the intermediate processing stage, if its output data includes at least one preset type of second intermediate layer feature extracted from the first intermediate layer feature, then for each preset type, the first optimization target corresponding to the preset type of the intermediate processing layer can be obtained based on the difference between the second intermediate layer features of the preset type of the teacher model and the student model at the intermediate processing stage. For details, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0077] Step S32: Determine the first parameter gradient corresponding to each processing stage based on the first optimization target of each processing stage.

[0078] Specifically, for each processing stage, the partial derivatives of each parameter can be obtained for the first optimization objective and expressed in vector form to obtain the first parameter gradient corresponding to the processing stage. It should be noted that for the intermediate processing stage, if its output data contains at least one preset type of second intermediate layer feature extracted from the first intermediate layer feature, then for each intermediate processing stage, the first optimization objective corresponding to each preset type can be used to obtain its first parameter gradient, that is, in each intermediate processing stage, each preset type corresponds to a first parameter gradient. For the specific calculation process of the parameter gradient, please refer to the technical details of the gradient, which will not be repeated here.

[0079] Step S33: using the first parameter gradient corresponding to each processing stage respectively, to obtain the contribution index corresponding to each processing stage.

[0080] Specifically, the difference between the prediction result of the student model and the label result of the first training sample can be used to obtain the second optimization target of the student model. The specific calculation method can refer to the relevant description in the aforementioned public embodiment, which will not be repeated here. On this basis, the second parameter gradient of the student model can be determined based on the second optimization target. The specific calculation process is similar to the first parameter gradient, which will not be repeated here. After obtaining the second parameter gradient, for each processing stage, the similarity between the second parameter gradient and the first parameter gradient corresponding to the processing stage can be obtained as the contribution index corresponding to the processing stage. The above method, by obtaining the similarity between the second parameter gradient and the first parameter gradient corresponding to the processing stage as the contribution index corresponding to the processing stage, can accurately measure the beneficial degree of training the student model corresponding to each processing stage, which is conducive to improving the accuracy of the contribution index.

[0081] In one implementation scenario, it should be noted that for an intermediate processing stage, if its output data includes at least one preset type of second intermediate layer feature extracted from the first intermediate layer feature, then for each intermediate processing stage, the similarity between the first parameter gradients corresponding to the various preset types of the second parameter gradient can be obtained as the contribution index corresponding to the various preset types. Taking at least one preset type including position features and texture features as an example, for the i-th intermediate processing stage, the similarity between the second parameter gradient and the first parameter gradient corresponding to the position feature can be obtained as the contribution index corresponding to the position feature in the i-th intermediate processing stage, and the similarity between the second parameter gradient and the first parameter gradient corresponding to the texture feature can be obtained as the contribution index corresponding to the texture feature in the i-th intermediate processing stage. Other situations can be deduced by analogy, and no examples are given here one by one.

[0082] In one implementation scenario, the similarity between the second parameter gradient and the first parameter gradient can be determined based on the angle between the second parameter gradient and the first parameter gradient, and the angle is negatively correlated with the similarity. That is, the larger the angle, the smaller the similarity, and vice versa, the smaller the angle, the greater the similarity. Specifically, the cosine similarity can be used to measure the similarity between the first parameter gradient and the second parameter gradient. For example, for ease of description, in the i-th processing stage, the first parameter gradient corresponding to the k-th preset type can be recorded as Where x represents the first training sample input to the teacher model, and the second parameter gradient can be recorded as Then in the i-th processing stage, the contribution index corresponding to the k-th preset type can be recorded as

[0083] In an implementation scenario, after obtaining the first parameter gradient and the second parameter gradient, the first parameter gradient and the second parameter gradient can be further converted into unit vectors. On this basis, the similarity between the second parameter gradient and the first parameter gradient can also be determined based on the distance between the second parameter gradient and the second parameter gradient, and the distance and the similarity are negatively correlated. That is, the greater the distance, the smaller the similarity, and vice versa, the smaller the distance, the greater the similarity. Specifically, the Euclidean distance can be used to measure the similarity between the first parameter gradient and the second parameter gradient.

[0084] It should be noted that the above-mentioned use of angle, distance, etc. to measure similarity are only two possible ways of measuring similarity in actual application, and it does not limit the use of other ways to measure similarity, and no specific limitation is made here.

[0085] The above scheme, for each processing stage, utilizes the difference between the output data of the teacher model and the student model in the processing stage to obtain the first optimization target of the student model in the processing stage, and determines the first parameter gradient corresponding to each processing stage based on the first optimization target of each processing stage, and obtains the contribution index corresponding to each processing stage by utilizing the first parameter gradient of each processing stage, so that the contribution index of the processing stage can be accurately measured by the difference between the output data.

[0086] See also Figure 4 , Figure 4 This is a flow chart of another embodiment of the knowledge distillation method of the present application. Specifically, it may include the following steps:

[0087] Step S41: Use the student model and the teacher model to perform prediction processing on the first training sample respectively.

[0088] In the disclosed embodiment, the prediction process includes multiple processing stages, and different processing stages are performed by different network layers in the student model and the teacher model. In addition, it should be noted that in order to improve the knowledge distillation effect, before the prediction process, the teacher model can be trained using the second training sample until the teacher model converges. For details, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0089] Step S42: Obtain the contribution index of the output data of the teacher model at each processing stage to the training of the student model.

[0090] In the disclosed embodiment, each processing stage may include at least one of an intermediate processing stage and a final processing stage, and the output data of the final processing stage includes the prediction result. In addition, the intermediate processing stage outputs the first intermediate layer feature, and the first intermediate layer feature is extracted to obtain at least one preset type of second intermediate layer feature, and the output data of the intermediate processing stage includes at least one preset type of second intermediate layer feature, or the first intermediate layer feature, which is not limited here.

[0091] In an implementation scenario, when the output data of the intermediate processing stage includes at least one second intermediate layer feature of a preset type, in the intermediate processing stage, each preset type corresponds to a contribution index, and the contribution index corresponding to each preset type is obtained based on the second intermediate layer feature of the preset type. The specific process can be referred to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0092] In an implementation scenario, when the output data of the intermediate processing stage includes the first intermediate layer features, a contribution index can be calculated corresponding to each intermediate processing stage. The specific process can refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0093] Step S43: Select a processing stage whose contribution index meets preset conditions as the target stage.

[0094] In an implementation scenario, when the output data of the intermediate processing stage includes at least one preset type of second intermediate layer feature, in the intermediate processing stage, each preset type corresponds to a contribution index. In the processing stage, if the contribution index corresponding to at least one preset type meets the preset condition, the processing stage can be used as the target stage, and the preset type whose contribution index meets the preset condition can be used as the target type. Conversely, if the contribution index corresponding to no preset type meets the preset condition, the processing stage can be not selected. For details, please refer to the relevant description in the aforementioned disclosed embodiment.

[0095] In an implementation scenario, when the output data of the intermediate processing stage includes the first intermediate layer features, a contribution index can be calculated for each intermediate processing stage, and then it can be directly detected whether the contribution index corresponding to the intermediate processing stage meets the preset conditions. If it does, it can be selected as the target stage, otherwise it can be not selected. Figure 5 , Figure 5 This is a schematic diagram of the process of one embodiment of knowledge distillation of this application. Figure 5As shown, the solid arrow in the dotted box represents the second parameter gradient (the specific meaning can refer to the aforementioned disclosed embodiment), and the dotted arrow in the dotted box represents the first parameter gradient (the specific meaning can refer to the aforementioned disclosed embodiment), and the number at the dotted arrow represents the processing stage to which the first parameter gradient belongs, such as the dotted arrow numbered 01 represents the first parameter gradient of the first processing stage, and so on, and no examples are given here one by one. Taking the cosine similarity between the first parameter gradient and the second parameter gradient as an example to characterize the contribution index, if the angle between the first parameter gradient corresponding to the jth processing stage and the second parameter gradient is greater than or equal to 90 degrees and less than or equal to 180 degrees, then the contribution index corresponding to the jth processing stage is less than or equal to 0, and if the angle between the first parameter gradient corresponding to the jth processing stage and the second parameter gradient is greater than or equal to 0 degrees and less than 90 degrees, then the contribution index corresponding to the jth processing stage is greater than 0. Therefore, when the preset condition is set to that the cosine similarity is greater than 0, during the t-th iteration training of the student model, since the angle between the first parameter gradient and the second parameter gradient corresponding to the first processing stage is greater than 90 degrees, the angles between the first parameter gradient and the second parameter gradient corresponding to the remaining second processing stages, third processing stages, and fourth processing stages are all between 0 and 90 degrees, then the second processing stage, third processing stage, and fourth processing stage can be used as the target stage. Similarly, during the t+i-th iteration training of the student model, since only the angle between the first parameter gradient and the second parameter gradient corresponding to the fourth processing stage is between 0 and 90 degrees, the angles between the first parameter gradient and the second parameter gradient corresponding to the remaining first processing stage, second processing stage, and third processing stage are all greater than 90 degrees, then the fourth processing stage can be used as the target stage.

[0096] Step S44: Using the output data of the teacher model and the student model at each target stage, adjust the parameters of the student model.

[0097] In an implementation scenario, for the target stage including the final processing stage, the first optimization target can be obtained based on the difference between the prediction results of the teacher model and the student model in the final processing stage, and for the target stage including the intermediate processing stage, if the output data of the intermediate processing stage includes at least one preset type of second intermediate layer feature, then based on the difference between the second intermediate layer features of the target type of the teacher model and the student model in the intermediate processing stage, the first optimization target of the intermediate processing layer is obtained. On this basis, the difference between the prediction result of the student model and the label result of the first training sample is used to obtain the second optimization target, and the first optimization target and the second optimization target are weighted to obtain the third optimization target, and based on the third optimization target, the third parameter gradient is obtained, and the parameters of the student model are adjusted using the third parameter gradient. For details, please refer to the relevant description in the aforementioned public embodiment, which will not be repeated here.

[0098] In an implementation scenario, for the target stage including the final processing stage, the first optimization target of the student model in the final processing stage can be calculated by referring to the above description, and for the target stage including the intermediate stage, if the output data of the intermediate processing stage includes the first intermediate layer feature, then based on the difference between the first intermediate layer features of the teacher model and the student model in the intermediate processing stage, the first optimization target of the intermediate processing layer is obtained. On this basis, the difference between the prediction result of the student model and the label result of the first training sample is used to obtain the second optimization target, and the parameters of the student model are adjusted based on the first optimization target and the second optimization target. For details, please refer to the relevant description in the aforementioned public embodiment, which will not be repeated here. Please refer to the relevant description in conjunction with the above description. Figure 5 , in the t-th iteration, since the second processing stage, the third processing stage and the fourth processing stage are taken as the target stage, in the optimization target calculation process, the switch of the first processing stage is open, which means that the first processing stage is not considered in the t-th iteration process, and the second processing stage, the third processing stage and the fourth processing stage are closed, which means that the second processing stage, the third processing stage and the fourth processing stage are considered in the t-th iteration process; similarly, in the t+i-th iteration process, since the fourth processing stage is taken as the target stage, in the optimization target calculation process, the switches of the first processing stage, the second processing stage and the third processing stage are open, which means that the first processing stage, the second processing stage and the third processing stage are not considered in the t-th iteration process, and the fourth processing stage is closed, which means that the fourth processing stage is considered in the t+i-th iteration process.

[0099] The above scheme selects the target stage according to whether the contribution index meets the preset conditions, so as to adjust the parameters of the student model according to the output data of the teacher model and the student model in the target stage. This can help eliminate the influence of the processing stage where the contribution index does not meet the preset conditions on the training of the student model, and help improve the effect of knowledge distillation.

[0100] See also Figure 6 , Figure 6 Schematic diagram of the framework of an embodiment of an electronic device 60 of the present application. Figure 6 As shown, the electronic device 60 may include a processor 61 and a memory 62 coupled to each other, the memory 62 stores program instructions, and the processor 61 is used to execute the program instructions to implement the steps in any of the above-mentioned knowledge distillation method embodiments. Specifically, the electronic device 60 may include but is not limited to: a server, a laptop computer, a desktop computer, etc., which is not limited here.

[0101] The processor 61 may also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 61 may also be any conventional processor, etc.

[0102] In the disclosed embodiment, the processor 61 is used to perform prediction processing on the first training sample using the student model and the teacher model respectively, wherein the prediction processing process includes multiple processing stages, and different processing stages are executed by different network layers in the student model and the teacher model; the processor 61 is used to obtain the contribution index of the output data of the teacher model in each processing stage to the training of the student model; the processor 61 is used to select at least one processing stage as the target stage based on the contribution index corresponding to each processing stage; the processor 61 is used to adjust the parameters of the student model using the output data of the teacher model and the student model in each target stage.

[0103] In the above scheme, during the knowledge distillation process, the contribution index of each processing stage to the training of the student model is measured, and the reference target stage is focused based on the contribution index. Therefore, the impact of the model capacity gap on knowledge distillation can be greatly alleviated, which is conducive to improving the knowledge distillation effect.

[0104] In some disclosed embodiments, output data of each processing stage includes at least one of the following: intermediate layer features of at least one intermediate processing stage and prediction results of the final processing stage, wherein the resolutions of the intermediate layer features of different intermediate processing stages are different.

[0105] In some disclosed embodiments, the intermediate-layer features of the intermediate processing stage are first intermediate-layer features output by the intermediate processing stage, or at least one preset type of second intermediate-layer features extracted from the first intermediate-layer features.

[0106] In some disclosed embodiments, when at least one preset type of second intermediate layer feature is obtained by extracting the first intermediate layer features, in the intermediate processing stage, each preset type corresponds to a contribution indicator, and the contribution indicator corresponding to each preset type is obtained based on the second intermediate layer feature of the preset type.

[0107] In some disclosed embodiments, the processor 61 is used to obtain, for each processing stage, a first optimization target of the student model in the processing stage by utilizing the difference between the output data of the teacher model and the student model in the processing stage; the processor 61 is used to determine the first parameter gradient corresponding to each processing stage based on the first optimization target of each processing stage; the processor 61 is used to obtain the contribution index corresponding to each processing stage by utilizing the first parameter gradient corresponding to each processing stage.

[0108] In some disclosed embodiments, the processor 61 is used to obtain a second optimization target of the student model by utilizing the difference between the prediction result of the student model and the label result of the first training sample; the processor 61 is used to determine the second parameter gradient of the student model based on the second optimization target; the processor 61 is used to obtain the similarity between the second parameter gradient and the first parameter gradient corresponding to the processing stage for each processing stage, as a contribution indicator corresponding to the processing stage.

[0109] In some disclosed embodiments, the similarity between the second parameter gradient and the first parameter gradient is determined based on an angle between the second parameter gradient and the first parameter gradient, wherein the angle is negatively correlated with the similarity.

[0110] In some disclosed embodiments, the processor 61 is configured to select a processing stage whose contribution index satisfies a preset condition as a target stage.

[0111] In some disclosed embodiments, the preset condition is that the contribution index is greater than a preset index threshold.

[0112] In some disclosed embodiments, the processor 61 is used to adjust the parameters of the student model using a first optimization target corresponding to the student model at each target stage and a second optimization target of the student model; wherein the first optimization target corresponding to each target stage is obtained by using the output data of the teacher model and the student model at each target stage, respectively, and the second optimization target is obtained by using the prediction result of the student model and the label result of the first training sample.

[0113] In some disclosed embodiments, the processor 61 is used to weight the first optimization objective and the second optimization objective to obtain a third optimization objective; the processor 61 is used to obtain a third parameter gradient based on the third optimization objective; the processor 61 is used to adjust the parameters of the student model using the third parameter gradient.

[0114] In some disclosed embodiments, the processor 61 is used to train the teacher model using the second training sample.

[0115] See also Figure 7 , Figure 7 Schematic diagram of a framework of an embodiment of a computer-readable storage medium 70 of the present application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps in any of the above-mentioned knowledge distillation method embodiments.

[0116] In the above scheme, during the knowledge distillation process, the contribution index of each processing stage to the training of the student model is measured, and the reference target stage is focused based on the contribution index. Therefore, the impact of the model capacity gap on knowledge distillation can be greatly alleviated, which is conducive to improving the knowledge distillation effect.

[0117] It should be noted that the storage device can specifically be a medium that can store program instructions, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the program instructions. The server can send the stored program instructions to other devices for execution, or it can also execute the stored program instructions itself.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic, such as the division of modules or units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0122] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A knowledge distillation method, characterized in that: include: Using a student model and a teacher model for performing an image processing task to perform prediction processing on a first sample image as a first training sample, respectively, wherein the prediction processing process includes multiple processing stages, and different processing stages are performed by different network layers in the student model and the teacher model; Obtaining the contribution index of the output data of the teacher model at each of the processing stages to the training of the student model; wherein, when the processing stage is an intermediate processing stage, the output data includes image features extracted by the network layer executing the intermediate processing stage and serving as intermediate layer features, and when the processing stage is a final processing stage, the output data includes the prediction result of executing the image processing task on the first sample image, and obtaining the contribution index of the output data of the teacher model at each of the processing stages to the training of the student model includes: for each of the processing stages, using the difference between the output data of the teacher model and the student model at the processing stage, respectively, to obtain the first optimization target of the student model at the processing stage; determining the first parameter gradient corresponding to each of the processing stages based on the first optimization target of each of the processing stages; and obtaining the contribution index corresponding to each of the processing stages using the first parameter gradient corresponding to each of the processing stages; Based on the contribution indicators corresponding to the processing stages, selecting at least one of the processing stages as a target stage; The parameters of the student model are adjusted by utilizing the difference between the output data of the teacher model and the student model at each target stage.

2. The method according to claim 1, characterized in that The resolutions of the intermediate layer features at different intermediate processing stages are different.

3. The method according to claim 2, characterized in that The intermediate layer features of the intermediate processing stage are the first intermediate layer features output by the intermediate processing stage, or at least one preset type of second intermediate layer features extracted from the first intermediate layer features.

4. The method according to claim 3, characterized in that In the case where at least one preset type of second intermediate layer feature is extracted from the first intermediate layer feature, in the intermediate processing stage, each of the preset types corresponds to the contribution index, and the contribution index corresponding to each of the preset types is obtained based on the second intermediate layer feature of the preset type.

5. The method according to claim 1, characterized in that Before respectively using the first parameter gradients corresponding to the processing stages to obtain contribution indicators corresponding to the processing stages, the method further includes: Using the difference between the prediction result of the student model and the label result of the first training sample, a second optimization target of the student model is obtained; Based on the second optimization objective, determining a second parameter gradient of the student model; The step of respectively using the first parameter gradient corresponding to each of the processing stages to obtain a contribution index corresponding to each of the processing stages includes: For each of the processing stages, a similarity between the second parameter gradient and the first parameter gradient corresponding to the processing stage is obtained as a contribution index corresponding to the processing stage.

6. The method according to claim 5, characterized in that The similarity between the second parameter gradient and the first parameter gradient is determined based on an angle between the second parameter gradient and the first parameter gradient, wherein the angle is negatively correlated with the similarity.

7. The method according to claim 1, characterized in that The selecting at least one of the processing stages as a target stage based on the contribution indicators corresponding to the processing stages includes: The processing stage in which the contribution index meets a preset condition is selected as the target stage.

8. The method according to claim 7, characterized in that The preset condition is that the contribution index is greater than a preset index threshold.

9. The method according to claim 1, characterized in that: The adjusting the parameters of the student model by utilizing the difference between the output data of the teacher model and the student model at each target stage comprises: Using the first optimization target corresponding to the student model at each target stage and the second optimization target of the student model, adjusting the parameters of the student model; Among them, the first optimization target corresponding to each target stage is obtained by measuring the difference between the output data of the teacher model and the student model in each target stage, and the second optimization target is obtained by using the prediction result of the student model and the label result of the first training sample.

10. The method according to claim 9, characterized in that The adjusting the parameters of the student model by using the first optimization target corresponding to the student model at each target stage and the second optimization target of the student model comprises: Weighting the first optimization objective and the second optimization objective to obtain a third optimization objective; Based on the third optimization objective, obtaining a third parameter gradient; The parameters of the student model are adjusted using the third parameter gradient.

11. The method according to claim 1, characterized in that: Before respectively using the student model and the teacher model for performing the image processing task to perform prediction processing on the first sample image as the first training sample, the method further includes: The teacher model is trained using the second sample image as the second training sample.

12. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, the memory stores program instructions, and the processor is used to execute the program instructions to implement the knowledge distillation method described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the knowledge distillation method described in any one of claims 1 to 11.

Images