Knowledge distillation method and system based on embedded feature regularization

Abstract

A knowledge distillation method and system based on embedded feature regularization. The method comprises: collecting labeled image data related to a recognition task; calculating a feature center of a teacher model in each category on a whole training set; projecting in the direction of a category center an embedded feature of a student model, rotating to the direction of the category center an embedded feature of the teacher model, and constructing a feature regularization loss by means of regularizing a projection feature of the student model and a rotation feature of the teacher model; increasing the norm of the feature of the student model, and constraining the direction of the feature of the student model to be consistent with the direction of the category center; inserting the feature regularization loss into an existing knowledge distillation framework, and training the student model; and deploying the trained student model to a terminal device, predicting a logits vector on the basis of new data received by the terminal device, and thus completing the related recognition task. In the present invention, by means of increasing the norm of a feature of a student and constraining the direction of the feature to be consistent with the direction of a category center, the knowledge distillation performance is better.

Description

A knowledge distillation method and system based on embedding feature regularization Technical Field

[0001] The present invention relates to the field of deep neural network model compression, and in particular to a knowledge distillation method and system based on embedding feature regularization. Background Art

[0002] In recent years, deep neural networks have achieved rapid progress in various computer vision tasks. However, using neural networks for inference incurs high computational costs and network parameter storage costs. As a lightweight alternative, model compression methods such as knowledge distillation, network pruning, and quantization have been identified as effective techniques for improving the efficiency of neural network parameters, especially for platforms with limited computing resources, such as mobile devices. Knowledge distillation refers to training a lightweight model (called the student) under the supervision of a heavier model (called the teacher) without significantly losing performance. Compared to network pruning and quantization, knowledge distillation can achieve performance transfer under heterogeneous teacher-student conditions, thus having greater flexibility.

[0003] Using the teacher's features as knowledge, knowledge distillation methods achieve knowledge transfer by encouraging the student's features to resemble those of the teacher. Existing mainstream knowledge distillation methods can be broadly divided into two categories: probability-based (logits) distillation and feature-based distillation. Probabilistic distillation minimizes the KL divergence between the student and teacher's predicted probabilities for the input data (the output of the last layer). The underlying assumption of this method is that if the student can produce a similar prediction distribution as the teacher, then the student's performance (e.g., accuracy) should approach that of the teacher. However, probability-based distillation methods do not fully utilize the teacher's knowledge, such as the teacher's features in other layers. To leverage these features, feature distillation methods use the L2 distance on intermediate layer features to encourage the student's features to resemble the teacher's.

[0004] Although probability-based and feature-based distillation methods have achieved impressive results, forcing students to produce probabilities or features similar to those of teachers cannot directly serve the final task, such as pedestrian and vehicle recognition and detection in traffic scenarios, component defects and positioning in industrial manufacturing scenarios, and cruise missile target identification.

[0005] Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned deficiencies in the prior art and provide a knowledge distillation method and system based on embedding feature regularization.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A knowledge distillation method based on embedding feature regularization includes the following steps:

[0009] S1, collects labeled image data related to the recognition task and divides it into training set and validation set;

[0010] S2, perform data augmentation on the training set and validation set respectively;

[0011] S3, loads the pre-trained weights to the teacher model, freezes its network parameters, and randomly initializes the parameters of the student model;

[0012] S4, traverses the training set, feeds the data into the teacher model in batches, extracts the embedding features of the teacher model for each input sample, and calculates the mean of each category as the feature center;

[0013] S5: Input the same batch of data to the student model and the teacher model, extract the embedded features respectively, and perform a linear dimensionality transformation on the embedded features of the student model so that they have the same dimension as the embedded features of the teacher model;

[0014] S6, extract the feature center of the category to which each input sample belongs; rotate the embedded features of the teacher model to be in the same direction as the feature center to obtain the rotated teacher features; project the embedded features of the student model along the direction of the feature center to obtain the projected student features;

[0015] S7, when the norm of the projected student feature is less than or equal to the norm of the rotated teacher feature, minimize the Euclidean distance between the two, normalize the Euclidean distance, and record it as loss Otherwise, the loss To maximize the cosine similarity between the embedding features of the student model and the feature center;

[0016] S8, will lose Inserted into the existing knowledge distillation framework, the total loss of training the student model is equal to the cross entropy loss plus the knowledge distillation loss and loss balanced by a pair of hyperparameters sum;

[0017] S9, adjust the pair of hyperparameters to obtain the student model with the highest accuracy on the validation set, and deploy it to the terminal device; the terminal device inputs the received new data into the trained model to obtain the predicted probability vector of each category.

[0018] Furthermore, the data enhancement of the training set described in step S2 includes: scaling the image data with maintaining the aspect ratio, random cropping, random horizontal flipping, removing the mean, and adding jitter; the data enhancement of the validation set includes: scaling the image data with maintaining the aspect ratio, center cropping, and removing the mean.

[0019] Furthermore, in step S4, for the k-th category, the feature center is calculated as follows:

[0020] in, is the set of all training samples belonging to the kth category.

[0021] Furthermore, in step S4, the calculation of the category center is performed on the entire training set, or the category center is calculated only on the sample set predicted correctly by the teacher model.

[0022] Furthermore, in step S5, when the dimension of the embedding features of the student model is the same as that of the teacher model, step S6 is directly entered; when the dimension of the embedding features of the student model is different from that of the teacher model, a fully connected layer with learnable parameters is introduced outside the student model to transform the dimension of the embedding features of the student model to the same as that of the teacher model.

[0023] Furthermore, in steps S6 and S7, four alternative feature regularization methods are constructed. The four alternative methods are divided into two categories: norm regularization and direction regularization.

[0024] Furthermore, the norm regularization includes two types: MSE method and SFN method; the MSE method uses L2 distance to measure the feature difference between the embedded features of the student model and the embedded features of the teacher model; the SFN method is to gradually increase the norm of the embedded features of the student model with a fixed step size during the student model training process, that is, for each training step, the norm of the embedded features of the student model is increased by one step size.

[0025] Furthermore, the directional regularization includes two types: the cosine method and the InfoNCE method; the cosine method maximizes the cosine similarity between the embedded features of the student model and the feature centers described in step S4; the InfoNCE method maximizes the cosine similarity between the embedded features of the student model and the paired feature centers described in step S4, and at the same time, minimizes the cosine similarity between the embedded features of the student model and the unpaired feature centers.

[0026] Further, in step S8, the distillation loss The specific form of depends on the selected distillation framework. When the probability-based distillation framework is selected, the distillation loss The specific form is:

[0027] where q s and q t They represent the predicted probabilities of the student model and the teacher model for the input samples, and KL represents the KL divergence.

[0028] Furthermore, in step S8, when the feature-based distillation framework is selected, the distillation loss The specific form is:

[0029] where f s and f t Represent the intermediate layer features of the student model and the teacher model for the input samples respectively.

[0030] Furthermore, the total loss of the training student model in step S8 is:

[0031] in is the cross entropy loss, is the knowledge distillation loss, whose specific form depends on the selected distillation framework. The pair of hyperparameters α and β are used to balance and

[0032] Furthermore, in step S9, the accuracy of the student model on the validation set is calculated as follows:

[0033] Where p = M(x i ; α; β), x i is the i-th input sample in the validation set, α and β take the values ​​during model training described in step S8, M represents the student model, p is the predicted probability of the student model for the category to which the input sample belongs, argmax(p) is the index of the maximum predicted probability, y is the category label to which the input sample belongs, and N is the number of samples contained in the validation set.

[0034] Furthermore, in step S4, for the kth category, its feature center is recorded as c k , for the convenience of expression, the present invention ignores the subscript k;

[0035] Furthermore, in step S5, the embedding features of the student model and the teacher model, i.e., the output of the penultimate layer, are extracted and recorded as f s and f t , for f s Make a linear dimension transformation to satisfy f t have the same dimensions;

[0036] Furthermore, in step S6, for each input sample, the category center c of the category to which the sample belongs is extracted as described in S4, and the unit vector e=c / ||c||2 of the category center c is calculated; the embedded feature f of the rotation teacher model is rotated. t To the same direction as the category center c, denoted as pt , that is, p t =e·||f t ||2; embedding feature f of the student model s The projection along the direction of the category center c is denoted as p s ;

[0037] Furthermore, in step S7, when the norm of the projected student feature is less than or equal to the norm of the rotated teacher feature, that is, ||f s |2≤||f t When ||2, minimize p s With p t The distance between them, that is, min||p s -p t ||2. The feature norms of different samples vary greatly, so a normalization constraint is applied to the minimized distance, namely, ND loss:

[0038] From formula (1), we can see that to reduce the ND loss, we need to increase the student's feature norm (Norm), or make the student's features keep the same direction (Direction) as the teacher's category center. That is, the ND loss constrains both the norm and direction of the student's features.

[0039] When || f s |2>||f t When || 2, only the direction of the student's features is constrained to be consistent with the direction of the teacher's category center, that is:

[0040] Formula (1) and formula (2) have the same form, so let’s combine them:

[0041] Furthermore, in steps S6 and S7, four alternative methods of feature regularization are constructed. The four alternative methods are divided into two categories: norm regularization and direction regularization. The norm regularization includes two methods: MSE method and SFN method. The MSE method uses L2 distance to measure the feature difference between the embedded features of the student model and the embedded features of the teacher model, that is,

[0042] The SFN method is to gradually increase the feature norm of the student:

[0043] Among them, x i For input data, is the student model, θ and θ0 represent the parameters updated in the current iteration and the parameters updated in the previous iteration, r is a positive number, which represents the step size of the feature norm increase in each iteration;

[0044] Among them, the directional regularization includes two methods: cosine method and InfoNCE method; the cosine method is to maximize the cosine similarity between the embedding features of the student model and the feature center described in step S4, that is,

[0045] The InfoNCE method maximizes the student characteristics f s The unit vector e of the matching category center k The cosine similarity between them, while minimizing f s The center of the category that does not match e j Cosine similarity between:

[0046] The present invention also includes a knowledge distillation system based on embedding feature regularization, comprising:

[0047] The data collection module is used by the server to collect annotated image data related to the recognition task and divide it into training and validation sets according to the proportion;

[0048] Data preprocessing module, which performs different data preprocessing on the training set and validation set respectively;

[0049] The model loading module loads the publicly available pre-trained weights into the teacher model and sets its network parameters to frozen mode, i.e., the parameters are not updated; the parameters of the student model are randomly initialized;

[0050] Category center calculation module, used to calculate the feature center of the teacher model in each category;

[0051] Embedded feature extraction module, which extracts the embedded features of students and teachers respectively, f s and f t , for f s Perform a linear transformation to make it equal to f t have the same dimensions;

[0052] The embedded feature projection module projects and rotates the embedded features of students and teachers in the direction of the corresponding category center to obtain the projection vector p s and p t ;

[0053] Embedding feature regularization module, used to constrain the student's embedding features: increase the feature norm and feature direction to be consistent with the teacher's category center;

[0054] Student model training module, used to train the student model and select and save the optimal student model weights;

[0055] The model deployment module is used to deploy the student model to the terminal device. The terminal device inputs the received new data into the trained model to obtain the predicted probability vector and then complete the relevant tasks.

[0056] The present invention also includes a knowledge distillation device based on embedded feature regularization, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a knowledge distillation method based on embedded feature regularization.

[0057] The present invention also includes a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a knowledge distillation method based on embedding feature regularization of the present invention.

[0058] Through in-depth research and experimental verification, this paper proposes that increasing the norm of the student's embedded features and constraining the direction of the student's embedded features to align with the teacher's category center—that is, constraining the student's embedded features through regularization—can improve the performance of knowledge distillation that directly serves the target task, such as achieving higher accuracy. The embedded features described here refer to the output features of the penultimate layer of the student and teacher networks. Embedded features are the model's representation of the input data.

[0059] The advantages of the present invention are threefold:

[0060] ① This paper proposes a new analysis and practice method in the field of model compression knowledge distillation, namely regularizing student features: 1) increasing the norm of student features; 2) aligning feature directions with the category center of the teacher;

[0061] ② This paper proposes a variety of basic feature regularization methods, and experimentally proves that these basic regularization methods can improve the effect of knowledge distillation;

[0062] ③ This paper proposes a new loss function (called ND), which can simultaneously regularize the feature norm and direction of the student model. The ND loss proposed in this paper can be directly inserted into the existing knowledge distillation framework. A large number of experiments show that the method of this paper can achieve better knowledge distillation effect.

[0063] The beneficial effects of the present invention are as follows:

[0064] By increasing the norm of the student's embedded features and constraining their orientation to align with the teacher's category center—that is, by regularizing the student's embedded features—this method improves knowledge distillation performance, directly serving the target task, such as achieving higher accuracy. The embedded features refer to the output features of the penultimate layer of the student and teacher networks. Embedded features are the model's representation of the input data. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] FIG1 is a schematic diagram of a probabilistic distillation method of knowledge distillation based on embedding feature regularization according to the present invention.

[0066] FIG2 is a schematic diagram of feature distillation of a knowledge distillation method based on embedding feature regularization of the present invention.

[0067] FIG3 is a schematic diagram of the embedding feature projection of the present invention.

[0068] FIG4 is a loss curve diagram of the present invention on a pedestrian and vehicle recognition dataset.

[0069] FIG5 is a graph showing the accuracy of the present invention on a pedestrian and vehicle recognition dataset.

[0070] FIG6 is a loss curve diagram of the present invention on a natural scene data set.

[0071] FIG7 is a graph showing the accuracy of the present invention on a natural scene dataset.

[0072] FIG8 is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the scope of the present invention.

[0074] Example 1

[0075] Taking pedestrian and vehicle recognition in traffic scenes as an example, which includes target categories such as pedestrians, bicycles, electric vehicles, trucks, buses, and cars, this embodiment provides a pedestrian and vehicle recognition method using the knowledge distillation method based on embedding feature regularization of the present invention. Referring to Figure 1, the specific process is as follows:

[0076] S1, build a dataset: Construct annotated image data of traffic scenes of categories such as pedestrians, bicycles, electric vehicles, trucks, buses, and cars, and divide it into a training set and a validation set according to the proportion, with 50,000 images in the training set and 10,000 images in the validation set;

[0077] S2, data preprocessing: perform different data augmentation on the training set and validation set respectively; for the training set: maintain aspect ratio scaling of image data, random cropping, random horizontal flipping, remove mean, add jitter and other operations; for the validation set: maintain aspect ratio scaling of image data, center cropping, remove mean;

[0078] S3. Select a teacher model and a student model. The teacher model and the student model can have the same architecture or different architectures. Load the teacher model with publicly available pre-trained weights and set its network parameters to frozen mode, i.e., do not update the parameters. Randomly initialize the parameters of the student model. In this embodiment, the present invention sets the teacher model and the student model to ResNet-50 and MobileNet-v1, respectively, as detailed explanations. It should be noted that the present invention is also applicable to other model architectures.

[0079] S4, calculate the embedding feature center of each category: traverse the entire training set (50,000 annotated images), feed the data into the teacher model ResNet-50 in batches, and for each input sample, extract the embedding feature of the teacher model for the input sample, that is, the output of the second-to-last layer. After completing the traversal on the training set, calculate the mean of each category as the feature center. For the kth category, its feature center is recorded as c k , for the convenience of expression, the present invention ignores the subscript k;

[0080] S5, extract embedding features: Input the same batch of data to the student model MobileNet-v1 and the teacher model ResNet-50, extract the embedding features of the student and teacher respectively, that is, the output of the second to last layer, and record it as f s and f t , for f s Make a linear dimension transformation to satisfy f t have the same dimensions;

[0081] S6, projecting embedded features: for each input sample, extract the category center c of the category to which the sample belongs as described in S4, calculate the unit vector e=c / ||c||2 of the category center c; rotate the teacher's embedded features f t To the same direction as the category center c, denoted as p t , that is, p t =e·||f t ||2; embed the student's feature f s The projection along the direction of the category center c is denoted as p s ; Please refer to Figure 3 for the projection or rotation of the student and teacher embedding features;

[0082] S7, regularized embedding features: when ||f s ||2≤||f t When ||2, minimize p s With p t The Euclidean distance between them, that is, min||p s -p t ||2. The feature norms of different samples vary greatly, so the Euclidean distance is normalized, i.e., ND loss:

[0083] From formula (1), we can see that to reduce the ND loss, we need to increase the student's feature norm (Norm), or make the student's features keep the same direction (Direction) as the teacher's category center. That is, the ND loss constrains both the norm and direction of the student's features.

[0084] When || f s ||2>||f t When || 2, only the direction of the student's features is constrained to be consistent with the teacher's category center, that is:

[0085] Formula (1) and formula (2) have the same form, and the present invention combines the two together:

[0086] S8, training student model: Inserting ND loss into the existing probability-based knowledge distillation framework, the total loss of training the student model is:

[0087] in is the cross entropy loss, is the knowledge distillation loss, whose specific form depends on the selected distillation framework, and α and β are used as hyperparameters to balance and

[0088] S9, Model Deployment: Adjust the parameters α and β to obtain the student model with the highest accuracy on the validation set and deploy it to the terminal device. The terminal device inputs the received new data into the trained model to obtain the predicted probability vector, thereby completing the recognition of pedestrians and vehicles in the traffic scene.

[0089] In step S4, for the kth category, the feature center is calculated as follows:

[0090] in, is the set of all training samples belonging to the kth category;

[0091] In step S5, the student's embedded feature f s With the teacher's embedding feature f t The dimension of f is preferably introduced into a fully connected layer with learnable parameters, followed by batch normalization. s The dimension is transformed to the same as f t same;

[0092] In steps S6 and S7, the present invention also creates four other feature regularization methods, and the four alternative methods can be further divided into two categories: norm regularization and direction regularization;

[0093] The above-mentioned alternative norm regularization includes two methods: MSE method and SFN method; the MSE method uses L2 distance to measure the feature difference between the student and the teacher:

[0094] The SFN method is to gradually increase the feature norm of the student:

[0095] Among them, x i For input data, is the student model, θ and θ0 represent the parameters updated in the current iteration and the parameters updated in the previous iteration, r is a positive number, which represents the step size of the feature norm increase in each iteration;

[0096] The above-mentioned alternative directional regularization includes two methods: cosine method and InfoNCE method; the cosine method maximizes the student feature f s Cosine similarity with the unit vector e of the class center:

[0097] The InfoNCE method maximizes the student characteristics f s The unit vector e of the matching category center k The cosine similarity between them, while minimizing f s The center of the category that does not match e j Cosine similarity between:

[0098] In the step S8, It is the distillation loss. Its specific form depends on the selected distillation framework. When the probability (logits) based distillation framework is selected, the distillation loss The specific form is:

[0099] where q s and q t Represent the predicted probabilities of the student model and the teacher model for the input samples respectively;

[0100] When the feature-based distillation framework is selected, the distillation loss The specific form is:

[0101] where f s and f t Represent the intermediate layer features of the student model and the teacher model for the input samples respectively.

[0102] Table 1 shows a comparison between our method and existing knowledge distillation methods. As can be seen, adding our four basic regularization methods to the original KD framework improves accuracy by 1.40%, 1.67%, 1.53%, and 1.41%, respectively. Adding our ND method to the original KD framework increases accuracy by 2.45%. See Figures 4 and 5 for the training curves of our method.

[0103] Table 1 Comparison of the method of the present invention with other methods

[0104] Example 2

[0105] Taking outdoor natural scene recognition as an example, including target categories such as animals, birds, plants, and people, this embodiment provides a scene recognition method based on the knowledge distillation method of embedding feature regularization of the present invention. Referring to Figure 2, the specific process is as follows:

[0106] S1, build a dataset: Construct annotated image data of categories such as animals, birds, plants, and people, and divide it into training and validation sets according to the proportion, with 1.2 million training sets and 50,000 validation sets;

[0107] S2, data preprocessing: perform different data augmentation on the training set and validation set respectively; for the training set: maintain aspect ratio scaling of image data, random cropping, random horizontal flipping, remove mean, add jitter and other operations; for the validation set: maintain aspect ratio scaling of image data, center cropping, remove mean;

[0108] S3. Select a teacher model and a student model. The teacher model and the student model can have the same architecture or different architectures. Load the teacher model with publicly available pre-trained weights and set its network parameters to frozen mode, i.e., do not update the parameters. Randomly initialize the parameters of the student model. In this embodiment, the present invention sets the teacher model and the student model to ResNet-101 and ResNet-18, respectively, as a detailed description. It should be noted that the present invention is also applicable to other model architectures.

[0109] S4, calculate the embedding feature center of each category: traverse the entire training set (1.2 million annotated images), feed the data into the teacher model ResNet-101 in batches, and for each input sample, extract the embedding feature of the teacher model for the input sample, that is, the output of the second-to-last layer. After completing the traversal on the training set, calculate the mean of each category as the feature center. For the kth category, its feature center is recorded as c k , for the convenience of expression, the present invention ignores the subscript k;

[0110] S5, extract embedding features: Input the same batch of data to the student model ResNet-18 and the teacher model ResNet-101, extract the embedding features of the student and teacher respectively, that is, the output of the second to last layer, and record it as f s and f t , for f s Make a linear dimension transformation to satisfy f t have the same dimensions;

[0111] S6, projecting embedded features: for each input sample, extract the category center c of the category to which the sample belongs as described in S4, calculate the unit vector e=c / ||c||2 of the category center c; rotate the teacher's embedded features f t To the same direction as the category center c, denoted as p t , that is, p t =e·||f t ||2; embed the student's feature f s The projection along the direction of the category center c is denoted as p s ;

[0112] S7, regularized embedding features: when ||f s ||2≤||f t When ||2, minimize p s With p t The distance between them, that is, min||p s -p t ||2, the feature norms of different samples vary greatly, so the above constraints are normalized, namely ND loss:

[0113] From formula (1), we can see that to reduce the ND loss, we need to increase the student's feature norm (Norm), or make the student's features keep the same direction (Direction) as the teacher's category center. That is, the ND loss constrains both the norm and direction of the student's features.

[0114] When || f s ||2>||f t When || 2, only the direction of the student's features is constrained to be consistent with the teacher's category center, that is:

[0115] Formula (1) and formula (2) have the same form, and the present invention combines the two together:

[0116] S8, training student model: Inserting ND loss into the existing feature-based knowledge distillation framework, the total loss of training the student model is:

[0117] in is the cross entropy loss, is the knowledge distillation loss, whose specific form depends on the selected distillation framework, and α and β are used as hyperparameters to balance and

[0118] S9, Model Deployment: Adjust the parameters α and β to obtain the student model with the highest accuracy on the validation set and deploy it to the terminal device. The terminal device inputs the received new data into the trained model to obtain the predicted probability vector, thereby completing the scene recognition.

[0119] In step S4, for the kth category, the feature center is calculated as follows:

[0120] in, is the set of all training samples belonging to the kth category;

[0121] In step S5, the student's embedded feature f s With the teacher's embedding feature f t The dimension of f is preferably introduced into a fully connected layer with learnable parameters, followed by batch normalization. s The dimension is transformed to the same as f t same;

[0122] In the step S8, It is the distillation loss. Its specific form depends on the selected distillation framework. When the feature-based distillation framework is selected,

[0123] where f s and f t Represent the intermediate layer features of the student model and the teacher model for the input samples respectively;

[0124] Table 2 shows a comparison between our method and existing knowledge distillation methods. It can be seen that incorporating our ND method into the original KD / DKD / ReviewKD frameworks improves accuracy by 1.45%, 0.16%, and 0.99%, respectively. The training curves for our method are shown in Figures 6 and 7.

[0125] Table 2 Comparison of the method of the present invention with other methods

[0126] Example 3

[0127] 8 , the present invention further includes a knowledge distillation system based on embedding feature regularization, comprising:

[0128] The data collection module is used by the server to collect annotated image data related to the recognition task and divide it into training and validation sets according to the proportion;

[0129] Data preprocessing module, which performs different data preprocessing on the training set and validation set respectively;

[0130] The model loading module loads the publicly available pre-trained weights into the teacher model and sets its network parameters to frozen mode, i.e., the parameters are not updated; the parameters of the student model are randomly initialized;

[0131] Category center calculation module, used to calculate the feature center of the teacher model in each category;

[0132] Embedded feature extraction module, which extracts the embedded features of students and teachers respectively, f s and f t , for f s Perform a linear transformation to make it equal to f t have the same dimensions;

[0133] The embedded feature projection module projects and rotates the embedded features of students and teachers in the direction of the corresponding category center to obtain the projection vector p s and p t ;

[0134] Embedding feature regularization module, used to constrain the student's embedding features: increase the feature norm and feature direction to be consistent with the teacher's category center;

[0135] Student model training module, used to train the student model and select and save the optimal student model weights;

[0136] The model deployment module is used to deploy the student model to the terminal device. The terminal device inputs the received new data into the trained model to obtain the predicted probability vector and then complete the relevant tasks.

[0137] Example 4

[0138] An embodiment of the present invention also includes a knowledge distillation device based on embedded feature regularization, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a knowledge distillation method based on embedded feature regularization.

[0139] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the knowledge distillation method based on embedded feature regularization of the above-mentioned embodiments 1 and 2 is implemented.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge distillation method based on embedding feature regularization, characterized in that: The steps include: S1, collects labeled image data related to the recognition task and divides it into training set and validation set; S2, perform data augmentation on the training set and validation set respectively; S3, load the pre-trained weights to the teacher model, freeze its network parameters, and randomly initialize the parameters of the student model; S4, traverse the training set, feed the data into the teacher model in batches, extract the embedding features of the teacher model for each input sample, and calculate the mean of each category as the feature center; S5, input the same batch of data to the student model and the teacher model, extract embedding features respectively, and perform linear dimension transformation on the embedding features of the student model so that they have the same dimension as the embedding features of the teacher model; S6, extract the feature center of the category to which each input sample belongs; rotate the embedded features of the teacher model to the same direction as the feature center to obtain the rotated teacher features; project the embedded features of the student model along the direction of the feature center to obtain the projected student features; S7, when the norm of the projected student feature is less than or equal to the norm of the rotated teacher feature, minimize the Euclidean distance between the two, normalize the Euclidean distance, and record it as the loss Otherwise, the loss To maximize the cosine similarity between the embedding features of the student model and the feature center; S8, will lose Inserted into the existing knowledge distillation framework, the total loss of training the student model is equal to the cross entropy loss plus the knowledge distillation loss and loss balanced by a pair of hyperparameters sum; S9, adjust the pair of hyperparameters to obtain the student model with the highest accuracy on the validation set, and deploy it to the terminal device; the terminal device inputs the received new data into the trained model to obtain the prediction probability vector of each category.

2. The knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: The data enhancement of the training set described in step S2 includes: scaling the image data with maintaining the aspect ratio, random cropping, random horizontal flipping, removing the mean, and adding jitter; the data enhancement of the verification set includes: scaling the image data with maintaining the aspect ratio, center cropping, and removing the mean.

3. The knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: In step S4, for the kth category, the feature center is calculated as follows: in, is the set of all training samples belonging to the kth category.

4. A knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: In step S4, the calculation of the category center is implemented for the entire training set, or the category center is calculated only on the sample set predicted correctly by the teacher model.

5. The knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: In step S5, when the dimension of the embedded features of the student model is the same as that of the embedded features of the teacher model, directly proceed to step S6; when the dimension of the embedded features of the student model is different from that of the teacher model, a fully connected layer with learnable parameters is introduced outside the student model to transform the dimension of the embedded features of the student model to the same as that of the embedded features of the teacher model.

6. The knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: In steps S6 and S7, four alternative methods of feature regularization are constructed. The four alternative methods are divided into two categories: norm regularization and direction regularization.

7. The knowledge distillation method based on embedding feature regularization according to claim 5, characterized in that: The norm regularization includes two types: MSE method and SFN method; the MSE method uses L2 distance to measure the feature difference between the embedded features of the student model and the embedded features of the teacher model; the SFN method is to gradually increase the norm of the embedded features of the student model with a fixed step size during the student model training process, that is, for each training step, the norm of the embedded features of the student model is increased by one step size.

8. The knowledge distillation method based on embedding feature regularization according to claim 5, characterized in that: The directional regularization includes two types: the cosine method and the InfoNCE method; the cosine method maximizes the cosine similarity between the embedded features of the student model and the feature centers described in step S4; the InfoNCE method maximizes the cosine similarity between the embedded features of the student model and the paired feature centers, and at the same time, minimizes the cosine similarity between the embedded features of the student model and the unpaired feature centers.

9. The knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: In step S8, the distillation loss The specific form of depends on the selected distillation framework. When the probability-based distillation framework is selected, the distillation loss The specific form is: where q s and q t They represent the predicted probabilities of the student model and the teacher model for the input samples, respectively, and KL represents the KL divergence.

10. The knowledge distillation method based on embedding feature regularization according to claim 8, characterized in that: In step S8, when the feature-based distillation framework is selected, the distillation loss The specific form is: where f s and f t Represent the intermediate layer features of the student model and the teacher model for the input samples respectively.

11. The knowledge distillation method based on embedding feature regularization according to claim 1, characterized in that: The total loss of training the student model described in step S8 is: in is the cross entropy loss, is the knowledge distillation loss, and its specific form depends on the selected distillation framework. The pair of hyperparameters α and β are used to balance and 12. A knowledge distillation system based on embedding feature regularization, characterized by: include: The data collection module is used by the server to collect annotated image data related to the recognition task and divide it into a training set and a validation set according to the ratio; The data preprocessing module performs different data preprocessing on the training set and the validation set respectively; The model loading module loads the publicly available pre-trained weights for the teacher model and sets its network parameters to frozen mode, i.e., does not update the parameters; the parameters of the student model are randomly initialized; Category center calculation module, used to calculate the feature center of the teacher model in each category; The embedding feature extraction module extracts the embedding features of students and teachers respectively, and performs a linear transformation on the embedding features of students so that they have the same dimension as the embedding features of teachers; The embedding feature projection module projects and rotates the embedding features of students and teachers in the direction of the corresponding category center; Embedding feature regularization module, used to constrain the student's embedding features: increase the feature norm and feature direction to be consistent with the teacher's category center; The student model training module is used to train the student model and select and save the optimal student model weights; The model deployment module is used to deploy the student model to the terminal device. The terminal device will receive new data and input Input it into the trained model to obtain the predicted probability vector, thereby completing the recognition task of the input data.

13. A knowledge distillation device based on embedded feature regularization, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a knowledge distillation method based on embedded feature regularization as described in any one of claims 1-10.

14. A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a knowledge distillation method based on embedding feature regularization as described in any one of claims 1 to 10.

Images