Knowledge distillation method and electronic device

Abstract

The embodiments of the present application relate to the technical field of knowledge distillation, and particularly relate to a knowledge distillation method and an electronic device. The knowledge distillation solution is applicable to a heterogeneous model. The method is applied to an electronic device, wherein at least a student model is deployed in the electronic device, the student model comprises K student modules, and K≥1. The method comprises: acquiring first prediction data; inputting sample data into a student model, so as to obtain K first features, and determining K pieces of second prediction data corresponding to the K first features, wherein the K pieces of second prediction data are obtained by means of executing mapping on the K first features, and the second prediction data is used for being aligned with the first prediction data which is output from a teacher model; and determining the value of a loss function, wherein the value of the loss function is calculated on the basis of the K pieces of second prediction data and the first prediction data, the value of the loss function is at least used for updating parameters of at least one student module among K student modules, so as to obtain the student model, and the updated student model is used for executing a prediction task.

Description

Knowledge distillation methods and electronic devices

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 24, 2023, with application number 202311387049.2 and application name “Knowledge Distillation Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of knowledge distillation technology, and in particular to a knowledge distillation method and electronic equipment. Background Art

[0003] Knowledge Distillation (KD) is a classic model compression method. The core idea is to improve the performance of the student model without changing its structure by guiding the lightweight student model to "imitate" the teacher model with better performance and more complex structure.

[0004] Current knowledge distillation methods are mostly designed for models with the same architecture and are difficult to apply to the distillation of heterogeneous models.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a knowledge distillation method and an electronic device to solve the knowledge distillation problem under heterogeneous models and improve the prediction performance of the student model trained under the heterogeneous models.

[0007] In the first aspect, an embodiment of the present application provides a knowledge distillation method, which is at least applied to an electronic device, in which at least a student model is deployed, and the student model includes K student modules, K≥1; the method includes: obtaining first prediction data; the first prediction data is the output data obtained by inputting sample data into the teacher model; inputting sample data into the student model, and outputting K first features through K student modules; determining K second prediction data corresponding to the K first features; the K second prediction data are obtained by mapping the K first features, and the second prediction data are used to align with the first prediction data; determining the value of a loss function, and the value of the loss function is calculated based on the K second prediction data and the first prediction data; the value of the loss function is used to update the parameters of at least one student module in the K student modules to obtain an updated student model, and the updated student model is used to perform the prediction task.

[0008] In this method, the first prediction data output by the teacher model is taken as the target knowledge, the K first features are the K-layer intermediate features output by the K student modules in the student model, and the K second prediction data are obtained by mapping the K-layer intermediate features (i.e., the K first features). The mapping operation can enable the second prediction data obtained based on the intermediate features of the student model to be aligned with the first prediction data, solving the problem that the intermediate features and / or prediction data output by the student model under the heterogeneous model are difficult to align or match with the target knowledge in the teacher model. Then, the value of the loss function can be calculated based on the second prediction data and the first prediction data, and the student model can be trained according to the value of the loss function to realize knowledge distillation under the heterogeneous model. Before this scheme was proposed, the student model could generally only learn from the teacher model of the same category. However, the proposal of this scheme enables the student model to learn from the teacher model of different categories, expands the scope of learning objects, and makes it easier to improve the performance of the student model.

[0009] In one possible implementation, the teacher model and the student model are heterogeneous models.

[0010] In one embodiment, the teacher model includes M teacher modules, M≥1; a teacher module includes at least one layer; the first prediction data is obtained based on the second feature, and the second feature is the output of a teacher module among the M teacher modules.

[0011] A teacher module can contain at least one layer, so this method can use the single-layer features of the teacher model to guide the multi-layer features of the student model, that is, the target knowledge can be specifically located in one layer of the teacher model. Compared with only using the final output features of the teacher model or its corresponding prediction data as the target knowledge, the solution provided by this application is more conducive to finding the optimal feature space for distillation, thereby improving the distillation effect.

[0012] In one embodiment, the teacher model includes a classifier; the classifier includes at least one fully connected layer; the first prediction data is logits, and logits is the output of the last fully connected layer in the classifier.

[0013] In one embodiment, determining the K pieces of second prediction data corresponding to the K first features includes: mapping the K first features output by K student modules to a latent space corresponding to the first prediction data via K feature mapping modules and K classifiers, thereby obtaining the K pieces of second prediction data; wherein the categories of the K feature mapping modules are determined based on the categories of the student model. Through the feature mapping modules, the features output by the student model can be mapped to a latent space that is aligned with the target knowledge output by the teacher model, thereby resolving the problem of misalignment of features and / or prediction data in heterogeneous models.

[0014] In one embodiment, among the K student modules, the i-th feature mapping module corresponding to the i-th student module includes Ki mapping units, 1≤i<K; the K-th feature mapping module corresponding to the K-th student module includes at least 1 mapping unit or 0 mapping units.

[0015] In one embodiment, when the category of the student model is a convolutional neural network (CNN), the mapping unit includes a separable convolution unit.

[0016] In one embodiment, the mapping unit includes a separable convolution unit, a batch normalization (BN) layer, and an activation function.

[0017] In one embodiment, when the student model is a neural network based on an attention mechanism, a mapping unit includes a Transformer-based encoder; the encoder includes a self-attention layer and a feedforward neural network FFN.

[0018] In one embodiment, the loss function includes a target loss term, which is used to characterize the loss corresponding to the target prediction value; the weight in the target loss term is determined based on the first prediction data. The loss function provided in the embodiment of the present application is an adaptive target information enhancement loss function that can adaptively obtain a larger weight on samples with high confidence in the teacher model, and a smaller weight on samples with low confidence in the teacher model, thereby achieving adaptive target information enhancement.

[0019] In one possible implementation, the loss function is obtained by adding a hyperparameter based on the original knowledge distillation loss function; wherein the hyperparameter is used to control the samples with a first prediction value in the first prediction data to obtain a first weight, and the samples with a second prediction value to obtain a second weight, and when the first prediction value is greater than the second prediction value, the first weight is greater than the second weight.

[0020] In one possible implementation, after calculating the value of the loss function, the method further includes: updating the student model based on the value of the loss function to obtain an updated student model.

[0021] In a second aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, wherein the processor is configured to execute a computer program or instruction in a memory to implement any of the methods described above.

[0022] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the method as described in any one of the above items is implemented.

[0023] In a fourth aspect, an embodiment of the present application further provides a computer program product, which includes a program. When the program is executed by an electronic device, the electronic device implements any of the methods described above.

[0024] In a fifth aspect, an embodiment of the present application also provides a chip system, comprising: a communication interface for inputting and / or outputting data; a processor for executing a computer executable program so that a device equipped with the chip system executes any of the methods described above.

[0025] The knowledge distillation method proposed in the embodiment of the present application can be applied to heterogeneous models. The student model can learn from different categories of teacher models with excellent performance, thereby helping to improve the performance of the student model trained under the heterogeneous model, such as improving the accuracy of the student model in performing tasks such as image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is an example diagram of a homogeneous model architecture for calculating loss based on classifier output in the related art;

[0027] FIG2 is an example diagram of a homogeneous model architecture for calculating loss based on intermediate layer features in the related art;

[0028] FIG3 is an example diagram of a homogeneous model architecture for calculating loss based on intermediate layer features and classifier output in the related art;

[0029] FIG4 is another example diagram of a homogeneous model architecture for calculating loss based on intermediate layer features and classifier output in the related art;

[0030] Figure 5(a) is a schematic diagram of the similarity between the middle-layer features under the homogeneous model architecture;

[0031] Figure 5(b) is a schematic diagram of the similarity between the middle-layer features under the heterogeneous model;

[0032] FIG6 is a schematic diagram of an application scenario of the knowledge distillation method provided in an embodiment of the present application;

[0033] FIG7 is a schematic diagram of the knowledge distillation method provided in an embodiment of the present application applied to a video surveillance scenario;

[0034] FIG8 is an example diagram of an image to be classified when the knowledge distillation method provided in an embodiment of the present application is used to perform an image classification task;

[0035] FIG9 is an example diagram of an image of a detection result when the knowledge distillation method provided in an embodiment of the present application is used to perform an object detection task;

[0036] FIG10 is an example diagram of the system architecture of the knowledge distillation method provided in an embodiment of the present application;

[0037] FIG11 is an exemplary diagram of the system architecture in some embodiments of the knowledge distillation method provided in the embodiments of the present application;

[0038] FIG12 is an example diagram of a system architecture in which the knowledge distillation method provided in an embodiment of the present application is applied to a server;

[0039] FIG13 is a flowchart of an embodiment of the knowledge distillation method provided in an embodiment of the present application in a scenario where the method is applied to a terminal device;

[0040] FIG14 is a flowchart of another embodiment of the knowledge distillation method provided in an embodiment of the present application in a scenario where the method is applied to a terminal device;

[0041] FIG15 is an exemplary diagram of a system architecture in one embodiment of a knowledge distillation method provided in an embodiment of the present application;

[0042] FIG16 is a schematic diagram of the internal structure hierarchy of a feature mapping module in one embodiment of the knowledge distillation method provided by an embodiment of the present application;

[0043] FIG17 is an exemplary diagram of a system architecture in another embodiment of the knowledge distillation method provided by an embodiment of the present application;

[0044] FIG18 is a schematic diagram of the internal structure hierarchy of a feature mapping module in another embodiment of the knowledge distillation method provided by an embodiment of the present application;

[0045] Figures 19(a) and 19(b) are schematic diagrams of the experimental evaluation results of different heterogeneous models on the industry's public evaluation dataset ImageNet;

[0046] FIG20 is a schematic structural diagram of an example of an electronic device. DETAILED DESCRIPTION

[0047] In order to better understand the technical solutions of this specification, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0048] It should be clear that the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this specification.

[0049] The terms used in the examples of this application are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a," "an," "the," and "the" used in the examples of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0050] With the advancement of artificial intelligence (AI), the number of parameters in artificial neural network (ANN) models continues to increase, and performance is also constantly improving. For example, neural network models based on deep learning can have hundreds or even thousands of layers, and the number of parameters also increases exponentially. This is accompanied by a continuous increase in model inference overhead, such as the consumption of computing and storage resources required to execute tasks. To address this issue, research in this field has begun on model compression methods—compressing models while minimizing the impact on model performance to reduce inference overhead. Model compression is also driven by the fact that, generally, training models requires a large amount of training data and computing resources to extract knowledge. However, large models are difficult to deploy in real-world application scenarios, such as mobile phones. This is because large models have slow inference speeds and require high device resources. Therefore, it is necessary to compress large trained models to reduce their size while ensuring inference performance.

[0051] Currently, mainstream model compression methods can be divided into three categories: quantization, pruning, and knowledge distillation. Knowledge distillation is a training method based on the "teacher-student network" concept. Its purpose is to transfer the knowledge learned by a large model (teacher model) or an ensemble of multiple models to another lightweight model (student model). The knowledge contained in the trained large model is distilled into the other lightweight model. The teacher model is generally a larger pre-trained model, that is, a pre-trained model with expected performance and a large number of layers and parameters. The student model is generally a smaller new model, that is, a lightweight model to be trained. During training, the supervisory signal from the teacher model is used as "knowledge" to guide the training of the student model, so that the student model can imitate the teacher model, so that the student model can achieve the same or similar prediction performance as the teacher model.

[0052] Depending on the type of "knowledge" used, knowledge distillation methods can include two types: knowledge distillation based on predicted data (Response-based) and knowledge distillation based on features. The Response-based algorithm uses the predicted data of the teacher model (such as classification results) as the target knowledge (i.e., the learning object). Among them, feature-based knowledge distillation can include Feature-based and Relation-based algorithms. Among them, the Feature-based algorithm uses the intermediate layer features generated by the feature extractor of the teacher model as the target knowledge; the Relation-based algorithm also performs knowledge distillation based on model features, but the target knowledge it uses is not the one-to-one difference between corresponding features, but the difference in feature relationships.

[0053] As shown in Figures 1 and 2, taking the image classification task as an example, there are two types of knowledge distillation methods: a method that uses the output of the classifier as knowledge and a method that uses the intermediate layer features of the model as knowledge. Specifically, as shown in Figure 1, the method of using the output of the classifier as knowledge first inputs the sample into the teacher model to obtain the output of the teacher model classifier. Subsequently, the same sample is input into the student model to obtain the output of the student model classifier. Next, the difference between the output of the teacher model classifier and the output of the student model classifier is evaluated, for example, using an appropriate distance measurement algorithm or using a knowledge distillation loss function, etc., to evaluate the difference between the output of the teacher model and the student model, calculate the gradient of the difference value with respect to the student model parameters, obtain the update direction of the student model parameters, and complete one iteration in the training process. Among them, the distance measurement algorithm can be one or more distance measurement algorithms such as Euclidean distance and Manhattan distance.

[0054] The main idea behind distillation methods that use model intermediate-layer features as knowledge is similar to methods that use classifier outputs as knowledge. The difference is that the former uses the model's intermediate-layer output features, rather than the final classifier output, as the source of knowledge. Therefore, feature distillation is often used to refer to methods that use model intermediate-layer features as knowledge. As shown in Figure 2, the teacher model includes three teacher modules: Teacher Module 1, Teacher Module 2, and Teacher Module 3, and the student model includes three student modules: Student Module 1, Student Module 2, and Student Module 3. The intermediate-layer feature loss is calculated by evaluating the difference between the intermediate-layer features output by each of the three modules. Sometimes, feature distillation methods also use knowledge from the model's classifier output to further improve the performance of the student model.

[0055] As shown in Figure 3, in the related art, a self-distillation technology based on knowledge distillation is also proposed. The student model and the teacher model are the same model, and the two are trained together. The scheme includes the following steps: (1) Input the sample into the model to obtain the output features of each layer in the middle of the model and the final classifier output; (2) Send the output features of each layer in the middle of the model to the branch module, which is composed of a feature extractor and a classifier to obtain the output features and classifier output of each branch; (3) Use the final classifier and the features before the classifier as supervision signals, calculate the loss function between the output features and classifier output of each branch, and use the backpropagation algorithm to update the model parameters; (4) After the training is completed, the auxiliary branches are discarded, and only the model trunk is retained as the trained model, that is, the student model and the teacher model are the same model, and the two are trained together. In this scheme, the self-distillation method does not use a teacher model with higher performance, does not realize the guiding role of the teacher model on the student model, and limits the further improvement of the final performance of the student network (i.e., the student model).

[0056] Most current knowledge distillation methods are designed for scenarios where the teacher and student models share the same architecture. These solutions achieve good performance in homogeneous model scenarios. However, when the teacher and student models have different architectures, successful alignment of the learned features cannot be guaranteed because the features extracted by the heterogeneous models reside in different latent feature spaces. For example, in the network architecture shown in Figure 2 for knowledge distillation based on intermediate-layer features, the teacher and student models are neural network models of the same category. The intermediate-layer features output by teacher module 1 can be aligned with those output by student module 1, and the loss can be directly calculated between them. Specifically, the intermediate-layer features output by student module 1 are learned from the intermediate-layer features output by teacher module 1. Similarly, the intermediate-layer features output by teacher module 2 and student module 2 can be aligned, and the intermediate-layer features output by teacher module 3 and student module 3 can be aligned. Feature loss is calculated based on the intermediate-layer features output by each of the three modules to train the student model. However, this solution only works for homogeneous models. If the student model and teacher model are heterogeneous, directly matching these unrelated features is not very effective.

[0057] For another example, as shown in Figure 4, another knowledge distillation scheme based on an isomorphic model uses both classifier output and intermediate layer features as knowledge. It mainly includes the following process: (1) Input the samples into the teacher model and the student model respectively to obtain the output features of the teacher model (the output of the first layer before the classifier) ​​and the final classifier output, as well as the output features of the intermediate layers of the student model and the final classifier output. The output features of the intermediate layers are the output features of each student module; (2) The output features of the final classifier and the output features before the classifier of the teacher model are used as supervision signals, and the loss function is calculated between the output features of each branch of the student model and the classifier output, and the model parameters are updated using the backpropagation algorithm; (3) After training is completed, the auxiliary branches in the student model are discarded, and only the student model trunk is retained as the trained model. The output features of each branch of the student model can be directly the output features of the intermediate layers of the student model, or the features of each branch obtained by feature extraction of the output features of the intermediate layers of the student model.

[0058] The solution shown in Figure 4, on the one hand, selects features at fixed positions in the teacher model (the output of the last layer) as supervision signals, limiting the possible optimal feature region. This is because the teacher model is generally larger than the student model, and the features most suitable for the student model to learn may appear in the teacher model's intermediate layers. On the other hand, this solution is also designed for distillation between models with the same architecture. Both the teacher model and the student model are based on convolutional neural networks. The learning method that matches the output features of the teacher model with the output features of the intermediate layers of the student model is not suitable for distillation of heterogeneous models. Applying knowledge distillation methods designed based on homogeneous model architectures to heterogeneous models may even hinder the performance of the student model.

[0059] Currently, most mainstream distillation technology solutions target networks with the same architecture, ignoring knowledge distillation across different architectures. This results in some newly proposed models lacking a high-performance teacher model with the same architecture as their learning model. Once a novel network is proposed, such as one based on a multi-layer perceptron (MLP), it may be difficult to find a high-performance teacher network (i.e., teacher model) with the same architecture. Forcibly using a teacher model with a different architecture and distillation techniques designed for the same model can result in the student model's final downstream task accuracy being too low.

[0060] Furthermore, in heterogeneous models, the intermediate-level features output by various modules in the teacher and student models are often dissimilar. Because the features extracted by heterogeneous models reside in different latent feature spaces, successful alignment of the learned features cannot be guaranteed. For example, Figure 5(a) shows the feature similarity matrix extracted from different layers of a homogeneous model, and Figure 5(b) shows the feature similarity matrix extracted from different layers of a heterogeneous model. The brightness of the pixels in the image represents the similarity between the features, with higher brightness indicating higher similarity. Figure 5(a) shows that the similarity between the intermediate-level features of the teacher and student models in the homogeneous model is higher, especially at the diagonal positions, where the brightness reaches a peak. In contrast, Figure 5(b) shows that the similarity between the intermediate-level features of the teacher and student models in the heterogeneous model is generally lower and shows no discernible pattern. Directly matching these unrelated features in heterogeneous models will likely hinder the performance of the student model.

[0061] In addition, most of the current distillation loss functions do not consider sample characteristics. That is, the loss function used for distillation training does not consider the characteristics of the training samples. It simply uniformly makes the output of the student network (i.e., the student model) fit the output of the teacher network, resulting in a larger number of simple samples dominating the training, which is not conducive to the recognition accuracy of downstream tasks.

[0062] In view of this, embodiments of the present application provide a knowledge distillation method that focuses on the knowledge distillation task of heterogeneous models and explores how to efficiently utilize features extracted from models with different architectures to assist in distillation. Different architecture models, i.e., heterogeneous models, refer to models where the teacher model and the student model are of different categories. Specifically, this method proposes a distillation paradigm that uses any single-layer or multi-layer features of the teacher model to guide the multi-layer features of the student model, and also proposes an adaptive target information enhancement loss function.

[0063] In some embodiments, by aligning the features extracted by the heterogeneous models in a latent space relevant to the final task (e.g., the latent space relevant to the final task can be the space where the predicted data is located without softmax processing), the student model can ignore task-irrelevant information and only learn knowledge directly related to the final task from the teacher model. The latent space refers to the space where the compressed (encoded) representation (i.e., feature vector) of the original input data is located.

[0064] It should be noted that the teacher model can include multiple teacher modules, and a teacher module has one or more layers. Therefore, the features output by a teacher module are single-layer features or multi-layer features. Since a teacher module can contain only one neural network layer, the learning accuracy of this scheme can be accurate to any single layer in the teacher model, and is not limited to the features of the model's final output, thereby increasing the scope of target knowledge that can be learned.

[0065] In addition, in the same embodiment or other embodiments, an adaptive target information enhancement loss function can be used to further reduce the interference of irrelevant information in the heterogeneous teacher model training on the student model training process.

[0066] The method proposed in the embodiment of the present application or the product obtained based on the method can be applied to any scenario where a neural network model needs to be deployed as an inference tool, wherein each type of scenario involves a neural network model, and these neural network models can be improved in performance using the method proposed in the embodiment of the present application to obtain higher prediction accuracy, improve the model's ability to model images, and other capabilities. In addition, it should be noted that the method provided in the embodiment of the present application, i.e., the general model knowledge distillation method for model compression, can be applied to computer vision tasks, as well as other fields or application scenarios, such as natural language processing and speech recognition.

[0067] For example, it can be applied to various application scenarios that require image classification, image recognition, target detection, semantic segmentation and other tasks. For example, it can be specifically applied to various application scenarios such as autonomous driving, security monitoring, medical image analysis, image search, etc. As shown in Figure 6, the student model trained based on the method proposed in the embodiment of the present application can be deployed in a server or terminal device. The terminal device can be a computer or a mobile phone, etc., to perform at least one of the corresponding image classification, target detection, semantic segmentation and other tasks. It should be noted that the teacher model is CNN and the student model is ViT is only an example. The teacher model or the student model can also be other neural network models.

[0068] The server may be one or more of a file server, a web server, a mail server, a database server, an application server, a game server, a DNS server, a proxy server, a media server, and the like.

[0069] Terminal equipment (TE) can be a smart phone, a personal computer (PC), a tablet PC, a computer with wireless transceiver function, a wearable electronic device (such as a smart watch, a bracelet, glasses, a collar), a robotic arm, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, a robot, a subscriber unit, an intelligent vehicle terminal, a cellular phone, a wireless data card, a wireless modem, a handheld device (handset), a laptop computer, a machine type communication (MTC) terminal, a smart home device, a vehicle (such as an electric car), a drone, etc., or it can also be an intelligent monitoring terminal device that can be installed on a means of transportation such as a helicopter, an airplane, a ship, a high-speed rail, etc. Among them, a smart home can be a smart speaker, a television, a smart screen, a set-top box, a smart toy, a learning machine, etc.

[0070] For example, as shown in FIG7 , in an intelligent monitoring application scenario, the student model obtained based on the knowledge distillation method of the embodiment of the present application is deployed in the server of the intelligent monitoring system. In video monitoring, video is obtained through image acquisition devices such as camera A, camera B, and camera C. Attribute recognition of target entities such as pedestrians (or vehicles) can be a key task, and it is necessary to identify common attributes of pedestrians, such as gender, age, hair, clothing, backpacks, etc. Knowledge distillation can improve the recognition accuracy of the student model and achieve more accurate recognition to provide better services. It should be noted that the “network” in the cloud-shaped graphic in FIG7 refers to a communication network, that is, a wired network or a wireless network, not a neural network. Unless otherwise specified, “network” elsewhere in this specification refers to a neural network.

[0071] For another example, in scenarios such as image search, the lightweight student model obtained based on the method proposed in the embodiment of the present application can be deployed in an intelligent terminal to perform image classification tasks, such as classifying the image shown in Figure 8 into images of mobile phones with cameras and images of mobile phones without cameras. Alternatively, as shown in Figure 9, in an autonomous driving application scenario, the lightweight student model can be deployed in an on-board intelligent terminal to perform target detection tasks and detect targets in the image (the detected targets are represented by boxes in Figure 9).

[0072] In this application specification, an isomorphic model refers to a knowledge distillation architecture composed of a teacher model and a student model when the teacher model and the student model belong to the same category. For example, the teacher model belongs to a convolutional neural network and the student model also belongs to a convolutional neural network. The knowledge distillation architecture composed of the teacher model and the student model is an isomorphic model; a heterogeneous model refers to a knowledge distillation architecture composed of a teacher model and a student model when the teacher model and the student model belong to different categories. For example, the teacher model is a convolutional neural network (CNN) and the student model is a vision transformer (ViT). The knowledge distillation architecture composed of the teacher model and the student model is a heterogeneous model. The method provided in the embodiment of the present application can be applied to multiple categories of neural network models. In the embodiment of the present application, a neural network that performs a convolution operation is classified as the first category, that is, a convolution operation is performed in the neural network, and the neural network is considered to belong to the first category; a neural network that adopts an attention mechanism is classified as the second category; and a network containing only a fully connected layer is classified as the third category. It can be understood that the methods provided in the embodiments of this application are applicable to at least the following three categories: convolution-based neural network models, hereinafter referred to as CNNs; attention-based neural network models, hereinafter referred to as Transformers; and neural network models containing only fully connected layers, hereinafter referred to as MLPs. MLPs are multilayer perceptrons.

[0073] It should be noted that two models of the same category can have different structures. For example, ResNet and ConvNeXt, although these two models differ in architectural details, are regarded as models of the same architecture in this application, rather than heterogeneous models, and can both be classified as CNNs.

[0074] For example, in some embodiments, the following models are classified as CNNs:

[0075] CNN, AlexNet (ImageNet Classification with Deep Convolutional Neural Networks)), VGG (Visual Geometry Group), GoogLeNet (Inception), ResNet (Deep Residual Network), ConvNeXt (ConvNet model for the 2020s), DenseNet (Densely connected convolutional networks), SENet (Squeeze-and-Excitation Networks), ResNeXt (suggesting the next dimension), FCN (Fully Convolutional Neural Network, full convolutional neural network), UNet, SegNet (Semantic Segmentation Net, semantic segmentation network), DeepLab, FastRCNN (Fast Region-based Convolutional Network, fast region-based convolutional neural network), RCNN, etc.;

[0076] The following models are classified as Transformers:

[0077] Transformer, ViTVision Transformer, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-Trained Transformer, generative pre-trained Transformer model);

[0078] The following models can be classified as MLPs if they only contain fully connected layers:

[0079] MLP, FCNN (Fully Connected Neural Network), DNN (Deep Neural Network), MLP-Mixer.

[0080] The above only uses the three categories of CNN, Transformer, and MLP as examples for illustrative explanation. The method provided in the embodiments of the present application can also be applied to other neural networks other than these three categories. For neural networks that do not belong to any of the above three categories, they can be divided into many other different categories. For example, in some embodiments, the neural network model used to process graph structures is summarized as GNN (Graph Neural Network), such as GCN (Graph Convolution Network), GAT (Graph Attention Network), GAE (Graph Autoencoders), GraphSAGE (Graph SAmple and aggreGatE), DeepWalk, TransE (Translation-based Model), etc. can be summarized as GNN category. It should be noted that in other embodiments, GCN can be summarized as CNN category, and GAT can be summarized as Transformer category.

[0081] For example, models based on diffusion mechanisms such as stable diffusion, DALL-E (Dali+WALL-E), and GLIDE (Generative Latent Introduction and Diffusion) can be classified into the diffusion category, and so on.

[0082] It should be noted that the teacher model or student model may contain models from two or more categories. In this case, as long as the student model contains models from a different category than the teacher model, the knowledge distillation architecture composed of the teacher and student models is considered a heterogeneous model. For example, the CoAtNet model combines convolution and attention mechanisms. If the teacher model is CoAtNet and the student model belongs to the CNN or Transformer category, the knowledge distillation architecture formed by the teacher and student models is considered a heterogeneous model.

[0083] It should be noted that the method provided in the embodiments of this application is designed for heterogeneous models, but it does not exclude that the method can also be applied to knowledge distillation of homogeneous models. The following examples are mainly based on heterogeneous models.

[0084] Specifically, an embodiment of the present application proposes a knowledge distillation method, which can be executed based on the system architecture example shown in Figure 10. The student model includes K student modules, where K ≥ 1 and K is an integer. Each student module can include at least one layer of a neural network. The layer here refers to the layer that constitutes the neural network, such as a convolutional layer, a pooling layer, a fully connected layer, etc. The teacher model includes M teacher modules, where M ≥ 1 and is an integer. Each teacher module can include at least one layer of a neural network. M can be equal to or different from K.

[0085] The knowledge distillation method provided in the embodiments of the present application can be applied to a first electronic device, which can be, for example, a server or a terminal device. Deploying at least a student model in the first electronic device can specifically include the following steps:

[0086] S10, obtaining first prediction data.

[0087] The first prediction data is the output data obtained by inputting the sample data into the teacher model.

[0088] For example, in some embodiments, in an application scenario where the teacher model is deployed on a server and the student model is deployed on a terminal device, obtaining the first prediction data may involve the terminal device sending a request to the server, and the server sending the first prediction data to the terminal device in response to the request. In other embodiments, in an application scenario where both the teacher model and the student model are deployed on a server, obtaining the first prediction data may involve obtaining the first prediction data from the output data of the teacher model.

[0089] S11, input sample data to the student model, output K first features through K student modules; determine K second prediction data corresponding to the K first features.

[0090] It should be noted that the sample data input to the teacher model is the same as the sample data input to the student model.

[0091] The first feature is the intermediate layer feature output by the student model, for example, the first feature 1 output by the student module 1, that is, the intermediate layer feature output by the student module 1.

[0092] K second prediction data are obtained by performing mapping on the K first features, and the second prediction data are used to align with the first prediction data output by the teacher model.

[0093] It should be noted that, in the embodiments of the present application, the predicted data is a feature expression directly related to the final task, and is not the final output result, but a feature expression that has not been normalized, that is, the space where the predicted data is located is the latent space (or feature space) before normalization processing such as softmax. Specifically, the predicted data can be a vector or a matrix, and the predicted data includes predicted values. For example, in an image classification task scenario, the predicted data can be a classification result, and the classification result includes predicted values ​​for each category, including the target category. For example, in a video surveillance scenario, it is necessary to classify the targets in the video surveillance image and identify female targets between the ages of 30 and 40. Then the target category is women between the ages of 30 and 40. The classification result will include the predicted value of the target category predicted by the intermediate layer features output by the teacher module or the student module for the people appearing in the current image. In the target detection task scenario, the predicted data is the target detection result, for example, it can be the coordinates, that is, the coordinates of each pixel point of the bounding box of the detected target.

[0094] The final output result after normalization processing such as softmax is sometimes a vector based on one-hot encoding, which loses a lot of feature information of non-target categories. Therefore, in an embodiment of the present application, the intermediate layer features output by the student model are mapped to a latent space related to the final task but without normalization processing.

[0095] To avoid confusion, in the embodiment of the present application, the feature output by the student module is defined as the first feature, and the feature output by the teacher module is defined as the second feature.

[0096] The first prediction data is obtained based on the second feature, and the second feature is the output of one of the M teacher modules. That is, the first prediction data can be obtained based on the output feature of any teacher module in the teacher model. As shown in FIG10 , the first prediction data can be the prediction data obtained from the output feature of the K-th teacher module, or the prediction data obtained from the output feature of any other teacher module. For example, the first prediction data is obtained based on the output feature of teacher module 1 or teacher module 2. Since a teacher module can contain at least one neural network layer, this method can use the prediction data corresponding to the feature output of any layer in the teacher model as the target knowledge for learning to guide the extraction of multi-layer features of the student model, so as to obtain prediction data that is closer to the teacher model, thereby improving the prediction accuracy of the student model.

[0097] It should be noted that in the embodiments of the present application, alignment has two levels of meaning. One level refers to the consistency of the second prediction data obtained by mapping with the first prediction data in the dimension of feature expression. Generally speaking, the prediction data is a vector or a matrix, and alignment means that the dimensions of the vector or matrix are consistent. For example, the feature map of a convolutional neural network can have three dimensions: height height, width width, and channel channel. The alignment operation for the feature map, that is, the three dimensions of height, width, and channel are consistent. That is to say, through mapping, the mapped feature map is consistent with the feature map in the target knowledge in the three dimensions of height, width, and channel. It should be noted that the feature map is only used to illustrate the meaning of alignment. In some embodiments of the present application, only the prediction data needs to be aligned, and the feature map does not need to be aligned. The meaning of another level is that through mapping, the value of the loss function calculated based on the second prediction data and the first prediction data has a reference meaning, because the loss function value calculation of the two unaligned prediction data has no reference meaning. It can be understood that the role of alignment is at least to enable the first prediction data to be used as a soft-target for learning the second prediction data. In addition to calculating the loss function value, a distance measurement algorithm may be used to evaluate the difference between the second predicted data and the second predicted receipt.

[0098] S12, determining the value of the loss function.

[0099] The value of the loss function can be calculated based on the K second prediction data and the first prediction data. In addition, the value of the loss function is used to update the parameters of at least one student module among the K student modules to obtain an updated student model.

[0100] The updated student model is used to perform prediction tasks. Prediction tasks include one or more tasks such as image classification, semantic segmentation, and object detection. For example, the updated student model is used to perform image classification or object detection tasks.

[0101] In some embodiments, determining the value of the loss function may involve calculating the value of the loss function using the K pieces of second predicted data and the first predicted data as independent variables. In other embodiments, determining the value of the loss function may involve receiving the value of the loss function from another device. For example, the loss function is calculated in a server, and the terminal device receives the value of the loss function from the server.

[0102] Optionally, in some embodiments, the first electronic device is a server, and both the teacher model and the student model are deployed in the server. After S12, the following operations can be performed: based on the value of the loss function, the student model is updated to obtain an updated student model.

[0103] Alternatively, in other embodiments, the first electronic device is a terminal device, the second electronic device is a server, the student model is deployed in the terminal device, and the value of the loss function can be calculated in the server, that is, after the value of the loss function is calculated in the server, the value of the loss function is sent to the terminal device (first electronic device), the first electronic device receives the value of the loss function, and based on the value of the loss function, the trainable parameters in the student model are updated. That is, after S12, the following operation can be performed: the terminal device (first electronic device) receives the value of the loss function sent by the server (second electronic device).

[0104] Optionally, S11-S12 are iteratively executed until the loss function converges to obtain a trained student model. Afterwards, the student model can be used to perform a prediction task, and the data to be predicted is input into the student model. The student model is used to perform predictions to obtain predicted data. For example, the student model is used to perform an image classification task. The image data of Figure 8 can be used as the data to be predicted and input into the student model. The student model is used to classify the image, for example, performing a binary classification on the image to identify the front (screen side) image of the phone and the back image of the phone.

[0105] In some embodiments, the loss function in S12 is the adaptive target information enhancement loss function proposed in the embodiment of the present application. The design of this loss function can further reduce the interference of irrelevant information from the heterogeneous teacher model training on the student model training process, which will be elaborated in detail in the subsequent content.

[0106] Heterogeneous models generally tend to extract feature representations, which hinders the feature distillation process. In order to align these different features, the embodiment of the present application designs a distillation paradigm that uses any single module feature of the teacher model to guide the multi-module features of the student model. Specifically, the intermediate layer features of the student model and the teacher's features are mapped to a latent space and aligned. At the same time, in order to increase the stability of the learning process, in some embodiments, only the features of a single layer of the teacher model can be used to match the multi-layer features of the student model.

[0107] For example, as shown in Figure 10, the output generated by teacher module K can be selected as the feature of the teacher model, the output generated by teacher module 2 can be selected as the feature of the teacher model, or the feature output by teacher module 1 can be selected as the feature of the teacher model. There is no limitation here.

[0108] For example, in some embodiments, the knowledge distillation method provided by the embodiments of the present application can be performed based on the system architecture example shown in Figure 11. In these embodiments, the teacher model includes a classifier; the classifier includes at least one fully connected layer.

[0109] In the embodiment of Figure 11, taking the image classification task as an example, the latent space directly related to the task objective (image classification), namely the logits space, is selected to implement the distillation process. In other words, the second predicted data or the first predicted data is the logits, and the logits are the output of the final fully connected layer, that is, the output of the last fully connected layer of the classifier.

[0110] As shown in FIG11 , based on the K first features output by the K student modules, K second prediction data are obtained in the following manner: the K first features output by the K student modules (not shown in the figure) are mapped through K feature mapping modules to obtain K intermediate layer features, and then the K intermediate layer features are input into K classifiers, which output K logits (i.e., Logits1, Logits2, ..., Logits K shown in FIG11 ). This process is equivalent to mapping the K first features to the latent space corresponding to the first prediction data to obtain K second prediction data. Among them, the first prediction data is the output of the last fully connected layer in the classifier in the teacher model, that is, the logits output by the teacher model, and the latent space corresponding to the first prediction data is the space where the logits are located. The space or feature space here can be understood as the feature vector or feature matrix obtained after compression, encoding, or feature extraction. The position in the model can be, for example, a certain layer in the model, or the spatial position between layers in the model, or the spatial position occupied by the output of a certain layer. In some embodiments, the first feature is mapped to the latent space corresponding to the first prediction data, that is, the first feature is mapped to the space corresponding to the output of the last fully connected layer of the teacher model classifier.

[0111] It should be noted that the method proposed in the embodiments of the present application is designed for heterogeneous models, and the design of the feature mapping module can refer to the student model. Specifically, the category of the K feature mapping modules is determined based on the category of the student model. In some embodiments, if the student model belongs to the CNN category, the feature mapping module should be designed as a mapping module based on convolution operations; if the student model adopts an attention mechanism and belongs to the Transformer category, the feature mapping module should be designed as a mapping module that adopts the attention mechanism.

[0112] For example, when the student model is a CNN model, a separable convolution can be used to construct the student model's feature mapping module to perform the mapping from the student model's original feature space to the latent space. When the student model is a ViT or MLP model, a Transformer unit can be used to construct the student model's feature mapping module.

[0113] In some embodiments, the feature mapping module can be specifically designed in the following manner:

[0114] Among the K student modules, the i-th feature mapping module corresponding to the i-th student module includes Ki mapping units, 1≤i<K; the K-th feature mapping module corresponding to the K-th student module includes at least 1 mapping unit or 0 mapping units.

[0115] For example, assuming K=5, feature mapping module 1 corresponding to student module 1 includes 4 mapping units, feature mapping module 2 corresponding to student module 2 includes 3 mapping units, and so on. Feature mapping module 3 and feature mapping module 4 include 2 mapping units and 1 mapping unit, respectively. Feature mapping module 5 includes 1 mapping unit.

[0116] In some embodiments, a classifier is connected at the end of each feature mapping module to further map the features output by the student module into logits. The output of the classifier corresponding to each student module in the student model can directly learn the output of the classifier corresponding to the selected teacher module features.

[0117] In other embodiments, it is not excluded that the number of mapping units in the K-th feature mapping module is 0, that is, it does not include a mapping unit, but directly inputs the first feature output by the student module K into the classifier.

[0118] When the student model is a convolutional neural network (CNN), in some embodiments, the mapping unit includes a separable convolution unit. In other embodiments, the mapping unit may include a traditional convolution unit, that is, a non-separable convolution unit. Separable convolution units can reduce the number of convolution operations and are therefore more suitable for deployment in lightweight student models.

[0119] The separable convolution unit can adopt depth separation or spatial separation. In some implementations, the separable convolution unit is a unit that adopts depth separation.

[0120] Exemplarily, the convolution-based mapping unit includes a separable convolution unit, a BN (Batch-Normalization) layer, and an activation function. This hierarchical structure design is only an example. In other embodiments, other layers can be appropriately added or reduced based on the separable convolution unit, such as adding a pooling layer, or replacing the batch normalization layer with other layers with normalization functions.

[0121] BN layer, that is, batch sample normalization layer.

[0122] In some embodiments, the activation function is ReLU (Rectified Linear Unit), which can effectively solve the gradient vanishing problem while maintaining the computing speed, thereby making training more efficient.

[0123] In the same embodiment or other embodiments, the activation function can be selected to use one or more activation functions such as sigmoid function, tanh (hyperbolic tangent) function, ReLU, P-ReLU, R-ReLU, Leaky-ReLU function, ELU (Exponential Linear Units) function, Maxout function, etc.

[0124] If the student model is a neural network based on an attention mechanism, then in some embodiments, the mapping unit may include a Transformer-based encoder; the encoder may include a self-attention layer and a feed-forward network (FFN). In other embodiments, the encoder may be an encoder based on a multi-head attention mechanism.

[0125] It should be noted that the feature mapping module and the classifier can be combined into a single module or provided separately. For example, as shown in FIG11 , in some embodiments, the classifier can be provided independently of the feature mapping module, with each feature mapping module being connected to a classifier. In other embodiments, the feature mapping module can include the classifier, i.e., the classifier can be provided as part of the feature mapping module.

[0126] It should be noted that under the homogeneous model architecture, the loss function loss can include the loss corresponding to the intermediate layer features and the loss corresponding to the predicted data. When the method provided in the embodiment of the present application is applied to a heterogeneous model, the loss function calculates the loss value based on the second predicted data output by the teacher model and the first predicted data output by the student model.

[0127] The knowledge distillation method proposed in the present embodiment uses a loss function including a target loss term. The target loss term is used to represent the loss corresponding to the target prediction value, which is a loss related only to the task objective. The target prediction value can be the prediction value of the target category. The weight in the target loss term is determined based on the first prediction data output by the teacher model.

[0128] For example, in some embodiments, an adaptive target information enhancement loss function is used to calculate the loss value. The adaptive target information enhancement loss function can be a function that introduces an additional loss that is only related to the target category on the basis of the original knowledge distillation loss. For the sake of convenience of description, the loss that is only related to the target category is defined as the target loss item.

[0129] The target loss term is weighted accordingly by the prediction results (i.e., predicted data) output by the teacher model. It can adaptively give greater weight to samples with high teacher model prediction values ​​and smaller weight to samples with low teacher model prediction values, thus achieving adaptive target information enhancement. The prediction value can be a confidence level or a coordinate value, for example.

[0130] Specifically, the adaptive target information enhancement loss function can be obtained by adding a controllable hyperparameter to the original knowledge distillation loss function. The hyperparameter is a hyperparameter used to control the information enhancement of the target category.

[0131] Specifically, by adding hyperparameters, the following objectives can be achieved: samples with a first prediction value in the first prediction data are controlled to receive a first weight, and samples with a second prediction value are controlled to receive a second weight. When the first prediction value is greater than the second prediction value, the first weight is greater than the second weight. For example, it is possible to adaptively give larger weights to samples with high confidence in the teacher model, while giving smaller weights to samples with low confidence in the teacher model, thereby achieving adaptive target information enhancement.

[0132] The description of the above embodiments mostly uses image classification tasks as examples, and the approach of using latent space is also applicable to other types of tasks. For example, in the semantic segmentation task in vision, in addition to distillation in the feature space or classification space, the features of any layer can also be mapped to a latent space of any dimension through a certain transformation for distillation, that is, it can be mapped to other latent spaces other than logits for distillation. For example, in the translation task of speech semantics, in addition to distillation in the original text feature space, the features of any part of the network can also be mapped to a predetermined latent space for distillation. It should be noted that any space in the model (such as a teacher model or a student model) can be specified as a predetermined latent space, not limited to logits. For example, the predetermined latent space can also be the space between modules, or the space between layers in a module, or a layer of a module.

[0133] In actual application scenarios, illustratively, the electronic device may be a server or a terminal device. That is, the method provided in the embodiments of the present application may be run on a server and / or a terminal device.

[0134] In one application scenario, as shown in FIG12 , the method can be applied to a server, and the teacher model and the student model can be deployed on the same server, or the student model and the teacher model can be deployed on different servers. At least the student model is deployed on one server. In this application scenario, the execution subject of the method provided in the embodiment of the present application can be a server. In combination with the system architecture shown in FIG12 , the method can specifically include the following steps:

[0135] S20, determine sample data, input the sample data into the teacher model; obtain first prediction data through the teacher model.

[0136] S21, input the same sample data into the student model, and obtain K first features corresponding to K student modules through the student model.

[0137] S22, mapping is performed on the K first features through a feature mapping module, the mapped features are input into corresponding K classifiers, and K second prediction data are output.

[0138] S23, calculating the loss function value corresponding to the first prediction data and each piece of the K second prediction data.

[0139] Specifically, the first prediction data and the i-th second prediction data are input as independent variables into a predetermined adaptive target information enhancement loss function, and the loss value corresponding to the i-th second prediction data is calculated. The loss value is the value of the adaptive target information enhancement loss function.

[0140] S24, enhances the value of the loss function according to the adaptive target information and updates the parameters in the student model.

[0141] For example, according to the loss value corresponding to the i-th second prediction data, the parameters in the i-th student module are updated. In some embodiments, the trainable parameters in the i-th feature mapping module are also updated, and the initial values ​​of the trainable parameters in the i-th feature mapping module can be randomly initialized.

[0142] It should be noted that the student model obtained by the knowledge distillation method provided in the embodiments of this application can be installed on a terminal device as a software program product or computer program product to perform the corresponding prediction task. For example, the software program product or computer program product can be one or more of a program installation package, an upgrade package, a functional plug-in, etc.

[0143] In another application scenario, the teacher model is deployed on the server side and the student model is deployed on the terminal device. In this scenario, the student model can be trained through remote online knowledge distillation. In this application scenario, as shown in Figure 13, the method provided in the embodiment of the present application can be applied to the terminal device, that is, the execution subject can be the terminal device, and specifically may include the following steps:

[0144] S30: Obtain first prediction data.

[0145] The terminal may proactively send a request to the server, and the server may respond to the request by sending the first prediction data to the terminal device. Alternatively, the server may proactively send the first prediction data to the terminal device. The first prediction data is the output data obtained by the teacher model after the sample data is input into the teacher model.

[0146] In some embodiments, the terminal device may also receive sample data information or directly receive sample data. The sample data information may be information used to specify the sample data. For example, if the sample data is an ImageNet dataset, the sample data information may be "ImageNet" or other characters or signals used to represent the semantics.

[0147] In other embodiments, the terminal device may obtain the sample data from other channels, such as sending a request to a third-party server or platform to obtain the sample data.

[0148] S31, input the sample data into the student model, and obtain K first features corresponding to K student modules through the student model;

[0149] S32, mapping the K first features through a feature mapping module, inputting the mapped intermediate layer features into corresponding K classifiers, and outputting K second prediction data;

[0150] S33, respectively calculating the loss function value corresponding to the first predicted data and each piece of the K second predicted data;

[0151] S34, enhancing the value of the loss function according to the adaptive target information, and updating the parameters (trainable parameters) of at least one student module in the student model.

[0152] It should be noted that, in the application scenario where the teacher model is deployed on the server and the student model is deployed on the terminal device, the process shown in FIG13 is only one feasible embodiment. In other embodiments, the method may also execute the process shown in FIG14:

[0153] S40: The terminal device receives first prediction data.

[0154] Exemplarily, the server may send the first prediction data to the terminal device in response to a request from the terminal device, and the terminal device receives the first prediction data.

[0155] S41, the terminal device inputs the sample data into the student model, and obtains K first features corresponding to K student modules through the student model.

[0156] S42, the terminal device performs mapping on the K first features through the feature mapping module, inputs the mapped intermediate layer features into the corresponding K classifiers, and outputs K second prediction data.

[0157] S43, the terminal device sends K pieces of second prediction data to the server.

[0158] S44, respectively calculating the loss function value corresponding to the first prediction data and each piece of the K pieces of second prediction data.

[0159] S45, the terminal device receives the value of the loss function.

[0160] For example, the server may send the first prediction data and the loss function value corresponding to each of the K pieces of second prediction data to the terminal device.

[0161] S46, updating the parameters of at least one student module in the student model according to the value of the loss function.

[0162] The following describes a specific example, combining the above application scenarios. In this example, the task being performed is image classification. The teacher model is the ViT model, and the student model is ConvNeXt. Both the teacher model and the student model are deployed on a server. As shown in Figure 15 , the teacher model is divided into three teacher modules, and the student model is divided into three student modules.

[0163] In this embodiment, feature mapping module 1 includes two mapping units, feature mapping module 2 includes one mapping unit, and feature mapping module 3 includes one mapping unit.

[0164] The student model belongs to the CNN category, so the internal hierarchical structure design of the mapping unit is shown in Figure 16, including a separable convolution unit, a BN layer, and a ReLu activation function layer.

[0165] In this embodiment, logits 4 obtained from the output features of the teacher module 1 is used as the soft-target (i.e., target knowledge), and the three first features output by the three student modules are respectively input into the feature mapping module, and then mapped into logits (i.e., logits1-logits3) through their respective corresponding classifiers; logits 4 corresponding to the feature output by the teacher module 1 (i.e., the second feature) is used as a label, and the loss function values ​​corresponding to logits1-logits3 are calculated respectively. For example, logits 1 and logits 1 are used as independent variables, and logits 4 and logits 1 are input into the adaptive target information enhancement loss function. The obtained value is the loss value corresponding to logits 1. Based on this loss value, the trainable parameters in the student module 1 can be updated by algorithms such as back propagation. Optionally, the trainable parameters in the feature mapping module 1 and the classifier 1 can also be updated together. In this way, based on the loss values ​​corresponding to logits 2 and logits 3, the trainable parameters in the student module 2 and the student module 3 can be updated. After multiple iterations, the student model imitates the teacher model, transfers the knowledge of the teacher model to the student model, and obtains a student model with performance close to that of the teacher model.

[0166] Optionally, in this embodiment, when the output features of the teacher module 1 are selected as the target knowledge, a classifier 4 dedicated to the teacher module 1 can be trained, and a feature mapping module 4 can be added between the teacher module 1 and the classifier 4 to map the features output by the teacher module 1 into compressed features suitable for classification. The category of the feature mapping module 4 is consistent with the category of the teacher model.

[0167] It should be noted that in other embodiments, the feature mapping module 4 may not be set between the teacher module 1 and the classifier 4. For some large models, such as large models with a depth of thousands of layers, the teacher module 1 may include dozens or even hundreds of layers. The student model may only need to learn the teacher module 1 to obtain its expected performance. In some cases, the features output by the teacher module 1 can be directly input into the classifier to obtain logits4 without the need for feature mapping.

[0168] In this embodiment, an adaptive target information enhancement loss function is used. Based on the traditional knowledge distillation loss function, an additional controllable parameter γ is introduced, allowing the distillation loss function to adaptively enhance the information strength of the target category. This parameter is used to control the degree of information enhancement of the target category.

[0169] The original knowledge distillation loss function is:

[0170] Adding a controllable parameter γ to the exponential term changes the original loss function to:

[0171] in, represents a labeled dataset, where is the sample set, is a label set; p t represents the predicted probability value of the teacher model, that is, the probability value that the current target belongs to the target category; p s Represents the predicted probability value of the student model, that is, the probability value that the current target belongs to the target category; represents the target category; γ is a controllable parameter, which is a hyperparameter, that is, a parameter that does not need to be changed during training, and is used to achieve information enhancement of the target category.

[0172] In order to analyze the performance of the improved loss function, the OFA loss function is expanded into two terms:

[0173] It can be seen that the improved loss function can be regarded as an additional loss (target loss term) that is only related to the target category on the basis of the original knowledge distillation loss. This loss is weighted accordingly by the teacher model prediction data, and can adaptively obtain a larger weight on samples with high confidence in the teacher model, and a smaller weight on samples with low confidence in the teacher model, thereby achieving adaptive target information enhancement.

[0174] It should be noted that, in other embodiments, for different original knowledge distillation functions, the adaptive target information enhancement loss function can be obtained in the following manner:

[0175] Add the above L to the original knowledge distillation loss function OFA , or just increase the target loss term. The target loss term is:

[0176] Let's take another specific example. In this example, the task being performed is object detection. The teacher model is the FastRCNN model, and the student model is ViT. The student model is deployed on the terminal device, and the teacher model is deployed on the server. As shown in Figure 17, the teacher model is divided into three teacher modules, and the student model is divided into four student modules.

[0177] In this embodiment, feature mapping module 1 includes three mapping units, feature mapping module 2 includes two mapping units, and feature mapping module 3 and feature mapping module 4 each include one mapping unit.

[0178] In this embodiment, the feature mapping module and the classifier are combined into one module, that is, the feature mapping module includes the classifier.

[0179] For example, taking feature mapping module 1 as an example, it includes 3 mapping units and 1 classifier, and the mapping units are connected in series in sequence, wherein one mapping unit includes a self-attention unit and an FFN network.

[0180] In this embodiment, the logtis 5 corresponding to the output feature of the teacher module 2 is selected as the soft-target. A classifier 5 dedicated to the teacher module 2 can be trained, and a feature mapping module 5 can be set between the classifiers 5, or it can not be set. Whether to set the feature mapping module 5 is specifically determined by the relative size or architecture between the teacher model and the student model. Similarly, the classifier 5 and the feature mapping module 5 can be set separately and independently, or they can be combined into one module, that is, the classifier 5 can be merged into the feature mapping module 5.

[0181] In this embodiment, assuming that the original knowledge distillation function is Loss, the loss function used in this embodiment can be Loss+L OFA , or Loss+Lc, Lc represents the target loss term.

[0182] Based on the methods described in S10 and S11 above, we conducted experimental tests on the ImageNet dataset, a publicly available industry evaluation dataset, using knowledge distillation for heterogeneous models in image classification tasks as an example. The results, shown in Table 1 below, show that the student models trained using the method proposed in this embodiment achieve higher accuracy than existing knowledge distillation methods across a variety of teacher / student combinations with different architectures.

[0183] As shown in Table 1 below, Table 1 shows OFA (i.e., using L OFA Comparison results of this application scheme of the function) and other knowledge distillation methods on the ImageNet dataset.

[0184] Table 1 shows the ImageNet-1K dataset, under the heterogeneous model, using the original knowledge distillation loss function KD (i.e., L KD Function) as the loss function method, as well as the test result data using the FitNet+OFA-KD scheme, FitNet+OFA-KD, that is, based on the FitNet model, OFA-KD is used as the loss function, OFA-KD is L OFA Function. The best result is in bold and the second best result is underlined.

[0185] Table 1

[0186] By comparison, it can be obtained that, based on the method provided in the embodiment of the present application, the use of an adaptive target information enhancement loss function can achieve the following beneficial effects: as shown in Figures 19(a) and 19(b), relevant experimental evaluations are carried out on the industry's public evaluation dataset ImageNet. When the teacher / student are ViT / CNN and CNN / ViT combinations respectively, the adaptive target information enhancement loss function proposed in the embodiment of the present application can bring significant accuracy improvement to the student network compared to the original loss function (γ=0).

[0187] In Figure 19(a), T represents the teacher model, using DeiT-T as the teacher model; S represents the student model, using ResNet18 as the student model. γvalue represents the value of γ. Accuracy represents the accuracy. In Figure 19(b), the teacher model is ResNet50, and the student model is DeiT-T.

[0188] The method provided in the embodiment of the present application utilizes any single-layer features of the teacher model to guide the distillation paradigm of the multi-layer features of the student model. Knowledge distillation in the latent space can eliminate the interference of irrelevant information of the heterogeneous teacher model; and the use of an adaptive target information enhancement loss function can adaptively enhance the supervision information related to the task goal and reduce the interference of irrelevant information.

[0189] An embodiment of the present application further provides an electronic device, comprising: a processor, wherein the processor is configured to execute a computer program or instruction in a memory to implement the method described in any of the above embodiments.

[0190] Specifically, the electronic device may be a server or a server cluster. Referring to FIG20 , the hardware structure of the electronic device may include: a processor, a memory, and a chipset.

[0191] Processor, the core that handles all computing tasks and requests, including at least the CPU.

[0192] Storage: This includes memory and hard disk. The larger the capacity, the more requests and files can be processed simultaneously.

[0193] Chipset: Includes the northbridge chip and the southbridge chip, which connect various I / O devices. The northbridge chip connects to the CPU, memory, and PCI bus, while the southbridge chip connects to hard drives, network devices, and other I / O devices.

[0194] I / O bus: Connects various I / O devices, such as network cards and USBs. The greater the server's I / O bus bandwidth, the more requests it can handle. I / O devices: These include network cards and USBs. Network cards are the primary device connecting the server to the external network, and their speed and performance directly impact the server's response and transmission speeds.

[0195] Exemplarily, a processor may include one or more processing units, for example, a neural-network processing unit (NPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a baseband processor, etc. The different processing units may be independent devices or integrated into one or more processors. The controller may generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0196] The memory can be used to store computer executable program code, which includes instructions. The internal memory may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, at least one application required for a function, etc. The data storage area can store data (such as input data, output data) created during the use of the electronic device. In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory and / or instructions stored in a memory provided in the processor.

[0197] Alternatively, the electronic device may be a terminal device with certain computing capabilities, such as a mobile phone, a personal computer (PC) (e.g., a laptop), a tablet PC, a wearable electronic device, etc. The terminal device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, etc.

[0198] It should be understood that the structure illustrated in the embodiment of the present invention is merely an example and does not limit the electronic device. The electronic device in the embodiment of the present application may include more or fewer components than shown, or combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0199] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored program, wherein when the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0200] An embodiment of the present application further provides a computer program product, which includes a program. When the program is executed by an electronic device, the electronic device implements the method described in any of the above embodiments.

[0201] An embodiment of the present application also provides a chip system, including: a communication interface for inputting and / or outputting data; a processor for executing a computer executable program, so that a device equipped with the chip system executes a method as described in any of the above embodiments.

[0202] The above-mentioned computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0203] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0204] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0205] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0206] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0207] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

Claims

1. A knowledge distillation method, characterized in that: The method is applied to an electronic device, in which at least a student model is deployed, and the student model includes K student modules, where K≥1; The method comprises: Obtaining first prediction data; the first prediction data is output data obtained by inputting sample data into a teacher model; Input the sample data into the student model, and output K first features through the K student modules; Determine K pieces of second prediction data corresponding to the K pieces of first features; the K pieces of second prediction data are obtained by performing mapping on the K pieces of first features, and the second prediction data are used to align with the first prediction data; Determine the value of the loss function; the value of the loss function is calculated based on the K second prediction data and the first prediction data; the value of the loss function, the value of the loss function is at least used to update the parameters of at least one student module among the K student modules to obtain an updated student model, and the updated student model is used to perform the prediction task.

2. The method according to claim 1, characterized in that The teacher model and the student model are heterogeneous models.

3. The method according to claim 1 or 2, characterized in that The teacher model includes M teacher modules, M≥1; a teacher module includes at least one layer; The first prediction data is obtained based on a second feature, and the second feature is the output of a teacher module among the M teacher modules.

4. The method according to any one of claims 1 to 3, characterized in that The teacher model includes a classifier; the classifier includes at least one fully connected layer; The first prediction data is logits, which is the output of the last fully connected layer in the classifier.

5. The method according to any one of claims 2 to 4, characterized in that: Determining K pieces of second prediction data corresponding to the K pieces of first features includes: Mapping the K first features to the latent space corresponding to the first prediction data through K feature mapping modules and K classifiers to obtain K second prediction data; Wherein, the categories of the K feature mapping modules are determined based on the category of the student model.

6. The method according to claim 5, characterized in that Among the K student modules, the i-th feature mapping module corresponding to the i-th student module includes Ki mapping units, 1≤i<K; the K-th feature mapping module corresponding to the K-th student module includes at least 1 mapping unit or includes 0 mapping units.

7. The method according to claim 6, characterized in that In the case where the category of the student model is a convolutional neural network (CNN), the mapping unit includes a separable convolution unit.

8. The method according to claim 7, characterized in that The mapping unit includes a separable convolution unit, a batch normalization (BN) layer, and an activation function.

9. The method according to claim 6, characterized in that In the case where the category of the student model is a neural network based on an attention mechanism, a mapping unit includes an encoder based on a Transformer; the encoder includes a self-attention layer and a feedforward neural network FFN.

10. The method according to any one of claims 1 to 9, characterized in that The loss function includes a target loss term, which is used to characterize the loss corresponding to the target prediction value, and the weight in the target loss term is determined based on the first prediction data.

11. The method according to claim 10, characterized in that The loss function is obtained by adding hyperparameters based on the original knowledge distillation loss function; wherein, the hyperparameters are used to control the samples with the first prediction value in the first prediction data to obtain the first weight, and the samples with the second prediction value to obtain the second weight, and when the first prediction value is greater than the second prediction value, the first weight is greater than the second weight.

12. The method according to any one of claims 1 to 11, characterized in that After determining the value of the loss function, the method further includes: Based on the value of the loss function, the student model is updated to obtain an updated student model.

13. An electronic device, characterized in that: The electronic device comprises: A processor, wherein the processor is configured to execute a computer program or instruction in a memory to implement the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a stored program, wherein the program implements the method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, characterized in that The program product includes a program, and when the program is executed by an electronic device, the electronic device implements the method according to any one of claims 1 to 12.

16. A chip system, characterized in that: include: A communication interface for inputting and / or outputting data; A processor, configured to execute a computer executable program so that a device equipped with the chip system executes a method as described in any one of claims 1 to 12.

Images