Knowledge distillation method and device based on weighted sorting correlation
By employing a knowledge distillation method based on weighted relevance, the problems of training difficulties and low classification accuracy of SNNs are solved, achieving efficient image classification in resource-constrained scenarios and improving the classification accuracy and robustness of student models.
Patent Information
- Application Number
- CN202510978344.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-16
- Publication Date
- 2025-11-18
AI Technical Summary
Existing spiking neural networks (SNNs) suffer from training difficulties and low classification accuracy in image classification tasks. Traditional knowledge distillation methods fail to fully utilize the temporal processing characteristics of SNNs, resulting in insufficient performance of student models in resource-constrained scenarios.
We employ a knowledge distillation method based on weighted ranking relevance. By designing a weighted relevance loss function that balances inter-class and intra-class ranking preservation, we guide the consistency of the ranking structure output by the student model and the teacher model, thereby enhancing the class relationship modeling ability of the student model.
It achieves high accuracy and low energy consumption for efficient image classification tasks in resource-constrained scenarios. The student model improves robustness and generalization ability to input perturbations while maintaining low power consumption and low latency characteristics.
Smart Images
Figure CN120975178A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and in particular to a knowledge distillation method and device based on weighted ranking correlation. BACKGROUND
[0002] Current image classification tasks mainly rely on deep learning models such as Convolutional Neural Networks (CNN) and the like, and these models have achieved remarkable results in improving image classification accuracy. However, deep learning models usually have problems such as large computational overhead, high energy consumption, and the like, and are difficult to meet the application requirements of low power consumption and real-time response. For this reason, researchers have turned their attention to Spiking Neural Networks (SNN), which is a third-generation neural network that is closer to the biological neural mechanism. Compared to the traditional Artificial Neural Network (ANN) model which generally relies on large-scale network structure and complex training strategy, SNN exhibits natural advantages in energy consumption control and support for spatio-temporal data processing by means of event-driven and sparse computing mechanisms, providing a new direction for solving the above problems. Therefore, SNN is widely considered to have great potential in efficient and low-power image classification tasks.
[0003] However, the training of SNN still faces many challenges. For example, information loss in the process of pulse propagation; gradient propagation difficulty caused by non-differentiable pulse activation function, which makes the overall performance of SNN in image classification and other tasks still lower than that of ANN. There are mainly two methods for training SNN at present: (1) directly training SNN using proxy gradient method; (2) through model conversion, mapping the trained ANN to SNN, thereby realizing the inheritance of the model. Although the latter can inherit the performance of ANN to some extent, this method requires the model structure of ANN and SNN to be consistent, and fails to effectively utilize the structural characteristics of SNN in temporal processing, thereby limiting the full play of the potential of SNN.
[0004] In order to overcome the above limitations, in recent years researchers have attempted to introduce the idea of Knowledge Distillation (KD) to guide the training of the student model by taking the ANN as the teacher model. However, the traditional KD method is mainly based on KL divergence or output probability matching, and fails to fully consider the essential differences between the teacher model and the student model in output distribution, activation characteristics and network structure, resulting in poor distillation effect, and even may inhibit the learning ability of the student model, leading to low accuracy in actual tasks such as image classification.
[0005] Therefore, overcoming the defects of the prior art is an urgent problem to be solved in the technical field. SUMMARY
[0006] In view of the deficiencies of the SNN in the prior art in terms of classification accuracy and training efficiency, the present application provides a knowledge distillation method and device based on weighted ranking correlation. The method guides the student model to maintain the consistency with the output ranking structure of the teacher model to the greatest extent in the training process by designing a weighted correlation loss function that takes into account the ranking maintenance between classes and within classes, thereby enhancing the ability of the student model in class relationship modeling. The present application takes into account the classification accuracy and training efficiency, and is particularly suitable for efficient image classification tasks in resource-constrained scenarios.
[0007] The embodiment of the present application adopts the following technical solutions: In a first aspect, the present application provides a knowledge distillation method based on weighted ranking correlation, specifically: constructing a teacher model of any structure; constructing a student model based on a leaky integral firing neuron; Using the teacher model to predict at least one sample to obtain the class probability distribution of the sample belonging to each class; using the student model to predict at least one sample to obtain the class output representation of the sample belonging to each class; Calculating the correlation loss of the class probability distribution and the class output representation; calculating the classification loss between the class output representation and the sample label; Based on the correlation loss and the classification loss, a total loss is calculated; Using the total loss to update the parameters of the student model; and recalculating the total loss based on the updated student model until the student model converges, obtaining a trained student model.
[0008] Preferably, the correlation loss includes an inter-class correlation loss and an intra-class correlation loss, and the calculation of the correlation loss of the class probability distribution and the class output representation includes: Obtaining the probability distribution T_A of sample A belonging to each class from the class probability distribution; obtaining the probability distribution S_A of sample A belonging to each class from the class output representation; According to the consistency of the class ordering of the probability distribution T_A and the probability distribution S_A, the inter-class correlation loss is calculated; Obtaining the probability distribution T_c of all samples in class c from the class probability distribution; obtaining the probability distribution S_c of all samples in class c from the class output representation; According to the consistency of the distribution structure of the probability distribution T_c and the probability distribution S_c, the intra-class correlation loss is calculated.
[0009] Preferably, the expression of the total loss is:
[0010] wherein, , and represent the hyperparameters balancing the respective losses, represents the classification loss, represents the inter-class correlation loss, represents the intra-class correlation loss.
[0011] Preferably, the expression of the intra-class correlation loss is:
[0012] wherein, represents the intra-class correlation loss, C denotes the total number of classes, represents the probability distribution of all samples in class c in the class output representation, represents the probability distribution of all samples in class c in the class probability distribution, represents the probability distribution and the probability distribution between the structural consistency loss.
[0013] Preferably, the expression of the structural consistency loss is: =1-
[0014] wherein, C denotes the total number of classes, represents the weight factor, represents a probability in the probability distribution , represents a probability in the probability distribution , represents the weighted mean of the probability distribution , represents the weighted mean of the probability distribution .
[0015] Preferably, the weight factor is calculated as follows:
[0016] wherein, denotes a mapping function linearly normalizing the input value to the interval [0, 1], denotes a sorting function returning the rank in ascending order of the input value, The absolute difference in structure between the probability distribution of all samples on the class c in the class output representation and the probability distribution of all samples on the class c in the class probability distribution.
[0017] Preferably, the prediction of the at least one sample by using the student model to obtain the class output representation of the sample belonging to each class comprises: Inputting the at least one sample into the student model and setting a total simulation time step; wherein the student model comprises a plurality of neurons; Updating the membrane potential of a layer of neurons step by step, and firing a pulse of the neuron when the membrane potential is greater than or equal to a firing threshold; Counting the number of pulses of each layer of neurons within a plurality of time steps, normalizing the number of pulses to obtain a pulse rate, and mapping the pulse rate to the class output representation of the sample belonging to each class.
[0018] Preferably, the updating of the parameters of the student model by using the total loss comprises: Taking the class output representation as a differentiable approximation signal; Calculating the gradient of the total loss with respect to the differentiable approximation signal to obtain a parameter change amount; Updating the parameters of the student model based on the parameter change amount.
[0019] In a second aspect, the present application provides a knowledge distillation device based on weighted ranking correlation, specifically comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the knowledge distillation method based on the weighted ranking correlation in the first aspect.
[0020] In a third aspect, the present application further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method provided by the method in the first aspect.
[0021] Compared with the prior art, the present application has the beneficial effects that: (1) supports the use of a pre-trained teacher model with any architecture to guide the training of a student model with different structure and usually lighter, realizes efficient knowledge transfer of a complex teacher model to a student model specially designed for edge devices, so that the finally deployed student model can simultaneously maintain its inherent advantages of event-driven, low power consumption and low latency, thereby meeting the demand for efficient and low-energy intelligent computing in resource-constrained scenarios such as Internet of Things and mobile terminals. (2) The entire knowledge distillation process only optimizes the weight parameters of the student model, without changing the underlying event-driven computing paradigm of the student model. The trained student model completely retains its inherent low power consumption and low latency characteristics during inference. The forward propagation process of online distillation directly uses the inherent multi-time step pulse simulation mechanism of the student model to calculate the soft firing rate representation, without introducing a calculation module that is not a student model characteristic, and without significantly increasing the simulation overhead, thereby ensuring that the core advantages of the student model are fully retained during final deployment, and thus guaranteeing the accuracy of the student model.
[0022] Further, by using the intra-class correlation loss, the student model is explicitly guided to learn the output relationship structure of the teacher model on the same class of samples, prompting the student model to pay attention to and capture the consistency pattern within the class, which helps to enhance the consistency of the learned feature representation of the student model on the same class of samples, and improve the robustness of the student model to input disturbances (such as noise or slight deformation), thereby structurally enhancing the generalization ability of the student model. BRIEF DESCRIPTION OF DRAWINGS
[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0024] Figure 1 is a flowchart of a knowledge distillation method based on weighted ranking correlation provided by an embodiment of the present application; Figure 2 is a flowchart of a method for obtaining class output representation provided by an embodiment of the present application; Figure 3 is a flowchart of a method for calculating correlation loss provided by an embodiment of the present application; Figure 4 is a framework structure diagram of a knowledge distillation method based on weighted correlation provided by an embodiment of the present application; Figure 5 is a structural schematic diagram of a knowledge distillation device based on weighted correlation provided by an embodiment of the present application; Among them, the reference signs are as follows: 21: processor; 22: memory. DETAILED DESCRIPTION
[0025] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.
[0026] Unless otherwise required by context, the term "including" as used in the specification is to be interpreted as "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "an exemplary embodiment", "an example", "specific examples" or "some examples" as used herein are intended to represent any one of the specific features, structures, materials, or characteristics described herein. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials, or characteristics described can be included in any appropriate manner in any one or more embodiments or examples, i.e., although they are carried by the embodiments or examples of the above terms due to the order of appearance and location, they are not limited to being carried by one embodiment or example in a combined manner.
[0027] In the description of the present application, the terms "first", "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise stated, the meaning of "a plurality of" is two or more. In addition, for example, in the description, the same type of nouns can also be described as two independent individuals by adding "A", "B" at the end, in which case the features limited by "A", "B" are only used for the purpose of distinguishing the same type of individual description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated.
[0028] In the description of the present application, the expression "A and / or B" (where A and B represent specific feature content in the form) is used, and the corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.
[0029] In the present application, "about", "approximately", or "approximately" includes the value stated and the average value within an acceptable deviation range of the specific value, wherein the acceptable deviation range is determined by considering the measurement being discussed and the error related to the measurement of the specific quantity (i.e., the limitations of the measurement system) by a person of ordinary skill in the art.
[0030] Furthermore, the technical features involved in each of the embodiments of the application described below can be combined with each other as long as there is no conflict.
[0031] Embodiment 1: As Figure 1 shown, the knowledge distillation method based on weighted ranking correlation provided by the embodiment 1 of the application specifically includes the following steps: Step 101: constructing a teacher model of any structure; constructing a student model based on leaky integrate-and-fire neurons.
[0032] In one embodiment, an ANN of any structure is used as a teacher model as a knowledge provider; and an SNN is constructed based on leaky integrate-and-fire (LIF) neurons, taking the SNN as a student model. The LIF neuron is the core computing unit of the SNN, which generates pulse events through the membrane potential integration and threshold triggering mechanism, thereby realizing the encoding and transmission of information.
[0033] Step 102: using the teacher model to predict at least one sample to obtain the class probability distribution of the sample belonging to each class; using the student model to predict at least one sample to obtain the class output representation of the sample belonging to each class.
[0034] In one embodiment, the teacher model and the student model predict the same sample to obtain two class distributions (class probability distribution and class output representation) of the same sample belonging to each class, wherein the student model calculates the soft firing rate output representation based on the membrane potential through multi-time step simulation, and approximates the soft firing rate output representation to the class output representation of the sample belonging to each class.
[0035] Step 103: calculating the correlation loss of the class probability distribution and the class output representation; calculating the classification loss between the class output representation and the sample label.
[0036] In one embodiment, the correlation loss includes an inter-class correlation loss and an intra-class correlation loss, where the inter-class correlation loss is used to measure the consistency of the ordering of the probability distribution of the same sample belonging to each class output by the teacher model and the student model. For one sample, the teacher model and the student model both output the probability of the sample belonging to each class, and the output class probability distribution has a certain order in arrangement according to the probability value. The inter-class correlation loss is used to measure whether the student model outputs the same probability distribution as the teacher model. The intra-class correlation loss is used to measure the structural consistency of the probability distribution of all samples belonging to the same class output by the teacher model and the student model. For all samples, the teacher model and the student model both output the probability of all samples belonging to the same class. The output of the teacher model is used as a standard, and the intra-class correlation loss is used to measure whether the student model correctly judges the same class sample each time.
[0037] Step 104: calculating a total loss based on the correlation loss and the classification loss.
[0038] In one embodiment, the total loss combines the classification loss, the inter-class correlation loss and the intra-class correlation loss, and the effects of the three losses are balanced by adjustable parameters.
[0039] Step 105: updating the parameters of the student model using the total loss; and recalculating the total loss according to the updated student model until the student model converges, to obtain a trained student model.
[0040] In one embodiment, the teacher model has been trained, and when updating the parameters, the parameters of the teacher model are fixed, only the gradient of the total loss with respect to the parameters of the student model is calculated, and the differentiability of the class output representation is used to approximate the non-differentiability of the impulse event to update the parameters of the student model. The specific method of updating the parameters of the student model using the total loss is: taking the class output representation as a differentiable approximation signal; calculating the gradient of the total loss with respect to the differentiable approximation signal to obtain a parameter change amount; and updating the parameters of the student model based on the parameter change amount. After obtaining the trained student model, multiple image samples are input into the student model for classification prediction, and the output classification accuracy is high.
[0041] In this embodiment, a trained teacher model with any architecture can be used to guide the training of a student model with different structure and usually lighter, so as to efficiently transfer the knowledge of the complex teacher model to the student model specially designed for edge devices, so that the finally deployed student model can simultaneously maintain its inherent advantages of event-driven, low power consumption and low latency, thereby meeting the demand for efficient and low-energy intelligent computing in resource-constrained scenarios such as Internet of Things and mobile terminals.
[0042] As Figure 2As shown, the embodiment provides a method for obtaining a category output representation, which specifically comprises the following steps: Step 201: input at least one sample into the student model, and set a total simulation time step; wherein the student model comprises a plurality of neurons.
[0043] In one embodiment, the total simulation time step is set as N.
[0044] Step 202: update the membrane potential of a layer of neurons step by step, and fire a pulse of the neuron when the membrane potential is greater than or equal to a firing threshold.
[0045] In one embodiment, at each time step 1 to N, the membrane potential of the lth layer of neurons at time step t is calculated by the membrane potential at the previous time step, the pulse at the previous time step, and the pulse at the current time step. , wherein, The expression of is as follows:
[0046] , wherein, represents the membrane potential of the lth layer of neurons at time step t-1, represents the synaptic weight matrix of the lth layer of neurons, represents the pulse of the lth layer of neurons at time step t, represents the pulse of the lth layer of neurons at time step t-1, represents the membrane potential decay factor (0<λ<1) of the LIF neuron, represents the simulation time step length, and l represents the neuron index.
[0047] After obtaining the membrane potential , it is determined whether the membrane potential exceeds the firing threshold to determine whether the neuron fires a pulse, and is expressed as The expression of whether the lth layer of neurons fires a pulse at time step t is as follows:
[0048] , wherein, represents a step function, represents a firing threshold, After the step function processing, the value is 0 or 1, when is 1, it represents that the lth layer of neurons fires a pulse at time step t, and when is 0, it represents that the lth layer of neurons does not fire a pulse at time step t.
[0049] Step 203: Count the number of pulses of each layer neuron in multiple time steps, normalize the number of pulses to obtain a pulse rate, and map the pulse rate to a class output representation of the sample belonging to each class.
[0050] In one embodiment, the number of pulses of each layer neuron in multiple time steps is time-decay weighted average to obtain a pulse rate, and the pulse rate is mapped to a class output representation of the sample belonging to each class, wherein the pulse rate is differentiable for subsequent gradient calculation, and the expression of the class output representation is:
[0051] wherein the class output representation is approximately the dynamic average firing rate of the neuron in the time window , so that the class output representation is aligned with the class probability distribution output by the teacher model, that is At time step n, the value represents the pulse sequence of the same neuron in the simulation time window, is the membrane potential decay factor of the LIF neuron, is the firing threshold, is the simulation time step.
[0052] After obtaining the class probability distribution and the class output representation, loss calculation can be performed, using classification loss to measure the prediction error of the student model for the true label, and using correlation loss to measure the prediction error of the class probability distribution and the class output representation.
[0053] In the traditional knowledge distillation method, KL divergence (English full name: Kullback-Leibler divergence) is usually used to measure the similarity between the probability distribution output by the teacher model and the student model, and the expression of the KL divergence is:
[0054] wherein, and are the class probability distributions output by the teacher model and the student model, respectively.
[0055] However, KL divergence mainly focuses on the absolute value matching of output probabilities, requiring the output probability of the student model to approximate that of the teacher model as closely as possible. However, in the knowledge distillation scenario from ANN to SNN, since ANN uses continuous activation functions (such as softmax) to output probabilities, while SNN uses discrete impulse events (such as firing based on LIF neurons), there is an essential difference in the output distribution of the two. Directly using KL divergence will lead to the following problems: (1) KL divergence forces the matching of probability values without paying attention to the relative order of class predictions, which may make it difficult for the student model to learn the decision boundary of the teacher model; (2) KL divergence treats the prediction error of all classes equally and does not pay special attention to the key classes (classes with higher prediction probabilities) that are more important to the classification task, resulting in insufficient prediction ability of the student model on high-confidence classes; (3) Since the activation mechanisms of ANN and SNN are different, the absolute value matching of probability distributions is not effective in heterogeneous networks, and may even inhibit the learning ability of SNN.
[0056] To address the aforementioned issues, this invention proposes an improved weighted Pearson correlation coefficient. This improved coefficient focuses on the ranking of the model output and assigns greater attention to categories with higher predicted probabilities. Since the number of training samples used to train the student model is large, all training samples can be divided into multiple batches. One batch of training samples is used for category prediction at a time. In this embodiment, the total number of training samples in the current batch is B. This represents the probability distribution of all samples in the current batch belonging to class c, as output by the student model. This represents the probability distribution of all samples in the current batch belonging to class c, as output by the teacher model, and is measured by the improved weighted Pearson correlation coefficient. and The improved Pearson correlation coefficient formula is as follows:
[0057] Where C represents the total number of categories, and c represents one of the categories. Represents probability distribution One of the probabilities, Represents probability distribution One of the probabilities, Represents probability distribution The weighted mean, Represents probability distribution The weighted mean, This represents the weighting factor.
[0058] In one embodiment, by introducing a weighting factor , the loss function can focus on the categories where the teacher model output and the student model output have significant differences in ranking, so that the loss function pays more attention to the consistency of the prediction of key categories, and the weight factor The expression of the weight factor is as follows:
[0059] The weight factor is composed of the teacher ranking, the student ranking, and the product of the difference between the teacher ranking and the student ranking, and is normalized to ensure that the sum of the weights of all categories is 1. represents a mapping function for linearly normalizing the input value to the interval [0, 1], represents a sorting function that returns the ranking of the input value from small to large, is used to measure the absolute difference in structure between the probability distribution of all samples in the category c in the category output representation and the probability distribution of all samples in the category c in the category probability distribution, and the probability distribution or the probability distribution The category with a higher ranking in the probability distribution usually has a larger weight factor.
[0060] In this embodiment, the value range of the weight factor is [-1, 1], when the teacher model output and the student model output are exactly the same, when the teacher model output and the student model output are exactly opposite, a new loss function can be defined to calculate the difference between the student model output and the teacher model output.
[0061] Compared with the KL divergence, the improved weighted Pearson correlation coefficient pays more attention to the matching of the teacher model and the student model output ranking, so that the student model can better learn the decision trend of the teacher model. The dynamic weight mechanism imposes greater punishment on key categories with large ranking errors, improving the prediction ability of the student model in high-confidence categories. By focusing on ranking rather than absolute value, the improved weighted Pearson correlation coefficient is not sensitive to the difference in activation mechanism between the teacher model and the student model, and is suitable for knowledge distillation of heterogeneous networks. In the image classification task, the classification accuracy of the student model is significantly improved, which makes up for the performance loss caused by the distribution mismatch of the traditional method.
[0062] As shown in Figure 3 , the method for calculating the correlation loss provided in embodiment 1 specifically includes the following steps: Step 301: obtaining the probability distribution T_A of sample A belonging to each category from the category probability distribution; and obtaining the probability distribution S_A of sample A belonging to each category from the category output representation.
[0063] In one embodiment, Figure 4 The upper right table represents the probability distribution of the samples output by the teacher model belonging to each category, each column represents the probability of each sample (such as sample A, sample B, sample C, sample D, sample E and sample F) belonging to the category corresponding to the column (such as cat, dog, bird, fish, cow and sheep), and each row represents the probability of a certain sample (such as sample A) belonging to each category.
[0064] Figure 4 The lower right table represents the probability distribution of the samples output by the student model belonging to each category, each column represents the probability of each sample (such as sample A, sample B, sample C, sample D, sample E and sample F) belonging to the category corresponding to the column (such as cat, dog, bird, fish, cow and sheep), and each row represents the probability of a certain sample (such as sample A) belonging to each category. Among them, the probability distribution T_A can be any row of the upper right table; the probability distribution S_A can be any row of the lower right table, but when any row of the upper right table is taken as the probability distribution T_A (such as the first row of the upper right table), the corresponding row of the lower right table must be selected as the probability distribution S_A (such as the first row of the lower right table), and the category can be determined according to the training requirements. The above examples are only for better illustration and do not limit the present application.
[0065] Step 302: According to the category ordering consistency of the probability distribution T_A and the probability distribution S_A, the inter-class correlation loss is calculated.
[0066] In one embodiment, the inter-class correlation loss significantly enhances the punishment for the misordering of key categories (i.e. high prediction probability, high ranking category), which is more conducive to guiding the student model to learn more accurate decision boundaries, thereby overcoming the limitations of traditional knowledge distillation based on KL divergence in this heterogeneous scenario, and laying a technical foundation for the improvement of the final classification accuracy. The expression of the inter-class correlation loss is:
[0067] Among them, represents the intra-class correlation loss, all training sample data is large in amount, and all training samples can be divided into multiple batches, and each batch is trained at a time, B represents the total number of samples in the current batch, represents the probability distribution of sample b belonging to each category in the category output representation, represents the probability distribution of sample b belonging to each category in the category probability distribution, represents the probability distribution and the probability distribution between the ordering consistency loss.
[0068] Step 303: obtaining the probability distribution T c of all samples in the category c from the category probability distribution; obtaining the probability distribution S c of all samples in the category c from the category output representation.
[0069] In one embodiment, the probability distribution T c can be any column of the upper right table, and the probability distribution S c can be any column of the lower right table, but when any column of the upper right table is taken as the probability distribution T c (such as the last column of the upper right table), the corresponding column of the lower right table must be taken as the probability distribution S c (such as the last column of the lower right table).
[0070] Step 304: calculating the intra-class correlation loss according to the distribution consistency of the probability distribution T c and the probability distribution S c.
[0071] In one embodiment, the traditional knowledge distillation method does not have a loss function specially designed to measure the output consistency between intra-class samples, and only relies on the KL divergence or cross-entropy loss. This method only focuses on the output similarity of a single sample and ignores the relative relationship between samples of the same class, resulting in the student model being unable to learn the structured knowledge of the teacher model on intra-class samples. The present application proposes an intra-class correlation loss, which groups the outputs of multiple samples of the same class into a vector to measure the overall consistency, and uses a weighted Pearson correlation coefficient to evaluate the output structural consistency. The intra-class correlation loss explicitly guides the student model to learn the output relationship structure of the teacher model on samples of the same class, which prompts the student model to focus on and capture the consistency patterns within the class, helps to enhance the consistency of the feature representation learned by the student model on samples of the same class, and improves the robustness of the model to input perturbations, thereby structurally enhancing the generalization ability of the model. The expression of the intra-class correlation loss is:
[0072] wherein, represents the intra-class correlation loss, C represents the total number of categories, represents the probability distribution of all samples in the category c in the category output representation, represents the probability distribution of all samples in the category c in the category probability distribution, represents the structural consistency loss between the probability distribution and the probability distribution .
[0073] In one embodiment, it is also necessary to calculate the classification loss between the category output representation and the true label of the sample, and the expression of the classification loss is:
[0074] wherein, B represents the total number of samples in the current batch, and C represents the total number of categories, represents the probability of sample b in category c, represents the encoding vector of the real label of sample b.
[0075] After obtaining the classification loss, the intra-class correlation loss and the inter-class correlation loss, the proportion between the three losses is adjusted by a hyperparameter to obtain a total loss, and the expression of the total loss is:
[0076] wherein, , and represent the hyperparameters for balancing the losses, the hyperparameter adjustment makes the method applicable to different data sets and tasks, overcomes the limitations of the traditional KL divergence method, provides a more efficient training framework for the ANN-SNN knowledge distillation method, and significantly improves the SNN performance, represents the classification loss, represents the inter-class correlation loss, and represents the intra-class correlation loss.
[0077] Since the gradient of the student model cannot be directly calculated, the present application uses a substitute gradient method to calculate the gradient of the total loss with respect to the parameters of the student model, wherein the expression of the gradient change is:
[0078] wherein, represents the parameter change amount, represents the derivative of the total loss, represents the derivative of the category output representation, represents the derivative of the pulse event, represents the parameters of the student model.
[0079] The expression for updating the parameters of the student model is:
[0080] wherein, by the parameter change amount the parameters of the student model are updated, is the parameter of the updated student model.
[0081] In summary, the knowledge distillation method based on weighted ranking correlation has at least the following beneficial effects: (1) supports using a pre-trained teacher model of any architecture to guide the training of a student model with different structure and usually lighter, achieving efficient knowledge transfer of a complex teacher model to a student model designed for edge devices, so that the finally deployed student model can maintain its inherent advantages of event-driven, low power consumption and low latency, thereby meeting the demand for efficient and low-power intelligent computing in resource-constrained scenarios such as Internet of Things and mobile terminals. (2) The entire knowledge distillation process only optimizes the weight parameters of the student model without changing the underlying event-driven computing paradigm of the student model. The trained student model completely retains its inherent low power consumption and low latency characteristics during inference. The forward propagation process of online distillation directly uses the inherent multi-time step pulse simulation mechanism of the student model to calculate the soft firing rate representation, without introducing non-student model specific calculation modules and without significantly increasing the simulation overhead, thereby ensuring that the core advantages of the student model are fully retained in the final deployment, and thus guaranteeing the accuracy of the student model.
[0082] Embodiment 2 The method provided in Embodiment 1 is described in association below, and specific outputs of the student model and the teacher model are provided to better understand the method provided in Embodiment 1.
[0083] As shown in Figure 4 , the present embodiment provides a knowledge distillation method framework diagram based on weighted correlation. The knowledge distillation method is to transfer the knowledge learned by a large and complex teacher model to a smaller and more efficient student model. This process enables the student model to maintain high accuracy while reducing the demand for computing resources. As shown in Figure 4 , there are two models (ANN and SNN), with ANN as the teacher model and SNN as the student model. The teacher model predicts the categories of multiple samples and outputs the category probability distribution (P(y|x)) in the upper right table of Figure 4 . Among them, Forward Pass (forward propagation) represents the processing process of the student model during the inference stage, which is as follows: at each time step , a pulse is generated , which is sent to the multi-layer neuron layer, and is propagated layer by layer. Each layer outputs a pulse response at each time step, and the pulse responses of the neurons in each layer at multiple time steps are counted and integrated into a feature representation. The feature representation is taken as the output feature of the student model (h(x)) in Figure 4The Backward Pass (backward pass) represents the process of calculating the gradient according to the loss function and updating the student model parameters, specifically: calculating the classification loss between the student model output and the true label, calculating the inter-class correlation loss of the student model and the teacher model on the same sample output consistency, calculating the intra-class correlation loss of the student model and the teacher model on the same sample output structure consistency, participating in the gradient calculation of the three kinds of losses, updating the student model parameters, and making the student model learn the consistent or more discriminative features with the teacher model. Among them, the figure and represent an approximate gradient mechanism for training a spiking neural network.
[0084] Embodiment 3: On the basis of the knowledge distillation method based on weighted ranking correlation provided in the foregoing embodiments, the application further provides a device for implementing the knowledge distillation method based on weighted ranking correlation, as shown in Figure 5 The device architecture schematic diagram of the device for the knowledge distillation method based on weighted ranking correlation in the embodiment of the application is shown in the figure. The device for the knowledge distillation method based on weighted ranking correlation in the embodiment of the application comprises one or more processors 21 and a memory 22. Among them, Figure 5 The processor 21 is taken as an example in the foregoing embodiments.
[0085] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 5 The connection through the bus is taken as an example in the foregoing embodiments.
[0086] The memory 22 is a kind of non-volatile computer readable storage medium based on the knowledge distillation method based on weighted ranking correlation, which can be used to store non-volatile software programs and non-volatile computer executable programs, such as the knowledge distillation method based on weighted ranking correlation in the foregoing embodiments. The processor 21 executes the non-volatile software programs, instructions and modules stored in the memory 22, thereby performing various functional applications and data processing of the knowledge distillation device based on weighted ranking correlation, that is, implementing the knowledge distillation method based on weighted ranking correlation in the foregoing embodiments.
[0087] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 can include a memory remotely arranged with respect to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0088] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, perform the knowledge distillation method based on the weighted ranking correlation in the foregoing embodiments, for example, perform the various steps shown in the above description Figures 1-3
[0089] The embodiments of the present application also provide a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are executed by one or more processors, for example, one processor 21 in the computer device 20, so that the one or more processors can perform the knowledge distillation method based on the weighted ranking correlation in the foregoing embodiments, for example, perform the various steps shown in the above description Figure 5 Figures 1-3
[0090] It is worth noting that the information interaction, execution process and the like between the modules and units in the above-described apparatus and system are based on the same concept as the processing method embodiments of the present application, and the specific content can be referred to the description in the method embodiments of the present application, which will not be described here.
[0091] Those skilled in the art can understand that all or part of the steps in the embodiments of the various methods can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0092] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A knowledge distillation method based on weighted ranking relevance, characterized in that, include: Construct teacher models with arbitrary structures; A student model is constructed based on neurons that distribute missing integrals. The teacher model is used to predict at least one sample to obtain the probability distribution of the sample belonging to each category. The student model is used to predict at least one sample to obtain the category output representation of the sample belonging to each category; Calculate the correlation loss between the category probability distribution and the category output representation; Calculate the classification loss between the category output representation and the sample label; The total loss is calculated based on the correlation loss and the classification loss. The parameters of the student model are updated using the total loss; and the total loss is recalculated based on the updated student model until the student model converges, resulting in a trained student model.
2. The knowledge distillation method based on weighted ranking relevance according to claim 1, characterized in that, The correlation loss includes inter-class correlation loss and intra-class correlation loss, and the method further includes: Obtain the probability distribution T_A of sample A belonging to each category from the category probability distribution; obtain the probability distribution S_A of sample A belonging to each category from the category output representation; The inter-class correlation loss is calculated based on the consistency of the class ranking of the probability distribution T_A and the probability distribution S_A. Obtain the probability distribution T_c of all samples in category c from the category probability distribution; obtain the probability distribution S_c of all samples in category c from the category output representation; The intra-class correlation loss is calculated based on the consistency of the distribution structures of the probability distributions T_c and S_c.
3. The knowledge distillation method based on weighted ranking relevance according to claim 2, characterized in that, The expression for the total loss is: in, , and The hyperparameter represents the balance between various losses. Represents classification loss, Represents the inter-class correlation loss. This represents the loss due to intra-class correlation.
4. The knowledge distillation method based on weighted ranking relevance according to claim 2, characterized in that, The expression for the intra-class correlation loss is: in, This represents the intra-class correlation loss, where C represents the total number of classes. The output representation shows the probability distribution of all samples in category c. This represents the probability distribution of all samples in class c within the class probability distribution. Represents probability distribution and probability distribution The loss of structural consistency between them.
5. The knowledge distillation method based on weighted ranking relevance according to claim 4, characterized in that, The expression for the structural consistency loss is: =1- Where C represents the total number of categories, Represents weighting factors. Represents probability distribution One of the probabilities, Represents probability distribution One of the probabilities, Represents probability distribution The weighted mean, Represents probability distribution The weighted average.
6. The knowledge distillation method based on weighted ranking relevance according to claim 5, characterized in that, The weighting factor The calculation formula is as follows: in, This represents a mapping function that linearly normalizes the input values to the interval [0,1]. This represents a sorting function that returns rankings based on the input values in ascending order. It is used to measure the absolute structural difference between the probability distribution of all samples in category c in the category output representation and the probability distribution of all samples in category c in the category probability distribution.
7. The knowledge distillation method based on weighted ranking relevance according to claim 1, characterized in that, The method further includes: At least one sample is input into the student model, and the total simulation time steps are set; wherein, the student model includes multiple neurons; The membrane potential of a layer of neurons is updated step by step at each time step. When the membrane potential is greater than or equal to the firing threshold, the neuron fires a pulse. The number of pulses in each layer of neurons is counted at multiple time steps. The pulse count is normalized to obtain the pulse rate. The pulse rate is then mapped to the category output representation of the sample belonging to each category.
8. The knowledge distillation method based on weighted ranking relevance according to claim 1, characterized in that, The method further includes: The category output is represented as a differentiable approximation signal; Calculate the gradient of the total loss with respect to the differentiable approximation signal to obtain the parameter change; The parameters of the student model are updated based on the changes in the parameters.
9. A knowledge distillation device based on weighted ranking relevance, characterized in that, The device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the knowledge distillation method based on weighted order correlation as described in any one of claims 1-8.
10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions that are executed by one or more processors to perform the knowledge distillation method based on weighted ordering relevance as described in any one of claims 1-8.