Hyperspectral Ground Object Classification Method Based on Lightweight Multi-Domain Joint Distillation

Abstract

The present invention discloses a hyperspectral ground object classification method based on lightweight multi-domain joint distillation, which mainly solves the problems of complex network, low operation efficiency and poor generality in the existing methods. The solution is as follows: perform normalization and spectral dimension expansion processing on the hyperspectral image, and divide the training set and the test set; construct a lightweight network model composed of point convolutions, and randomly initialize N teacher models and one student model; train one teacher model with each hyperspectral image separately; calculate the feature similarity loss between the teacher model and the student model and the classification loss of the student model on the training set, and add the two to obtain the total loss; use the total loss to train the student model; input the test set into the trained student model to obtain the classification result, and output the classification map. The present invention uses the same model to achieve efficient classification of multi-domain data sets, simplifies the network structure, improves the classification efficiency and classification accuracy, and can be used for hyperspectral geological detection, ground object recognition and ground object tracking.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a hyperspectral ground object classification method, which can be used for hyperspectral ground object recognition, geological detection, and ground object tracking. Background Art

[0002] Hyperspectral images usually have hundreds of spectral channels and the unique property of one object one spectrum, and are widely used in fields such as land use property classification, military surveillance, and object recognition. In recent years, with the continuous development and wide application of hyperspectral image acquisition devices, the demand for hyperspectral image processing has been continuously increasing.

[0003] Traditional hyperspectral classification algorithms are limited by the professional domain knowledge required for manual feature design and have relatively high research barriers. Although deep learning algorithms have solved the problem of the professionalism of manual feature design, in the application process of hyperspectral image classification tasks, they have problems such as a large number of model parameters and computational complexity, and low operating efficiency. At the same time, due to the diversity of hyperspectral image acquisition devices and the differences in the acquisition image scenes, the data distributions between different scenes are different and cannot adapt to each other. Therefore, most of the existing deep learning-based hyperspectral image classification algorithms need to re-initialize and retrain the model when processing different data sets. However, this conflicts with the actual situation that it is difficult to adjust the model on on-orbit devices and satellite devices. Therefore, how to achieve good classification for different scene data sets in the same model is an urgent problem to be solved in the practical application of current hyperspectral image classification algorithms.

[0004] In recent research work, lightweight processing of the network has become an important research direction for improving model efficiency. A lightweight network refers to improving or redesigned certain structures in a convolutional neural network to improve the network operation efficiency and achieve better network performance than traditional networks with fewer parameters. By lightweight processing of the network, the network operation efficiency is improved and the network classification performance is improved.

[0005] In the hyperspectral image HSI ground object classification task, the superiority of the lightweight network has also been confirmed.

[0006] The paper "Zhang H, Li Y, Jiang Y, et al. Hyperspectral classification based on lightweight 3-D-CNN with transfer learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8): 5813-5828." proposed an end-to-end 3D lightweight convolutional network 3-DLWNet, which is composed of lightweight units constructed by stacking point convolution, pooling operation and 3D depthwise separable convolution. Compared with the traditional 3D-CNN model, this network has a deeper network structure, fewer parameters and lower computational cost. However, due to the use of a large number of 3D convolutional layers, the running efficiency of this method is not good and the performance improvement is weak.

[0007] The lightweight multi-scale attention fusion network LMAFN proposed in the paper "Wang J, Huang R, Guo S, et al. NAS-guided lightweight multi-scale attention fusion network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(10): 8754-8767.", based on the results of neural architecture search, adaptively adjusts the weights of different channels by combining multi-scale Ghost modules and efficient channel attention ECA modules, achieving fewer parameters and lower computational cost.

[0008] The LiteDepthwiseNet, a lightweight network for hyperspectral image classification, proposed in the paper "Cui B, Dong X M, Zhan Q, et al. LiteDepthwiseNet: A lightweight network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-15.", decomposes the standard 3D convolution into 3D depthwise convolution and 3D point convolution, and deletes the ReLU layer and batch normalization BN layer in the original 3D depthwise convolution, which can achieve high performance with fewer parameters and improve the overfitting phenomenon of the model on a small dataset.

[0009] However, since the above-mentioned LMAFN and LiteDepthwiseNet networks both use carefully designed modules, which greatly rely on the rich professional experience of researchers themselves, it will lead to high design thresholds and poor generality.

[0010] In summary, the existing lightweight hyperspectral ground object classification methods have problems such as complex networks, low operating efficiency, and poor generality. Summary of the Invention

[0011] The purpose of the present invention is to propose a lightweight multi-domain joint distillation hyperspectral ground object classification method for the deficiencies of the above-mentioned existing technologies, so as to simplify the network structure, improve the classification efficiency and accuracy, and expand the generality.

[0012] The technical idea to achieve the purpose of the present invention is: by making full use of point convolution to extract spectral information, reducing the number of model parameters, simplifying the network structure, and improving the network operation efficiency; by using knowledge distillation to distill the knowledge contained in multiple teacher models into the same student model, reducing the storage cost of the compressed model, and achieving a classification effect comparable to or even better than that of the teacher model.

[0013] According to the above idea, the implementation solution of the present invention includes the following:

[0014] S1) Normalize and expand the spectral dimension of N hyperspectral images in sequence to obtain N hyperspectral images with equal dimensions, and then divide the data of each hyperspectral image into a training set and a test set, obtaining a total of N training sets and N test sets;

[0015] S2) Construct a lightweight network composed of a cascade of multiple point convolution layers, a pooling layer, and a fully connected layer, and create and randomly initialize N teacher models and a student model according to its network structure, and ensure that the teacher models and the student model have the same network structure;

[0016] S3) Use the N training sets to independently train the N teacher network models respectively using the gradient descent algorithm. During the training iteration process, calculate the accuracy of the training set for each iteration, and use the weights of the network model of the generation with the highest accuracy on the training set as the final classification model weights to obtain N trained teacher models;

[0017] S4) Perform feature-based knowledge distillation on the same student model using the N trained teacher models to obtain a trained student model:

[0018] S41) Randomly sample the same number of training samples from the N training sets respectively, and input them into the corresponding trained teacher models respectively to obtain the output features of the teacher models;

[0019] S42) Input all the sampled training samples into the student model to obtain the output features and prediction probabilities of the student model;

[0020] S43) Calculate the similarity between the output features of the student model and the output features of the teacher model, and use it as the similarity loss MLoss; calculate the classification loss CLoss between the prediction probability of the student model and the true class label using cross-entropy loss, and obtain the total loss Loss = MLoss + CLoss;

[0021] S44) Use the total loss Loss to iteratively update the student model, and take the weights of the network model with the lowest total loss Loss as the final weights of the student model to obtain the trained student model;

[0022] S5) Input N test sets into the trained student model for testing to obtain the final classification results, and output the classification maps of N hyperspectral image data according to the classification results.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects:

[0024] 1) Based on the characteristics of hyperspectral images with many spectral dimensions and one spectrum for one object, the present invention uses a multi-layer point convolution lightweight model to extract spectral information in hyperspectral images, which can better perform spectral dimension information fusion. At the same time, due to the small number of parameters of point convolution, the classification network can be effectively simplified and the model complexity can be reduced.

[0025] 2) Since the present invention uses multiple teacher models to perform feature-based knowledge distillation on the same student model, the student model can learn relatively accurate multi-domain dataset classification features from multiple teacher models, and the obtained student model can achieve good classification performance on multiple datasets without multiple trainings, reducing the model parameters for different data.

[0026] 3) Since the present invention performs normalization processing on the original hyperspectral image, it not only avoids the incomparability caused by numerical differences, but also retains the overall continuous information and change characteristics of the spectral curve, and can reduce the complexity in the model inference process and improve the operation efficiency without affecting the classification accuracy.

[0027] 4) Since the present invention uses the similarity loss based on feature knowledge distillation and the cross-entropy loss of student model classification to jointly guide the optimization of the student model, it ensures that the student model can not only learn accurate multi-domain dataset classification features, but also perform accurate label prediction;

[0028] 5) The present invention only involves the basic knowledge of point convolution and knowledge distillation to achieve effective model lightweight design, has low requirements for professional domain knowledge, and does not require a complex module design process. It is applicable to tasks related to hyperspectral classification, is easy to reproduce, and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is the implementation process of the present invention;

[0030] Figure 2 is a schematic diagram of the network model constructed in the present invention;

[0031] Figure 3 is the result graph obtained by classifying ground objects for the three-domain data in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The following will clearly and completely describe the embodiments and effects of the present invention with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0033] Refer to Figure 1 , the implementation steps of this example are as follows:

[0034] Step 1, data preprocessing.

[0035] Since the spectra of the data in hyperspectral images have different value ranges, which will interfere with model inference, and at the same time, the number of channels of different hyperspectral images is different, which will cause the inability to use the same model for inference calculation. Therefore, it is necessary to preprocess the hyperspectral image data first, and the specific implementation is as follows:

[0036] 1.1) Obtain hyperspectral image data from a public website, and sequentially perform normalization processing on the obtained N hyperspectral images, so that the hyperspectral data x is mapped between [0, 1], and the normalized hyperspectral image is obtained where μ and σ are the mean and variance of the hyperspectral image data x, respectively;

[0037] 1.2) Perform zero-padding on the N normalized hyperspectral images in the channel dimension, so that the N normalized hyperspectral images have the same number of channels, and the number of channels is the maximum value of the original number of channels of the N hyperspectral images, obtaining the hyperspectral image after dimension expansion, so that different hyperspectral image data can be processed by the same model structure to ensure subsequent knowledge distillation.

[0038] Step 2, data sampling and partitioning.

[0039] The original hyperspectral image is a data containing all ground objects. It is necessary to sample it first to obtain the data and its labels related to each pixel point before inputting it into the model for inference operations. The implementation is as follows:

[0040] 2.1) Set the sampling window of the hyperspectral image to the spatial size of the patch block to sample samples containing spectral and spatial dimension information from N hyperspectral images after dimension expansion;

[0041] 2.2) Use the label of the central pixel point of the patch block as the label of the patch block, and randomly divide all labeled patch blocks in N hyperspectral images into a training sample set and a test sample set where and represent the training sample set and the test sample set of the i-th hyperspectral image respectively.

[0042] In this example, a patch block of size 7×7 is used for sampling, and 5 labeled samples of each class are taken as the training sample set, and the remaining labeled samples are used as the test sample set for hyperspectral image classification.

[0043] Step 3, construct a network model.

[0044] Refer to Figure 2 and the specific implementation of this step is as follows:

[0045] 3.1) Construct a lightweight network composed of multiple point convolutional layers, a pooling layer, and a fully connected layer in cascade. Each point convolution includes multiple convolutional kernels, the size of the convolutional kernels is 1×1, the stride is 1, the padding is 0, and the number of convolutional kernels gradually increases with the depth of the point convolutional layer; this multi-layer point convolution lightweight model is used to extract the spectral information in the hyperspectral image to better perform spectral dimension information fusion. At the same time, by using point convolutions with fewer parameters, the classification network is effectively simplified and the model complexity is reduced;

[0046] 3.2) According to the structure of the lightweight network in step 3.1), create and randomly initialize N teacher models and a student model, and ensure that the teacher models and the student model have the same network structure.

[0047] Step 4, train the teacher model.

[0048] 4.1) Let N teacher models classify one of the N hyperspectral images respectively, and randomly sample the same number of training samples from the N hyperspectral images respectively:

[0049] 4.2) Input the training samples into the corresponding teacher model to obtain the predicted class of the teacher model And calculate its cross-entropy loss with the true ground object category Y:

[0050]

[0051] Where CLoss Ti is the loss of the i-th teacher model, N x is the training sample size, C is the total number of categories, and are the true label and predicted label of the k-th training sample of the i-th hyperspectral image with category c, respectively;

[0052] 4.3) Take the losses CLoss T1 , CLoss T2 ,..., CLoss Ti ,..., CLoss TN of all teacher models and take the derivative of their model weights to obtain the gradients of all teacher model weights:

[0053]

[0054] Where represents the gradient of the loss of the i-th teacher model with respect to the model weights;

[0055] 4.4) Perform weight iterative update according to the gradients of the teacher model weights, that is, subtract the moving step from the t-th generation model weights of the i-th teacher model and multiply it by the weight gradient of the i-th teacher model in the t-th generation. The formula is expressed as:

[0056]

[0057] 4.5) Every 20 times of repeated iteration, calculate the overall classification accuracy of the current teacher model for all training samples; take the weights of the teacher model of the generation with the highest classification accuracy of the training samples during the iteration process as the final teacher model weights, and obtain N trained teacher models.

[0058] In this example, the number of teacher training generations is set to 4000, the single-sample input amount is 72, the loss function is the cross-entropy loss function, and the initial learning moving step is 0.0001.

[0059] Step 5, perform feature-based knowledge distillation on the student model using the teacher model.

[0060] The feature-based knowledge distillation refers to using the intermediate layer feature maps of a trained large and complex teacher model as knowledge to train a small and simple student model, so that the trained student model has a performance similar to that of the teacher model. The specific implementation is as follows:

[0061] 5.1) Randomly sample the same number of training samples from each of the N training sets:

[0062]

[0063] And input them into the corresponding trained teacher models respectively to obtain the output features of the teacher models:

[0064]

[0065] Where represents the output feature of the i-th teacher model on the training sample ;

[0066] 5.2) Input all the sampled training samples into the student model to obtain the output feature of the student model and the prediction probability

[0067] Where and respectively represent the output feature and the prediction probability obtained by inputting the i-th training sample into the student model.

[0068] 5.3) Calculate the similarity loss MLoss between the output feature of the student model and the output feature of the teacher model:

[0069]

[0070] where n is the dimension of the feature vector, N x is the number of training samples, and are respectively the j-th dimension of the features extracted by the student network and the teacher network for the k-th sample of the i-th training data.

[0071] 5.4) Calculate the classification loss CLoss of the prediction probability of the student model and the true class label :

[0072]

[0073] where N xis the number of training samples, and Onehot(·) is one-hot encoding. and are respectively the true label and predicted probability of the k-th sample of the i-th training data;

[0074] 5.5) Add the similarity loss MLoss between the output features of the student model and the output features of the teacher model and the classification loss CLoss of the student model to obtain the total loss Loss = MLoss + CLoss for training the student model;

[0075] 5.6) In each iteration, take the derivative of the total loss Loss of the student model with respect to the weights ω of the student model to obtain the gradient ▽ ω Loss;

[0076] 5.7) In each iteration update, subtract the product of the moving step size η ω and the gradient ▽ ω Loss of the weights of the student model from the weights of the student model to obtain the weights ω t of the student model in the t-th generation to the weights ω t+1 of the student model in the t+1-th generation, which is expressed by the formula:

[0077] ω t+1 ← ω t - η ω ▽ ω Loss;

[0078] 5.8) As the iteration update progresses, the total Loss will gradually decrease to a stable state. Save the network weights of the iteration with the smallest total Loss as the final weights of the student network model to obtain the trained student network model;

[0079] In this example, the number of training generations of the student model is set to 10000, the single-sample input volume is 72, and the initial learning moving step size is 0.0001.

[0080] Step 6, input the hyperspectral image data to be classified into the trained student model for inference to obtain the classification result, and output the classification map of the hyperspectral image data to be classified according to the classification result; in this example, the student model performs ten independent inferences on the test set, and take the average of the classification accuracies of the ten inference results as the final classification accuracy.

[0081] The effect of the present invention can be further illustrated by the following experiments:

[0082] I. Experimental conditions:

[0083] Hardware environment: Intel W-2123 processor, main frequency 3.60 GHz, memory 32 GB, NVIDIA GeForce GTX2080Ti graphics processing unit GPU, memory 27.8 GB.

[0084] Software environment: 64-bit Windows 10 system and deep learning framework of Pytorch 1.6.0.

[0085] The evaluation metrics include overall accuracy OA, average accuracy AA, and Kappa coefficient KAPPA, where:

[0086] The overall accuracy OA represents the proportion of correctly classified samples among all samples. The larger the value, the better the classification effect.

[0087] The average accuracy AA represents the average of the classification accuracies of each class. The larger the value, the better the classification effect.

[0088] The Kappa coefficient KAPPA represents different weights in the confusion matrix. The larger the value, the better the classification effect.

[0089] II. Experimental content

[0090] Experiment 1: On the Pavia University hyperspectral image data, use the joint distillation method of this example, the existing independent training method, and the joint training method to classify the hyperspectral image respectively, obtain the classification result map and calculate the evaluation metrics of their respective classifications. The results are shown in Table 1 and Figure 3 (a).

[0091] Table 1 Comparison of independent training, joint training, and the proposed joint distillation results on Pavia University

[0092]

[0093] As can be seen from Table 1, the accuracy of the present invention in various categories of Pavia University data is higher than that of the existing independent training method and joint training method. Compared with independent training and joint training, the OA of the present invention is 0.08% and 3.17% higher respectively, the AA is 0.60% and 1.90% higher respectively, and the KAPPA is 0.09% and 3.7% higher respectively.

[0094] From Figure 3 (a), it can be seen that the classification result map of the present invention has a smoother classification result and higher edge recognition accuracy.

[0095] Experiment 2: On the Indian Pines hyperspectral image data, the hyperspectral image was classified using the joint distillation method of this example, the existing independent training method, and the joint training method respectively. The classification result images were obtained and the evaluation indexes of each classification were calculated. The results are shown in Table 2 and Figure 3 (b).

[0096] Table 2 Comparison of Indian Pines independent training, joint training and the proposed joint distillation results

[0097]

[0098] As can be seen from Table 2, the accuracy of the present invention in various categories of Indian Pines data is higher than that of the existing independent training method and joint training method. Compared with independent training and joint training, the OA of the present invention is 1.50% and 2.89% higher respectively, the AA is 1.80% and 1.76% higher respectively, and the KAPPA is 1.69% and 3.23% higher respectively.

[0099] From Figure 3 (b), it can be seen that the classification result image of the present invention has a smoother classification result and higher edge recognition accuracy.

[0100] Experiment 3: On the Salinas hyperspectral image data, the hyperspectral image was classified using the joint distillation method of this example, the existing independent training method, and the joint training method respectively. The classification result images were obtained and the evaluation indexes of each classification were calculated. The results are shown in Table 3 and Figure 3 (c).

[0101] Table 3 Comparison of Salinas independent training, joint training and the proposed joint distillation results

[0102]

[0103] In Table 3, the accuracy of the present invention in various categories of Salinas data is higher than that of the existing independent training method and joint training method. Compared with independent training and joint training, the OA of the present invention is 0.38% and 1.98% higher respectively, the AA is 1.02% and 1.82% higher respectively, and the KAPPA is 0.42% and 2.21% higher respectively.

[0104] From Figure 3 (c), it can be seen that the classification result image of the present invention has a smoother classification result and higher edge recognition accuracy.

[0105] Experiment 4: On the Pavia University hyperspectral image data, the present invention is compared with the existing lightweight models LiteDepthwiseNet and LMAFN in terms of the number of parameters, computational complexity, and training time to verify the effect of the present invention on reducing the number of model parameters and improving the operation efficiency. The results are shown in Table 4.

[0106] Table 4 Comparison of the results of independent training, joint training, and the proposed joint distillation in Jiangsu

[0107] Pavia University LiteDepthwiseNet LMAFN The present invention Number of parameters 218326 153060 37908 Computational complexity (Millon) 14.85 7.001 1.766 Training time (s) 366.54 280.33 30.38

[0108] As can be seen from Table 4, compared with the existing LiteDepthwiseNet and LMAFN, the present invention reduces the number of parameters by 82.63% and 75.23% respectively, reduces the computational complexity by 88.11% and 74.78% respectively, and reduces the training time by 91.71% and 89.16% respectively, effectively reducing the model complexity and improving the operation efficiency of the model.

[0109] The experimental results show that the hyperspectral ground object classification method of lightweight multi-domain joint distillation of the present invention uses a point convolution model to extract spectral features, distills the same student model through multiple teacher models, and jointly uses the feature distillation loss and classification loss to update the student model, enabling the same student model to achieve better results than the prior art with fewer parameters, lower computational resource requirements, and less storage cost.

Claims

1. A lightweight multi-domain joint distillation hyperspectral ground object classification method, characterized in that, It includes the following steps: S1) Normalize and expand the spectral dimension of N hyperspectral images in sequence to obtain N hyperspectral images with equal dimensions. Then divide the data of each hyperspectral image into a training set and a test set, obtaining a total of N training sets and N test sets; S2) Construct a lightweight network composed of a cascade of multiple point convolutional layers, a pooling layer, and a fully connected layer. Create and randomly initialize N teacher models and a student model according to its network structure, and ensure that the teacher models and the student model have the same network structure; S3) Use the N training sets to independently train the N teacher network models respectively using the gradient descent algorithm. During the training iteration process, calculate the accuracy of the training set for each iteration, and use the weights of the network model of the generation with the highest accuracy on the training set as the final classification model weights to obtain N trained teacher models; S4) Use the N trained teacher models to perform feature-based knowledge distillation on the same student model to obtain a trained student model: S41) Randomly sample the same number of training samples from the N training sets respectively, and input them into the corresponding trained teacher models respectively to obtain the output features of the teacher models; S42) Input all the sampled training samples into the student model to obtain the output features and prediction probabilities of the student model; S43) Calculate the similarity between the output features of the student model and the output features of the teacher models and use it as the similarity loss MLoss; calculate the classification loss CLoss between the prediction probabilities of the student model and the true class labels using cross-entropy loss to obtain the total loss Loss = MLoss + CLoss; S44) Use the total loss Loss to iteratively update the student model, and use the weights of the network model of the generation with the lowest total loss Loss as the final student model weights to obtain a trained student model; S5) Input the N test sets into the trained student model for testing to obtain the final classification results, and output the classification maps of the N hyperspectral image data according to the classification results.

2. The method according to claim 1, wherein In step S1), the normalization processing and spectral dimension expansion of the N hyperspectral images are carried out in sequence as follows: S11) Perform a linear transformation on the hyperspectral image data x to map the hyperspectral data x to the range [0, 1], obtaining the normalized hyperspectral image where μ and σ are the mean and variance of the hyperspectral image data x, respectively; S12) Perform zero-padding in the channel dimension on the N normalized hyperspectral images so that the N normalized hyperspectral images have the same number of channels, and the number of channels is the maximum value of the original number of channels of the N hyperspectral images, obtaining the hyperspectral images after dimension expansion.

3. The method according to claim 1, wherein In step S1), dividing the data of each hyperspectral image into a training set and a test set is realized as follows S13) Set the hyperspectral image sampling window, where the sampling window is the patch block spatial size, and sample samples from the N hyperspectral images after dimension expansion respectively; S14) Use the label of the central pixel of the patch block as the label of the patch block, and randomly divide all labeled patch blocks in the N hyperspectral images into a training sample set and a test sample set 4. The method according to claim 1, wherein Each point convolution layer in the lightweight network constructed in step S2) includes multiple convolutional kernels, the size of each convolutional kernel is 1×1, the stride is 1, the padding is 0, and the number of convolutional kernels gradually increases with the depth of the point convolution layer.

5. The method according to claim 1, characterized in that, In step S3), using the N training sets to independently train the N teacher network models respectively using the gradient descent algorithm is realized as follows: S31) Set the teacher training algebra to 4000, the single - sample input volume to 72, and the initial learning rate to 0.0001; S32) Let N teacher models classify one of the N hyperspectral images respectively, and randomly sample the same number of training samples from the N hyperspectral images respectively: S33) Input the training samples into the corresponding teacher model to obtain the predicted classes of the teacher model and calculate the cross-entropy loss between the predicted classes and the true ground truth class Y: where CLoss Ti is the loss of the i-th teacher model, N x is the number of training samples, C is the total number of classes, and are the true label and the predicted label of the k-th training sample of the i-th hyperspectral image belonging to class c, respectively; S34) Iterative optimization: Calculate the loss CLoss of all teacher models during iterative optimization T1 , CLoss T2 ,..., CLoss Ti ,..., CLoss TN , and calculate the gradients of these with respect to the weights of all teacher models: Update the weights according to the gradient of the teacher model weights. The single - iteration update formula for the i - th teacher model is: where is the step size, and are the model weights and model weight gradients of the i-th teacher model at the t-th generation, respectively; S35) Repeat S34) 20 times, calculate the overall classification accuracy of the current teacher model for all training samples; take the weights of the teacher model of the generation with the highest classification accuracy of the training samples during the iteration process as the final teacher model weights.

6. The method according to claim 1, characterized in that, In step S43), calculate the similarity loss MLoss between the output features of the student model and the output features of the teacher model and calculate the classification loss CLoss between the predicted probability of the student model and the true class label using cross - entropy loss, as follows: S431) Input the training data into N teacher models respectively to obtain the output features of the teacher models S432) Input the training data into the student model to obtain the output features of the student model and the prediction probability S433) Output features according to the student model and the output features of the teacher model Calculate the similarity loss MLoss between the output features of the student model and the output features of the teacher model: where n is the dimension of the feature vector, N x is the number of training samples, and are the j-th dimension of the features extracted by the student network and the teacher network for the k-th sample of the i-th training data, respectively; S434) Predict the probability according to the student model and the true class label Calculate the classification loss CLoss: where N x is the number of training samples, Onehot(·) is one-hot encoding, and are the true label and predicted probability of the k-th sample of the i-th training data, respectively.

7. The method according to claim 1, characterized in that In step S44), update the student model weights ω using the total loss Loss, as follows: S441) Set the student model training algebra to 10000; S442) Calculate the gradient of the student model weights ω based on the total loss Loss for each iteration, and use the gradient to update the student model weights. The update formula for each iteration is: where η ω is the step size, ω t and are the weights and weight gradients of the t-th generation student model respectively; S443) As the iterative update progresses, the total Loss will gradually decrease to a stable state. Save the network weights of the iteration with the smallest total loss Loss as the final student network model weights.

Images