Class center knowledge distillation method for remote sensing image scene classification

Abstract

The application belongs to the technical field of remote sensing image scene classification, and discloses a class center knowledge distillation method for remote sensing image scene classification. First, a pre-trained teacher network is fine-tuned, then the powerful feature extraction capability of the teacher network is migrated to a student network based on a designed class center distillation loss, the knowledge transfer is completed by constraining the distance between the same feature distribution centers extracted by the teacher and student networks, and the true value label training is combined in the distillation process, finally, the student network is used for prediction alone. The application has good generalization capability, and the extracted features have good intra-class compactness and inter-class discreteness, and the class center knowledge distillation method improves the classification precision of the compact network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image scene classification, and particularly relates to a class center knowledge distillation method for remote sensing image scene classification. BACKGROUND

[0002] In recent years, thanks to the development of remote sensing imaging technology, the spatial resolution of remote sensing images has been greatly improved, which gradually improves the expression ability of images on target details, and meets the practical application needs of intelligent knowledge extraction and information mining. High-resolution remote sensing image scene classification, as an important research field of remote sensing data intelligent interpretation task, can provide support for environmental monitoring, urban planning, resource investigation and the like.

[0003] Remote sensing image scene classification identifies the scene category of each image patch based on global semantic information. Traditional remote sensing image scene classification methods based on artificial feature descriptors directly classify global descriptors such as texture features, or encode the entire scene using visual bag-of-words based on local descriptors such as scale-invariant feature transform. Since the representation ability of artificial features is limited and not suitable for complex scene images, researchers turn to unsupervised learning methods such as sparse coding. However, unsupervised learning methods cannot fully utilize class information. In recent years, with the rapid development of deep learning, convolutional neural networks have been widely used in scene classification due to their strong feature extraction ability and have achieved great success.

[0004] Deep learning-based algorithms have rapidly developed in the field of remote sensing image scene classification and have become dominant. However, high-performance network models usually have many training parameters, high computational cost, and large resource consumption, which makes it difficult for mobile embedded devices and on-orbit processing to meet such computational complexity and high storage requirements. Therefore, compressed models are inevitable. Model compression aims to simplify the model while compensating for accuracy, which is an inevitable trend for practical application. The current mainstream algorithms are network pruning, parameter quantization and knowledge distillation. Network pruning and parameter quantization respectively delete redundant parameters based on designed standards and replace original floating-point parameters with low-bit-width to achieve the purpose of model parameter simplification. Knowledge distillation migrates implicit knowledge from a complex teacher network to a lightweight student network, so that the lightweight network approximates the performance of the complex network, achieving model structure compression.

[0005] The idea of knowledge distillation can be traced back to the theory that complex ensemble models can be transformed into simple neural networks through model compression proposed by Bucilua et al. [Buciluǎ C, Caruana R, Niculescu-Mizil A. Model compression [C] / / Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2006: 535-541. DOI: 10.1145 / 1150402.1150464]. Ba et al. [Ba J and Caruana R. Do deep nets really need to be deep? [C]. Advances in Neural Information Processing Systems, 2014, 27. DOI: 10.48550 / arXiv.1312.6184] completed the verification through experiments, and proposed that minimizing the L2 loss between the logical unit values of the large model and the small model can realize the simulation of the small model. However, the output value of the full connection layer without the softmax function is not constrained, which may contain noise when the model is trained and tested. Therefore, Hinton et al. [Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network [EB / OL]. 2015: arXiv:1503.02531. https: / / arxiv.org / abs / 1503.02531] proposed to use “soft targets”, which softens the class probability of the output by using the softmax function with a temperature coefficient, and fits the class probability distribution through the KL divergence. This process of transferring knowledge from a complex teacher model to a simple student model is defined as knowledge distillation in a pioneering way. With further exploration of knowledge distillation, the distilled knowledge is enriched and expanded. According to the different types of knowledge, the existing methods can be divided into knowledge distillation based on response, feature, instance relationship and network layer relationship.

[0006] As an efficient means of model compression, knowledge distillation has achieved remarkable results in computer vision and other fields. However, the research on knowledge distillation in the field of remote sensing image scene classification is relatively less. Chen et al. [Chen G Z, Zhang XD, Tan X L, et al. Training small networks for scene classification of remote sensing images via knowledge distillation[J]. Remote Sensing, 2018, 10(5): 719. DOI:10.3390 / rs10050719] first introduced the classic knowledge distillation into remote sensing image scene classification, which can effectively improve the performance of the shallow network by matching the outputs of the softmax layers of deep and shallow networks. Yang et al. [Yang H B, Chi Y X, Wang J G. Knowledge distillation method for remote sensing satellite image classification based on pruning network[J]. Application Research of Computers, 2021, 38(8): 2469-2473. DOI:10.19734 / j.issn.1001-3695.2020.07.0387] introduced knowledge distillation to compensate for the accuracy loss caused by model pruning to compress the model. Zhao et al. [Zhao H R, Sun X, Gao F, et al. Pair-wise similarity knowledge distillation for RSI scene classification[J]. Remote Sensing, 2022, 14(10): 2483. DOI:10.3390 / rs14102483] introduced pair-wise similarity knowledge distillation and used mixup technology to mix samples with different labels, which improved the accuracy of the student network by transferring the correlation knowledge of the similarity between virtual samples. However, most of these methods directly apply existing knowledge distillation algorithms, ignoring the challenges of high intra-class diversity and low inter-class separability in scene classification tasks, losing the discriminative information of intra-class diversity and inter-class similarity of scene data, and to some extent reducing the classification accuracy of the student network, resulting in general compression performance. SUMMARY

[0007] The application faces the remote sensing image scene classification task, proposes a class center knowledge distillation method for compressing heavy network to obtain a lightweight network, and the overall framework includes three parts of teacher network fine-tuning, teacher-student network distillation and student network prediction.

[0008] In order to achieve the above purpose, the technical scheme adopted by the application is as follows:

[0009] A class center knowledge distillation method for remote sensing image scene classification includes:

[0010] Firstly, the remote sensing image scene classification dataset is sent into the pre-trained teacher network for parameter fine-tuning.

[0011] Then, the feature distribution of the same class is condensed into knowledge, the knowledge is extracted from the adjusted teacher network, and the class center knowledge in the intermediate hidden layer of the network is used to guide the student network for distillation training.

[0012] Finally, the trained student network is used alone for remote sensing image scene prediction.

[0013] Further, it further includes:

[0014] During the teacher-student network distillation stage, the student network is supervised in combination with the true value label.

[0015] Further, in the network training stage, random flip and random radius Gaussian blur are used for data enhancement, and in the test stage, the test data is not enhanced.

[0016] Further, the feature distribution of the same class is condensed into knowledge, and the knowledge is extracted from the adjusted teacher network.

[0017] The designed class center distillation loss function migrates the feature extraction capability of the teacher network to the student network, and the distance between the feature distribution centers of the same class extracted by the teacher and student networks is constrained to complete the transfer of knowledge: the output features of one or several specific layers of the teacher and student networks are classified according to the label, the center of the feature distribution of each instance of the same label is solved, and the distance between the class centers of the teacher and student is minimized to realize the student model learning the class feature knowledge extracted by the teacher model.

[0018] Further, the class center distillation loss function is:

[0019]

[0020] wherein,

[0021]

[0022] wherein, respectively represent the clustering centers of the teacher network and the student network corresponding to the i-th class center distance, represent the distance between the clustering centers and N represents the number of instances with the same label, k(·,·) represents a kernel function for projecting a feature vector into a higher-dimensional or infinite-dimensional feature space, C T , C S respectively represent the number of channels of the teacher network and the student network, respectively represent the feature map of the i-th channel of the teacher network and the corresponding transpose, respectively represent the feature map of the j-th channel of the student network and the corresponding transpose.

[0023] Further, during the training process, the overall loss function used is:

[0024]

[0025] wherein,

[0026]

[0027] wherein, represents the predicted probability of the sample in the i-th class, z i represents the logit of the i-th class, y true represents the true value label, ∑ represents the summation of all matching layers selected by the present application, and λ is a hyperparameter for balancing the two parts of the loss.

[0028] Further, the teacher network uses ResNet50, and the student network uses ResNet-18 or MobileNetV2.

[0029] Compared with the prior art, the present application has the beneficial effects of:

[0030] The application faces the remote sensing image scene classification task, proposes a class center knowledge distillation method for compressing heavy network to obtain a lightweight network, and the overall framework includes three parts of teacher network fine-tuning, teacher-student network distillation and student network prediction. In order to enable the lightweight network to cope with the challenge of high intra-class difference and low inter-class separability in the scene classification task, a new knowledge distillation loss function is designed, the distance between the same class feature distribution centers extracted by the teacher and student networks is constrained, the powerful feature extraction capability of the teacher network is efficiently transferred, and the extracted feature of the student network is compact within the class and discrete between the classes. The performance of the proposed method and the existing knowledge distillation methods based on response, feature, instance relationship and network layer relationship in the remote sensing image scene classification task is evaluated on four public data sets, and the experimental results prove the efficiency of the class center knowledge distillation method. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The model framework of the class center knowledge distillation method for the remote sensing image scene classification of the embodiment of the application;

[0032] Figure 2 The example sample of the four remote sensing image scene classification data sets of the embodiment of the application;

[0033] Figure 3 The precision confusion matrix (%) of the comparative experiment of the embodiment of the application on the RSC11 data set; wherein (a) corresponds to the student network trained alone, (b) corresponds to the fine-tuned teacher network, (c) corresponds to the student network trained based on the improved NST method, and (d) corresponds to the student network trained based on the method of the application;

[0034] Figure 4 The test error curve of the embodiment of the application; wherein (a) is the method of the application compared with the single training, and (b) is the method of the application compared with the improved NST;

[0035] Figure 5 The feature scatter plot of the RSSCN7 data set of the embodiment of the application visualized by the T-SNE algorithm, wherein the red circle in the figure represents the area where the feature cluster compactness difference is obvious; wherein (a) corresponds to the student network trained alone, (b) corresponds to the improved NST method, and (c) corresponds to the method of the application;

[0036] Figure 6 The heat map of the output feature layer of the embodiment of the application; wherein (a) corresponds to the student network trained alone, (b) corresponds to the fine-tuned teacher network, (c) corresponds to the improved NST method, and (d) corresponds to the method of the application. DETAILED DESCRIPTION

[0037] The application will be further explained in conjunction with the drawings and specific embodiments:

[0038] The model framework of the class center knowledge distillation method for remote sensing image scene classification proposed in the application is as shown in the figure Figure 1 The model framework of the class center knowledge distillation method for remote sensing image scene classification proposed in the application is as shown in the figure

[0039] 1.1 Distillation loss function

[0040] Remote sensing image scene classification faces the challenges of high intra-class diversity and low inter-class separability. The existing knowledge distillation methods ignore the discriminative information of intra-class diversity and inter-class similarity of scene data, and thus cannot well learn the feature extraction ability of the teacher model. Therefore, the application designs a class center knowledge distillation loss function for remote sensing image scene classification, which completes the knowledge transfer about classes by constraining the distance between the feature distribution centers of the same class extracted by the teacher and student networks, and expects the features extracted by the student model to have good intra-class compactness and inter-class discreteness as the teacher model.

[0041] 1.1.1 Neuron Selectivity Transfer algorithm

[0042] The Neuron Selectivity Transfer (NST) algorithm realizes knowledge transfer by minimizing the maximum average discrepancy measure of the neuron selectivity feature distribution between the teacher and student networks. The Maximum Mean Discrepancy (MMD) is used to measure the difference between the probability distributions of data samples. Specifically, the distance is calculated by mapping two non-linearly separable distributions into a high-dimensional linearly separable Reproducing Kernel Hilbert Space (RKHS), as shown in formula (1).

[0043]

[0044] In the formula, samples x i and y j are collected from sample set φ(·) represents an explicit mapping function.

[0045] The NST algorithm takes the activation value of each spatial location as a feature f ij The feature map of all positions of each channel is flattened into a 1xHW-dimensional vector and regarded as a sample The samples of all channels are combined into a CxHW distribution The samples of all channels are combined into a CxHW distribution The samples of all channels are combined into a CxHW distribution The maximum mean discrepancy between the two distributions is matched to the feature of the output layer of the teacher-student network, as shown in equation (2). T And F S The maximum mean discrepancy between the two distributions is matched to the feature of the output layer of the teacher-student network, as shown in equation (2).

[0046]

[0047] On the basis of the above method, the application further condenses the same type of feature distribution into knowledge. The feature distribution of the same type of instance is similar, and mapping it into a high-dimensional space will form a cluster. The compactness within the class and the dispersion between the classes of the features extracted by the teacher model are reflected in the information of the cluster, which is characterized as class feature knowledge for the student model to learn. Specifically, the output features of one or several specific layers of the teacher-student network are classified according to the label, the center of the feature distribution of each instance with the same label is solved, and the distance between the centers of each class of the teacher and student is minimized to realize the learning of the class feature knowledge extracted by the teacher model by the student model.

[0048] 1.1.2 Class center knowledge distillation loss

[0049] The application represents the feature map of a specific output layer in the network as An instance k generates feature maps (F T ) k and (F S ) k after training through the teacher-student network, which can be regarded as two feature distributions and Mapping them to the reproducing kernel Hilbert space φ((F T ) k ) and φ((F S ) k ), the clustering centers of the feature distributions of N instances under the same label are solved and and the distance between the two is calculated

[0050]

[0051] Note that the feature maps of the corresponding output layers of the teacher-student network should have the same spatial dimension HxW. If the sizes of the feature maps do not match, interpolation is needed.

[0052] The mapping function phi(·) is complex, and the kernel trick K(x,y) = <phi(x), phi(y)> = phi(x) T Simplify the calculation by normalizing the sample with L2 norm Ensure that the samples are compared at the same order of magnitude, so Reformulated as shown in equation (4).

[0053]

[0054] In the formula: k(·,·) represents the kernel function of projecting the feature vector into a higher dimensional or infinite dimensional feature space. A polynomial kernel function is used, Where c=0, d=2. C T , C S Respectively represent the number of channels of the teacher network and the student network, Respectively represent the feature map of the i-th channel of the teacher network and the corresponding transpose, Respectively represent the feature map of the j-th channel of the student network and the corresponding transpose.

[0055] Then, the class center distance is summed to obtain the class center knowledge distillation loss function value of a certain output layer As shown in equation (5).

[0056]

[0057] Where, Respectively represent the clustering centers of the teacher network and the student network corresponding to the i-th class center distance.

[0058] The present application designs to distill the feature maps of multiple output layers in the network, such as Figure 1 As shown, the selection of the output layer will be introduced in the experimental setup.

[0059] 1.2 Overall loss function

[0060] During the training process, it is beneficial for the student model to improve performance by forcing the student model to match the ground truth label using the standard cross-entropy loss function, as shown in equation (6).

[0061]

[0062] Where, Represents the predicted probability of the sample in the i-th class, z i Points to the logit of the i-th class, y true Represents the true value label.

[0063] Therefore, the entire objective function during the training process contains two parts of class center knowledge distillation loss and standard cross-entropy loss, which can be represented as

[0064]

[0065] Note that ∑ refers to the summation of all matching layers selected by the present application, and λ is a hyperparameter that balances the two-part loss.

[0066] 1.3 Algorithm specific flow

[0067] We first train the teacher model with standard supervised learning strategy. To make the teacher model have good feature representation ability, we use the pre-training-fine-tuning mechanism, pre-train the teacher model on the ImageNet dataset in advance, and then fine-tune the model on the remote sensing scene classification dataset. The fine-tuned teacher network guides the student network training, and the student network learns the ability of the teacher network to extract features according to the class center distillation loss, and is supervised by the true value label. Finally, the prediction performance of the student network is tested separately. The whole training process is as follows:

[0068]

[0069] To verify the effect of the present application, the following experiments are performed:

[0070] 2. Experiment and analysis

[0071] 2.1 Dataset

[0072] The present application performs comprehensive experiments on four mainstream remote sensing image scene classification datasets, RSC11, UC Merced Land-use (UCM), RSSCN7 and Aerial Image Dataset (AID), and the detailed information of the datasets is shown in Table 1.

[0073] Table 1 Remote sensing image scene classification dataset

[0074]

[0075] Some images are randomly extracted from the dataset as example samples, as shown in Figure 2 From the left two columns, it can be found that these scene classification datasets have large intra-class diversity, such as planting land, residential area, grassland, tourist attraction, etc.; from Figure 2 The right four columns can be observed that some scene classes in the dataset have high similarity, such as the three classes of highway, interchange and railway in the RSC11 dataset, various subdivided residential areas and buildings in the UCM dataset, industrial areas and residential areas, farmland and grassland in the RSSCN7 dataset, and desert and bare land, lake and park in the AID dataset are very similar and difficult to distinguish. This poses a great challenge to the compact and lightweight classification network. Figure 2

[0076] 2.2 Experimental setup ​

[0077] Network structure. The present application adopts ResNet and MobileNet series as the basic architecture of the teacher-student network. Among them, the teacher network is ResNet-50, and the student network uses ResNet-18 and MobileNetV2 respectively to study the knowledge distillation performance of the teacher-student network model belonging to the same series and different series. The present application designs the class center knowledge distillation of the feature maps of the four output layers in the network, and the network structure details and the size information of each output layer feature are shown in Table 2.

[0078] Table 2 Teacher-student network structure and output layer feature information

[0079]

[0080] Among them, F represents the feature map of the designed distillation network middle output layer, N x M represents the size of the feature map, and C represents the number of channels.

[0081] Experimental configuration. The present application uses NVIDIA Tesla 4 in the PyTorch environment to conduct comprehensive experiments. In the training stage, we use random flip and random radius Gaussian blur for data enhancement. In the test stage, the test data is not enhanced. The teacher network fine-tuning process sets the batch size to 64, the initial learning rate to 1e-4, adjusts the learning rate by exponential decay, uses the stochastic gradient descent (SGD) with a momentum of 0.9 as the optimizer, and sets the iteration number to 160. The student network single training process and knowledge distillation process are both set to a batch size of 32, an initial learning rate of 0.05, still adjust the learning rate by exponential decay, use the stochastic gradient descent (SGD) with a momentum of 0.9 as the optimizer, and set the iteration number to 240. All training adopts the early termination strategy, if the validation loss does not decrease after 30 consecutive iteration calculations, the training is terminated.

[0082] Hyperparameter setting. The present application sets the balance factor λ to 50, and adopts 20 generations of linear warm-up. The best setting of the parameters is obtained by experiment, λ has little effect on the overall accuracy in the appropriate range, but the accuracy has a decreasing trend with the increase of λ value, analysis may be that large λ value leads to larger initial loss, so a certain period of linear warm-up is used to reduce the initial loss, and finally 20 generations of linear warm-up are obtained to effectively improve the accuracy.

[0083] 2.3 Experimental results and analysis

[0084] 2.3.1 Effectiveness experiment

[0085] To verify the effectiveness of the class center knowledge distillation method, the present application first carries out experiments on different teacher-student architectures and training ratios on the classic remote sensing image scene classification dataset UCM, and compares with eight advanced knowledge distillation methods: KD, DKD, NST, VID [Ahn S, Hu S X, Damianou A, et al. Variational information distillation for knowledge transfer [C] / / 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 9155-9163. DOI: 10.1109 / CVPR.2019.00938], KDSVD, ReviewKD, RKD, SP, for easy analysis, these methods are divided into four categories according to the type of knowledge. The experimental results are shown in Table 3, and the method proposed in the present application achieves the optimal accuracy.

[0086] In the experiment of the same series of teacher-student networks, it can be observed that the class center knowledge distillation method improves the overall classification accuracy by more than 5% compared with the student network trained alone, and compared with the most advanced DKD method, it realizes an improvement of 1.43% at a training ratio of 80%, and the overall accuracy is as high as 97.14%, and when the training ratio is 60%, it is improved by 2.74% compared with the ReviewKD method with the optimal accuracy, and the overall accuracy is 95.36%.

[0087] In the experiment of the same series of teacher-student networks, it can be observed that the class center knowledge distillation method improves the overall classification accuracy by more than 5% compared with the student network trained alone, and compared with the most advanced DKD method, it realizes an improvement of 1.43% at a training ratio of 80%, and the overall accuracy is as high as 97.14%, and when the training ratio is 60%, it is improved by 2.74% compared with the ReviewKD method with the optimal accuracy, and the overall accuracy is 95.36%.

[0088] The experimental results show that the method proposed in the present application has a significant improvement compared with the single training, and is superior to the current advanced knowledge distillation method. Therefore, the present application considers that the method proposed can effectively extract valuable information from the teacher model and transfer it to the student model for learning. From the two groups of experiments, the lower the training ratio, the greater the accuracy improvement of the proposed method, and the more obvious the advantage of class center knowledge distillation.

[0089] Table 3 Overall accuracy (%) of various knowledge distillation methods on UCM dataset

[0090]

[0091] Note that Baseline refers to the result of training the student network alone. The result with the highest accuracy among all methods is shown in bold, and the second highest is underlined. The results are the average of 10 experiments.

[0092] Table 4 shows the comparison of the size of the network model before and after compression. The complexity of the model operation is measured by the amount of calculation (Floating point operations, FLOPs), and the size of the model storage is measured by the amount of parameters (Parameters). The compression rate refers to the ratio of the size of the student network (after model compression) to the size of the teacher network (before model compression). The results show that on the 60% UCM dataset, compared with the ResNet50 model, the ResNet18 model achieves a compression rate of 47.52% and compensates for an accuracy of 5.36%, and the MobileNetV2 model achieves a compression rate of 9.55% and reduces the calculation complexity to 7.6% of the original.

[0093] Table 4 Comparison of network model size

[0094]

[0095] 2.3.2 Applicability Experiment

[0096] To verify the universality of the class center knowledge distillation method, the present application will perform supplementary experiments on the RSC11 small dataset with few classes and few samples, the RSSCN7 multi-scale dataset with few classes and many samples, and the AID large and complex dataset with many classes and uneven samples. Table 5 shows the results of a series of experiments on the teacher and student networks on multiple datasets with a training rate of 60%. The method proposed in the present application achieves an accuracy of 94.37% and 94.10% on the RSC11 and AID datasets, respectively, which is an improvement of 4.23% and 5.12% compared to training alone, and an improvement of 2.42% and 1.07% compared to the second best knowledge distillation method. It is worth noting that the current advanced knowledge distillation method performs poorly on the RSSCN7 dataset, with an accuracy that is lower than that of the student network trained alone, and only the RKD method maintains the accuracy, while the method proposed in the present application has an improvement of 2.95%, with an accuracy of 91.70%. It can also be observed that on the RSC11 and RSSCN7 datasets with fewer classes, the distillation method not only achieves significant improvement compared to training alone, but even exceeds the accuracy of the teacher model with pre-training and fine-tuning, indicating that the class center knowledge distillation method transfers more specialized knowledge compared to other distillation methods. The specific reasons will be analyzed later.

[0097] Table 5 Overall accuracy (%) of multiple datasets under the condition of 60% training rate and isomorphic teacher and student networks

[0098]

[0099] Note that the teacher network is ResNet-50, and the results shown are the classification results after pre-training and fine-tuning, and the student network is ResNet-18, and the results shown are the results of separate training without distillation. The results with the highest accuracy in all methods are shown in bold, the second highest are underlined, and italic bold means exceeding the teacher classification accuracy. The results are the average of 10 experiments.

[0100] Based on the above analysis, in the three groups of experiments of different architectures of teacher and student networks, different ratios of training samples and different sizes of data sets, the method proposed in the application performs excellently, and the accuracy is improved compared with the separately trained student network and the advanced knowledge distillation method, which shows that the method has good generalization ability.

[0101] 2.3.3 Technical comparison experiment

[0102] To verify the superiority of the method proposed in the application, the application performs a comparative experiment on two distillation methods on the RSC11 Dataset data set, and analyzes the experimental results of the fine-tuned ResNet-50 teacher network and the separately trained ResNet-18 student network. In order to ensure the uniqueness of the variable, the NST distillation method is improved first, and the same output layer features are matched to study the effectiveness of the class center knowledge.

[0103] The overall classification accuracy is shown in Table 6, and the method proposed in the application has an improvement of 1.811% compared with the improved NST method. The precision confusion matrix is calculated from the experimental results to intuitively view the classification accuracy of each class and the misclassification and missed classification. The value of the (i, j) element in the confusion matrix represents the proportion of test samples with label i and class j to the total number of test images, and the results are shown in Table 7. Figure 3

[0104] Table 6 Results of comparative experiment on RSC11 data set

[0105]

[0106] As Figure 3 can be seen, the method proposed in the application improves the accuracy of all classifications, among which the port and high building increase by 7.69% and 6.80% compared with the improved NST distillation method, and the accuracy of other classes also increases by 1-4%. The classification accuracy is relatively low for highway, interchange and railway, and it is worth noting that the classification accuracy of these three classes not only exceeds the separately trained student model, but also greatly exceeds the teacher model. The application analyzes that these challenging classes not only maintain the student network's ability to distinguish these classes after distillation, but also learn the knowledge in the teacher network, achieving high accuracy.

[0107] ​In addition, from the visualized test error curves Figure 4 ) it can be observed that the proposed method (blue line) can converge quickly compared with the student training alone (orange line), and the test error curve of the improved NST distillation method (red line) also converges quickly but then oscillates and has a floating trend, indicating that the NST distillation loss is greatly affected by random samples, and the trained model has the risk of overfitting, while the proposed class center distillation loss can well handle noise, and the loss value quickly converges and tends to be stable.

[0108] 2.3.4 Visual analysis

[0109] The present application visualizes the feature extraction capability of the model using the T-SNE algorithm on the challenging RSSCN7 dataset. The T-SNE algorithm is used for high-dimensional data dimensionality reduction, which can represent high-dimensional features in a two-dimensional visual space. The high-dimensional features extracted by the model are visualized by the T-SNE algorithm to measure whether the proposed method can effectively solve the intra-class diversity and inter-class similarity problem. As shown in Figure 5 compared with the student model trained alone and the improved NST method, the features extracted by the proposed method have more compact feature clusters of the same class and relatively dispersed feature clusters of different classes, especially the three classes of feature clusters in the red circle, which are the most obvious. It shows that the features extracted by the proposed method have good intra-class compactness and inter-class dispersion, effectively addressing the intra-class diversity and inter-class similarity problem of remote sensing image scenes.

[0110] To further analyze the learning effect of the student network, the present application visualizes the four network intermediate output layers designed to match during the distillation process, and uses the Grad-CAM (Gradient-weighted Class Activation Mapping) proposed by Selvaraju et al. [Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization [C] / / 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017:618-626. DOI:10.1109 / ICCV.2017.74] to draw a heat map to show the areas and feature information that the network intermediate layer focuses on, so as to explore the feature extraction capability of the student network. As shown in Figure 6As shown, the heat value represents the visualization result of model attention, and the higher the value is, the more the model focuses on the area, and the darker the area in the heat map is, the higher the attention is. The horizontal analysis can find that different network output layers have different focuses, and the deeper the level is, the more abstract the features are focused on. For example, feature layer 1 mainly focuses on edge features, and feature layer 4 mainly focuses on semantic scene features. The vertical analysis of the attention areas of the same output layer of different methods shows that, compared with the single training method and the improved NST method, the method of the application comprehensively considers the attention areas of the teacher network and the student network, which shows that the method of the application well transmits the feature extraction capability of the teacher network. From the heat maps of all output layers, it can be seen that the method of the application focuses on a larger receptive field and is less likely to make prediction errors due to excessive focus on the intra-class similarity of scene features. The original image is an industrial area, and the improved NST method makes an error prediction of a residential area due to excessive focus on the houses in the image, which is greatly affected by noise, and also shows that the features extracted by the method of the application can effectively cope with the inter-class similarity challenge.

[0111] The application proposes a class center knowledge distillation method for the scene classification task of remote sensing images. Through two-step training of teacher network fine-tuning and teacher-student network distillation, a high-performance lightweight network capable of being deployed on edge computing devices is obtained. The class center distillation loss function designed in the application matches the centers of the same class features extracted by the teacher and student networks, efficiently transferring the powerful feature extraction capability of the complex network, so that the lightweight network can cope with the challenges of intra-class high diversity and inter-class low separability in the scene classification task. The application conducts a series of comprehensive experiments on four public remote sensing image scene classification benchmark datasets to evaluate the effectiveness of the class center knowledge distillation method. The experimental results are summarized as follows:

[0112] (1) The application conducts effectiveness experiments on the classic UCM high-resolution land use dataset under the conditions of two training rates of 60% and 80%, and two teacher-student architectures of ResNet50 and ResNet18 and ResNet50 and MobileNet-V2, and compares them with four categories and eight existing advanced knowledge distillation methods. The experimental results show that the class center knowledge distillation method performs best in isomorphic and heterogeneous teacher-student networks, especially when the training rate is lower, the accuracy improves more. Subsequently, the application conducts applicability experiments on the small RSC11 dataset with few classes and samples, the multi-scale RSSCN7 dataset with few classes and many samples, and the AID large and complex dataset with many classes and uneven samples. The experimental results prove that the method has good generalization ability.

[0113] (2) The application also carries out technical comparison experiment with improved NST algorithm, finds that the class center knowledge distillation method can still maintain excellence in the challenging class through confusion matrix, not only maintains the classification ability of the student network, but also learns the knowledge in the teacher network, and the test error curve indicates that the proposed distillation loss function can better handle noise to make the loss value quickly converge and tend to be stable, verifies the superiority of the method of the application. In addition, the application visualizes the feature extraction ability of the model based on the T-SNE algorithm, and draws a heat map based on Grad-CAM to visualize the attention area of the output layer, and the results show that the features extracted by the method of the application have good intra-class compactness and inter-class discreteness.

[0114] In summary, the class center knowledge distillation method proposed in the application improves the classification precision of the compact network, and is optimal compared with other distillation methods.

[0115] The above only shows the preferred embodiments of the application, and it should be noted that for ordinary skilled persons in the art, some improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements should also be considered as the protection scope of the application.

Claims

1. A class-center knowledge distillation method for scene classification of remote sensing images, characterized in that, include: First, the remote sensing image scene classification dataset is fed into a pre-trained teacher network for parameter fine-tuning; Then, the characteristic distribution of the same type is condensed into knowledge, knowledge is extracted from the adjusted teacher network, and the class center knowledge of the hidden layer in the middle of the network is used to guide the student network to carry out distillation training. Finally, the trained student network was used alone for scene prediction in remote sensing images; During training, the class center distillation loss function used is: In the formula, in, , These represent the cluster centers of the teacher network and student network corresponding to the i-th cluster center distance, respectively. Representing cluster centers and The distance between them This indicates the number of instances with the same tag. This represents a kernel function that projects feature vectors onto a higher-dimensional or infinite-dimensional feature space. , These represent the number of channels in the teacher network and the student network, respectively. , Let these represent the feature map of the i-th channel of the teacher network and its corresponding transpose, respectively. , Let represent the feature map of the j-th channel of the student network and its corresponding transpose, respectively.

2. The class center knowledge distillation method for scene classification of remote sensing images according to claim 1, characterized in that, Also includes: During the online distillation phase between teachers and students, the student network is monitored using truth labels.

3. The class-center knowledge distillation method for scene classification of remote sensing images according to claim 1, characterized in that, During the network training phase, data augmentation is performed using random flipping and Gaussian blur with random radius. During the testing phase, no augmentation is performed on the test data.

4. The class center knowledge distillation method for scene classification of remote sensing images according to claim 1, characterized in that, The characteristic distributions of similar types are condensed into knowledge, and knowledge is extracted from the adjusted teacher network, including: The designed class center distillation loss function transfers the feature extraction capabilities of the teacher network to the student network. Knowledge transfer is achieved by constraining the distance between the distribution centers of similar features extracted by the teacher and student networks: the output features of one or several specific layers of the teacher and student networks are classified by label, the center of feature distribution of each instance with the same label is solved, and the student model learns the class feature knowledge extracted by the teacher model by minimizing the distance between each class center between the teacher and student.

5. The class center knowledge distillation method for scene classification of remote sensing images according to claim 1, characterized in that, The overall loss function used during training is: In the formula, in, Indicates the sample at the 1st Predicted probability of class Refers to the first Logit class, Represents the truth value label. This refers to summing over all selected matching layers. It is a hyperparameter that balances the two parts of the loss.

6. The class-center knowledge distillation method for scene classification of remote sensing images according to claim 1, characterized in that, The teacher network uses ResNet50, and the student network uses ResNet-18 or MobileNetV2.

Images