A teacher-supervised mutual learning knowledge distillation method for satellite remote sensing scene classification based on standardized contrast learning

Abstract

The present application belongs to the technical field of intelligent processing of satellite-borne remote sensing, and particularly relates to a teacher-supervised mutual learning knowledge distillation satellite-borne remote sensing scene classification method based on standardized contrast learning. The specific process is as follows: the output of the teacher-student classification network is standardized by using RobustScaler, and the SCL loss is calculated; the mutual learning loss of N lightweight student models is calculated; the projector is set to ensure feature alignment, and the bridge loss is calculated; the TSML total loss is set based on the above losses, and the hyperparameters of each loss are set; during training, each hyperparameter of the TSML total loss is fine-tuned and optimized based on the greedy genetic algorithm; and the satellite-borne remote sensing scene is classified by using the trained teacher-student classification network.

Description

Technical Field

[0001] This invention belongs to the field of spaceborne remote sensing intelligent processing technology, specifically involving a spaceborne remote sensing scene classification method based on standardized comparative learning, teacher-supervised mutual learning knowledge distillation. Background Technology

[0002] In traditional spaceborne remote sensing scene classification (RSSC), remote sensing images need to be transmitted to ground base stations for processing. However, with the increase in image resolution and the surge in data volume, the downlink bandwidth pressure increases significantly, and there are problems such as high processing latency and poor real-time performance. Therefore, deploying CNN models directly on spaceborne edge devices to achieve a real-time "on-board processing - ground feedback" mode has become a key direction for solving the above problems.

[0003] However, high-performance CNN models typically contain billions of parameters and massive multiply-accumulate operations, whose computational complexity and storage requirements far exceed the resource constraints of spaceborne edge devices. To achieve lightweight models, existing techniques mainly include network pruning, low-bit quantization, lightweight network design, and knowledge distillation (KD). Existing knowledge distillation methods perform well in natural image processing, but when directly applied to RSSC, they suffer from insufficient feature extraction capabilities, difficulty in aligning teacher and student model logit, and low hyperparameter tuning efficiency, making them unsuitable for real-time processing tasks in spaceborne remote sensing. Summary of the Invention

[0004] To address the limitations of existing knowledge distillation methods in meeting the real-time processing requirements of spaceborne remote sensing tasks, this invention provides an efficient knowledge distillation (SCML) method suitable for spaceborne remote sensing scene classification. Through three main stages—standardized contrastive learning (SCL), teacher-supervised student learning (TSML), and hyperparameter tuning based on a greedy genetic algorithm (GGA)—this method solves the problems of weak feature extraction, difficult logit alignment, and cumbersome hyperparameter tuning in existing methods. Ultimately, it enables high-precision, low-latency deployment of lightweight models on spaceborne edge devices.

[0005] The method of this invention is achieved through the following technical solution:

[0006] Firstly, this invention provides a teacher-supervised mutual learning knowledge distillation method for spaceborne remote sensing scene classification based on standardized contrastive learning. The specific process is as follows: Standardized Contrastive Learning (SCL): Positive and negative sample pairs are generated and input into the teacher-student classification network. RobustScaler is used to standardize the output of the teacher-student classification network, and the SCL loss is calculated. ; Teacher-Supervised Student Mutual Learning (TSML): Set up N lightweight student models, implement knowledge transfer between student models using KL divergence, and calculate their mutual learning loss. Set the projector to display the student model. Layer output features Mapping to the teacher model Layer features In the same dimension, ensure feature alignment and calculate bridge loss. After feature alignment, the student model uses the teacher classifier for classification; SCL loss is included. Mutual learning loss Bridge losses and knowledge distillation cross-entropy loss The total TSML loss is calculated, and hyperparameters are set for each loss. Hyperparameter tuning: The teacher-student classification network is trained based on the total loss of TSML. During training, each hyperparameter of the total loss of TSML is fine-tuned and optimized based on a greedy genetic algorithm. The trained teacher-student classification network is then used to classify spaceborne remote sensing scenes.

[0007] Optionally, the specific process for generating positive and negative sample pairs according to the present invention is as follows: randomly select one sample as an "anchor sample" and construct a "same-category index pool" and a "different-category index pool" according to its category; sample one positive sample from the same-category index pool and k negative samples from the different-category index pool to form a positive and negative sample pair. .

[0008] Optionally, this invention transforms hyperparameter tuning into a multi-objective optimization problem, with the objective function being:

[0009] in, , , , For hyperparameters, For student model parameters, To measure the classification accuracy of the student model on the validation set, This represents the total number of student models; The constraints are: .

[0010] Optionally, the mutual learning loss described in this invention is:

[0011] in, Representing the With the The student model predicts the probability. and The KL divergence.

[0012] Optionally, the bridge loss described in this invention is:

[0013] in, For projector functions, for Norm, The teacher model is represented by the first... Layer features, The student model is represented by the first... Layer output features.

[0014] Optionally, the standardized comparison loss described in this invention is: SCL loss function By contrast loss True target cross-entropy loss and standardized distillation loss composition:

[0015] Among them, comparative loss The similarity between positive and negative samples is used to guide feature differentiation. The loss was calculated based on the RobustScaler normalization results.

[0016] Optionally, the RobustScaler standardization described in this invention:

[0017] in, The median of the logit vector. ,in It is the 75th percentile. It is the 25th quantile. This refers to the temperature parameter.

[0018] Optionally, the standardized distillation loss described in this invention is:

[0019] in, The student model is logit. The representative teacher model is logit. This represents the softmax function.

[0020] Optionally, when fine-tuning each hyperparameter of the total TSML loss based on the greedy genetic algorithm described in this invention, a delayed tuning mechanism is also added: let the total number of training rounds be set to... ,exist Wheel start-up optimization.

[0021] Optionally, when fine-tuning each hyperparameter of the total TSML loss based on the greedy genetic algorithm described in this invention, a greedy multi-point mutation mechanism is also added, including: Gradient importance score: The absolute value of the gradient of the loss function with respect to each hyperparameter is calculated through backpropagation, and the normalized value is used as the hyperparameter importance score; Targeted mutation: Prioritize fine-tuning of high-importance hyperparameters and make minor adjustments to low-importance hyperparameters to avoid performance fluctuations caused by random mutations.

[0022] Beneficial effects: First, this invention extracts high-quality features from the teacher model through robust standardization and contrastive learning, reducing the difference in Logit between teachers and students; it introduces a teacher supervision and student mutual learning mechanism, reuses the teacher classifier, and strengthens the semantic feature extraction capability of the student model; and it automatically optimizes the loss function hyperparameters through GGA tuning, improving the distillation effect and adapting to the limited computing resources of spaceborne equipment.

[0023] Second, this invention can effectively improve the classification accuracy of the distillation model, achieving an average accuracy improvement of 4.304% on five benchmark datasets, while reducing the model parameters to less than 20% of those of the teacher model. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of the system operation of the method of the present invention; Figure 2 Here is a structural diagram of the SCL module; Figure 3 This is the core algorithm flow of the TSML module; Figure 4 The algorithm flow for GGA; Figure 5 These are the actual measurement results. Detailed Implementation

[0026] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0029] This application presents an embodiment of a teacher-supervised mutual learning knowledge distillation method for spaceborne remote sensing scene classification based on standardized contrastive learning. The workflow is as follows: Figure 1 As shown, its specific components include: Standardized Contrastive Learning (SCL): Positive and negative sample pairs are generated and input into the teacher-student classification network. RobustScaler is used to standardize the output of the teacher-student classification network, and the SCL loss function is calculated. ; Teacher-Supervised Student Mutual Learning (TSML): Set up N lightweight student models, implement knowledge transfer between student models using KL divergence, and calculate their mutual learning loss. Set the projector to display the student model. Layer output features Mapping to the teacher model Layer features In the same dimension, ensure feature alignment and calculate bridge loss. After feature alignment, the student model uses the teacher classifier for classification; SCL loss is included. Mutual learning loss Bridge losses and knowledge distillation cross-entropy loss The total TSML loss is calculated, and hyperparameters are set for each loss. Hyperparameter tuning: The teacher-student classification network is trained based on the total TSML loss. During training, each hyperparameter of the total TSML loss is fine-tuned and optimized based on a greedy genetic algorithm. The trained teacher-student classification network is used to classify spaceborne remote sensing scenes.

[0030] The above process will be explained in detail below: Component 1: Setting up the Standardized Contrastive Learning (SCL) module, used to perform Standardized Contrastive Learning (SCL). The SCL module improves the efficiency of teacher knowledge extraction and the matching degree of teacher-student Logit through positive and negative sample pair generation and robust normalization. The specific process is as follows: (1) Generation of positive and negative sample pairs Anchor sample selection: During the training iteration, one sample is randomly selected as the "anchor sample", and a "same class index pool" (positive sample pool) and a "different class index pool" (negative sample pool) are constructed according to its category. Sample sampling: Sample one positive sample from the same type of index pool and k negative samples from the different type of index pool to form a positive-negative sample pair. ; Logit calculation: Input positive and negative sample pairs into the teacher model and student model respectively, and output the classification logit respectively.

[0031]

[0032] in, The table shows the classification of student network output logit. The classifier logit representing the network output of the teacher. For the generated set of positive and negative samples, , These are the classification network layers for the student and teacher models, respectively.

[0033] (2) Standardized comparative loss calculation The loss function of SCL is derived from contrastive loss. True target cross-entropy loss and standardized distillation loss composition:

[0034] Among them, comparative loss The formula for distinguishing features by calculating the similarity between positive and negative samples is as follows:

[0035]

[0036]

[0037] in, For batch size, The number of negative sample pairs For negative sample pairs, the prior probability distribution is given. It is the first 100 samples (here, "samples" refers to samples with index 0.05) The The similarity between the image and the positive sample. For the first The sample and the first Similarity of negative samples These are small constants used for stable calculations.

[0038] To address the issue of logit outliers in remote sensing scenarios, RobustScaler is used instead of traditional Z-Score standardization to filter out extreme values ​​(such as logit values ​​from misactivated teacher models). The formula is as follows: RobustScaler Standardization:

[0039] in, The median of the logit vector. ,in It is the 75th percentile. It is the 25th quantile. The temperature parameter is used; the final standardized distillation loss is calculated using the KL divergence formula: Standardized distillation loss (KL divergence):

[0040] in, The student model is logit. The representative teacher model is logit. This represents the softmax function, which is used to eliminate logit outliers and improve the logit matching degree of teacher and student models.

[0041] Component 2: Set up the Teacher-Supervised Student Learning (TSML) module to implement teacher-supervised student learning; The TSML module further narrows the knowledge gap between teachers and students through a "teacher-guided + student-interactive learning" mechanism, and reuses the teacher classifier to improve semantic discrimination ability. The specific design is as follows: (1) Student peer learning mechanism Set up N lightweight student models (such as ResNet18, MobileNetV2), and realize knowledge transfer among students through average KL divergence: For the j-th student model, its mutual learning loss is:

[0042] in, Representing the With the The student model predicts the probability. and The KL divergence is used to measure the difference in probability distributions between two models, guiding students to imitate each other's models and improve their generalization ability.

[0043] (2) Losses in teacher supervision and bridging To address the issue of insufficient semantic extraction capabilities in student models, a projector and bridging loss are designed. The student features are mapped to the teacher feature space, and the teacher classifier is reused for semantic guidance. 1. Projector Design: Structure: 3 "convolution + batch normalization + ReLU" (CBR) modules (convolution kernels are 1×1, 3×3, and 1×1 respectively) + 1 layer of 1×1 average pooling; Function: Transfer the student model to the first Layer output features (Low-dimensional) mapping to the teacher model Layer features (High-dimensional) same dimension, ensuring feature alignment; 2. Bridge loss calculation:

[0044] in, For projector functions, for Norms measure the difference between student projected features and teacher features, guiding students to learn the teacher's key semantic features. 3. Reuse of teacher classifier: After feature alignment, the student model uses the teacher classifier for classification, directly utilizing the teacher's strong semantic discrimination ability to improve classification accuracy.

[0045] (3) TSML Total Loss Integration The total loss in the TSML phase is the sum of four parts:

[0046] in, Cross-entropy loss ensures the accuracy of the student model in classifying the true labels; Average KL divergence loss enables mutual learning between student models; Standardized contrastive learning loss is used to extract structured knowledge from the teacher model; Bridging loss guides the alignment of student characteristics with teacher characteristics; , , , Hyperparameters control the weights of each loss term and are automatically optimized by GGA.

[0047] Component 3: Set up a hyperparameter tuning module based on the Greedy Genetic Algorithm (GGA) to perform hyperparameter tuning. For distillation loss function To address the inefficiency of manual tuning of the four hyperparameters, the GGA module employs a "greedy strategy + genetic algorithm" to achieve automatic optimization, balancing model accuracy and computational resources. The specific process is as follows: (1) Optimization objectives and constraints The hyperparameter tuning is transformed into a multi-objective optimization problem, with the objective function being:

[0048] in, For student model parameters, The constraint is the classification accuracy of the student model on the validation set:

[0049] (2) GGA core strategy 1. Delay optimization mechanism: The total number of training rounds is set to ,exist Initiate tuning in the middle of model training to avoid tuning bias caused by early model instability; in the middle stage, the model performance tends to stabilize, and tuning at this time can more accurately find the optimal combination of hyperparameters while reducing the consumption of computing resources. 2. Greedy multi-point mutation: Gradient importance score: The absolute value of the gradient of the loss function with respect to each hyperparameter is calculated through backpropagation, and the normalized value is used as the hyperparameter importance score (the larger the absolute value of the gradient, the more significant the influence of the hyperparameter on the model). Targeted mutation: Prioritize fine-tuning of high-importance hyperparameters and make minor adjustments to low-importance hyperparameters to avoid performance fluctuations caused by random mutations; 3. Genetic Algorithm Flow: Initialization: Generate K sets of candidate hyperparameter solutions (satisfying the constraints); Selection: Calculate the fitness (e.g., validation set accuracy) of each solution set and retain the top 50% of the best solutions; Crossover: Perform crossover operations (such as single-point crossover) on the optimal solution to generate new candidate solutions; Mutation: A greedy strategy is used to perform targeted mutations on new solutions; Iteration: Repeat the selection-crossover-mutation process until the maximum number of iterations or precision convergence is reached, and output the optimal combination of hyperparameters.

[0050] Example I. Experimental Environment Configuration 1. Dataset Selection The effectiveness of the SCML method was validated using five publicly available RSSC benchmark datasets. Details of the network training parameters are shown in Table 1. Table 1 Network training parameters

[0051] Table 2 Hardware Resource Costs

[0052] Data partitioning: UC-Merced, WHU-RS19, RS-C11: 80% training, 20% testing; NWPU-RESISC45, AID: 20% training, 80% testing.

[0053] 2. Data Preprocessing Resize: All images are uniformly resized to 256×256; Training enhancement: Randomly crop to 224×224 and flip horizontally (to enhance data diversity); Test preprocessing: Center cropped to 224×224 (to ensure test stability).

[0054] 3. Model Selection Teacher-student model combination: ResNet50 (Teacher) → ResNet18 (Student);

[0055] like Figure 5 As shown, SCML outperforms existing methods on all datasets, with significant improvements on NWPU-RESISC45 (complex scenarios) and AID (large data volume), demonstrating its strong feature extraction capabilities.

[0056] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A teacher-supervised, peer-learning knowledge distillation-based spaceborne remote sensing scene classification method based on standardized contrastive learning, characterized in that... The specific process is as follows: Positive and negative sample pairs are generated and input into the teacher-student classification network. RobustScaler is used to standardize the output of the teacher-student classification network, and the SCL loss is calculated. ; Set up N lightweight student models, implement knowledge transfer between student models using KL divergence, and calculate their mutual learning loss. Set the projector to display the student model. Layer output features Mapping to the teacher model Layer features In the same dimension, ensure feature alignment and calculate bridge loss. The student model uses the teacher classifier for classification; SCL loss is included. Mutual learning loss Bridge losses and knowledge distillation cross-entropy loss The total TSML loss is calculated, and hyperparameters are set for each loss. The teacher-student classification network was trained based on the total loss of TSML. During training, each hyperparameter of the total loss of TSML was fine-tuned and optimized based on a greedy genetic algorithm. The trained teacher-student classification network was then used to classify spaceborne remote sensing scenes.

2. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 1, characterized in that, The specific process for generating positive and negative sample pairs is as follows: One sample is randomly selected as the "anchor sample," and a "same-category index pool" and a "different-category index pool" are constructed based on its category; one positive sample is sampled from the same-category index pool, and k negative samples are sampled from the different-category index pool to form a positive and negative sample pair. .

3. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 1, characterized in that, The hyperparameter tuning is transformed into a multi-objective optimization problem, with the objective function being: in, , , , For hyperparameters, For student model parameters, To measure the classification accuracy of the student model on the validation set, This represents the total number of student models; The constraints are: 。 4. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 3, characterized in that, The mutual learning loss for: in, Representing the With the The student model predicts the probability. and The KL divergence.

5. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 3, characterized in that, The bridge loss for: in, For projector functions, for Norm, The teacher model is represented by the first... Layer features, The student model is represented by the first... Layer output features.

6. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 3, characterized in that, The SCL loss for: SCL loss function By contrast loss True target cross-entropy loss and standardized distillation loss composition: Among them, comparative loss The similarity between positive and negative samples is used to guide feature differentiation. The loss was calculated based on the RobustScaler normalization results.

7. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 6, characterized in that, The RobustScaler standardization: in, The median of the logit vector. ,in It is the 75th percentile. It is the 25th quantile. This refers to the temperature parameter.

8. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 7, characterized in that, The standardized distillation loss is: in, The student model is logit. The representative teacher model is logit. This represents the softmax function.

9. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 3, characterized in that, When fine-tuning each hyperparameter of the total TSML loss using the greedy genetic algorithm, a delayed tuning mechanism is also incorporated: Let the total number of training rounds be set to... ,exist Wheel start-up optimization.

10. The teacher-supervised mutual learning knowledge distillation spaceborne remote sensing scene classification method based on standardized contrastive learning according to claim 9, characterized in that, When fine-tuning each hyperparameter of the total TSML loss using the greedy genetic algorithm, a greedy multi-point mutation mechanism is also incorporated, including: Gradient importance score: The absolute value of the gradient of the loss function with respect to each hyperparameter is calculated through backpropagation, and the normalized value is used as the hyperparameter importance score; Targeted mutation: Prioritize fine-tuning of high-importance hyperparameters and make minor adjustments to low-importance hyperparameters to avoid performance fluctuations caused by random mutations.

Images