Cross-model knowledge distillation method facing remote sensing semantic segmentation task

Abstract

The invention discloses a cross-model knowledge distillation method for a remote sensing semantic segmentation task, and relates to a cross-model knowledge distillation method. The method aims at solving the problems that an existing remote sensing image semantic segmentation model is large in parameter quantity and large in resource consumption. The method comprises the steps of selecting a student model and a teacher model which are complementary in ability, selecting a data set to train the teacher model, and using a feature map output by the trained teacher model as knowledge to be input into the student model; carrying out attention weighting on the feature map input into the student model according to channels, and calculating loss; the knowledge transmitted by the teacher model is decoupled into a target area and a non-target area, the two independent target areas are subjected to prediction and knowledge distillation, and loss is calculated; and carrying out weighted summation on the loss of the semantic segmentation of the student model and the loss of the above steps, calculating the total loss, updating the parameters of the student model through the total loss, and finally completing the training of the student model. The invention belongs to the technical field of remote sensing image processing.

Description

Technical Field

[0001] The invention relates to a cross-model knowledge distillation method and belongs to the technical field of remote sensing image processing. Background Art

[0002] Remote sensing image semantic segmentation is an important task in remote sensing image processing. The main goal is to classify each pixel in the remote sensing image, and then identify and extract different types of objects, such as vegetation, buildings, roads, water bodies, etc. This technology has a wide range of applications in land and resources monitoring, ecological environment assessment, urban planning, disaster prevention and mitigation, etc. With the rapid improvement of remote sensing data acquisition methods and resolution, remote sensing image semantic segmentation tasks face increasingly complex data processing requirements, and traditional methods are difficult to meet the requirements for efficient and accurate segmentation.

[0003] At present, deep learning has made significant progress in the field of semantic segmentation. Many deep learning-based models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), graph neural networks (GNN), Transformer, Mamba, etc., have achieved good results in semantic segmentation tasks. However, due to the particularity of remote sensing data, these models still face some challenges.

[0004] First, remote sensing images have high resolution and complex scene features, often containing objects of various scales, and the distribution of object categories is highly unbalanced. High-resolution images increase the computational complexity of the model, and the morphological differences of objects in large-scale scenes are large, which brings difficulties to segmentation accuracy. Secondly, the amount of data in remote sensing images is usually very large, and traditional deep learning models require a lot of computing resources for training and reasoning, which is difficult to apply to actual tasks with limited resources.

[0005] In order to improve the efficiency and accuracy of remote sensing semantic segmentation and reduce the dependence on computing resources, the knowledge distillation method has gradually attracted attention. Knowledge distillation is a technology that transfers the knowledge of a complex model (teacher model) to a lightweight model (student model) to reduce the computational overhead of the model and improve its performance in practical applications. This method distills the output or intermediate layer features of the teacher model so that the student model can achieve performance similar to that of the teacher model at a lower complexity.

[0006] Although knowledge distillation technology has made some progress in general computer vision tasks, its application in remote sensing semantic segmentation is still in the initial exploration stage. Most current research focuses on knowledge transfer between homogeneous models. However, when facing multi-source heterogeneous remote sensing data, cross-model knowledge distillation technology has not been fully studied. Therefore, developing a distillation method that can effectively transfer knowledge across models can not only maintain the efficiency of the model under multimodal data, but also reduce the dependence on computing resources, which has important research significance and application value. Summary of the invention

[0007] In order to solve the problems of large number of parameters and high resource consumption of existing remote sensing image semantic segmentation models, the present invention proposes a cross-model knowledge distillation method for remote sensing semantic segmentation tasks.

[0008] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:

[0009] Step 1: Select a student model and a teacher model with complementary capabilities, select a data set to train the teacher model, and use the feature map output by the trained teacher model as knowledge input to the student model;

[0010] Step 2: Weight the feature map input to the student model in step 1 by channel, assign different channel ratios to the attention mapping operation, assign corresponding weights to each layer of the feature map, and calculate the loss of step 2;

[0011] Step 3: Decouple the knowledge transferred by the teacher model in step 1 into the target area and the non-target area, perform prediction and knowledge distillation on these two independent target areas, and calculate the loss of step 3;

[0012] Step 4: Take the weighted sum of the semantic segmentation loss of the student model and the losses of steps 2 and 3 to calculate the total loss. Update the parameters of the student model through the total loss, and finally complete the training of the student model.

[0013] Furthermore, the specific process of step 1 is:

[0014] Step 101: According to the characteristics and requirements of the target task, a complementary student model and teacher model are selected; according to the complexity of the task and actual needs, a suitable data set is selected to ensure the diversity and representativeness of the data set; the teacher model is trained using the data set to obtain high-quality feature representations; in the process of training the teacher model, the structure and parameters of the model are repeatedly adjusted to ensure that its performance on a given task is optimal; the training process of the teacher model includes multiple rounds of optimization to maximize its ability to represent the input data;

[0015] Step 102: The images in the processed data set Input into the trained teacher model;

[0016] The teacher model first uses its encoder to extract features and maps the original input image layer by layer into multi-scale feature maps.

[0017] Each layer of the encoder performs a higher level of abstraction on the input data to capture structural information, texture features, and spatial context information in the image;

[0018] Subsequently, the decoder of the teacher model gradually restores these feature maps to the same resolution as the original image through upsampling operations, generating the output Logit map The Logit graph contains the prediction results for each pixel and is the final output of the teacher model for the input image. In this step, a loss function is defined to measure the difference between the teacher model's prediction and the true label. The loss function includes cross entropy loss or mean square error loss. The form of the loss function is:

[0019]

[0020] In formula (1), H represents the original image height, W represents the original image width, σ represents the softmax function, and CE represents the cross entropy loss. represents the pixel-by-pixel value of the image under training, A pixel-by-pixel value representing the true value;

[0021] Step 103: In the process of knowledge distillation, the key is to align the output probability distribution between the teacher model and the student model. Specifically, the Logit graph generated by the teacher model is processed and used as a target to guide the training of the student model. In this process, the pixel-level alignment loss of the two is calculated to ensure that the student model can fully learn the rich feature representation in the teacher model. The loss function of the above task is defined as:

[0022]

[0023] In formula (2), KL represents the Kullback-Leibler divergence between two probabilities, α is a hyperparameter weight, and τ represents the temperature coefficient;

[0024] Step 104: Freeze all hyperparameters of the teacher model to ensure that the teacher model remains stable during the training process. The hyperparameters of the student model are continuously updated through back propagation. The training goal of the student model is to minimize the above alignment loss and task loss until the loss function converges or reaches the preset performance indicator.

[0025] Furthermore, the specific process of step 2 is:

[0026] Step 201: The feature map output by the teacher model in step 1 After the spatial average pooling operation, it is squeezed into a feature of length C, and then it is restricted to the range of (0, 1) through the normalization function to obtain the channel attention embedding The mathematical expression of this process is:

[0027]

[0028] In formula (3), represents the feature map height, represents the feature map width, δ represents the activation function, Feature map;

[0029] Step 202: Multiply the channel attention embedding obtained in step 201 by the original feature map channel by channel to generate a weighted channel feature map, and perform normalization on the weighted feature map. The mathematical expression of this process is:

[0030]

[0031] In formula (4), represents the normalized feature map, Represents a feature map;

[0032] Step 203: After obtaining the weighted channel feature map, the channel feature map is further processed by a spatial average pooling layer to obtain a spatial attention map. The mathematical expression of this step is as follows:

[0033]

[0034] In formula (5), C represents the feature map dimension;

[0035] Step 204, by calculating the spatial attention map between the two model outputs Distance, to ensure that the spatial representation of the student model is consistent with the teacher model. The loss function of this step is defined as follows:

[0036]

[0037] In formula (6), represents the spatial attention map of the teacher model, Represents the spatial attention map of the student model.

[0038] Furthermore, the specific process of step 3 is as follows:

[0039] Step 301: In step 1, the Logit graph output by the teacher model By applying the argmax function, the score of the target category is selected from N categories; this process converts the Logit map into a hard label, that is, assigning a clear category label to each pixel; after using the hard label as the true value of the student model input, the label smoothing regularization strategy is introduced. The mathematical representation of the loss function is as follows:

[0040]

[0041] In formula (7), Logit plot representing the student model, represents the Logit plot of the teacher model, ζ represents the label smoothing operation;

[0042] Step 302: For knowledge distillation of non-target categories, Kullback-Leibler divergence is used to adjust the difference in pixel-by-pixel non-target category probabilities between the teacher model and the student model, where the loss function is expressed as:

[0043]

[0044] In formula (8), represents the soft category of the teacher model, represents the soft category of the student model;

[0045] Step 303: Perform a weighted summation of the loss functions calculated in step 31 and step 32 to construct the total loss function of step 3. The total loss function of step 3 is expressed as follows:

[0046]

[0047] In formula (9), λ1λ2 represent two weight hyperparameters.

[0048] Furthermore, the final total loss function in step 4 is:

[0049]

[0050] Finally, the training is completed and the student model after distillation is obtained.

[0051] The beneficial effects of the present invention are:

[0052] 1. This paper proposes a novel cross-model knowledge distillation framework, which aims to significantly improve the performance of remote sensing semantic segmentation by integrating the advantages of two models. The framework effectively extracts the complementary characteristics of the two models and the advantages of each model, so that the model can achieve more accurate segmentation effects when facing complex remote sensing scenes;

[0053] 2. The present invention effectively focuses on key area information by introducing spatial and channel attention mechanisms in the feature transfer process, improves the perception of multi-scale objects, and enhances the segmentation effect of complex backgrounds and mixed scenes; in addition, in the classification process, by decoupling the feature representation of targets and non-targets, the flexibility of the model in processing different types of objects is improved, avoiding the interference of non-target information, thereby enhancing the robustness and generalization ability of the model;

[0054] 3. The present invention not only improves the accuracy of semantic segmentation of remote sensing images, but also significantly reduces the dependence on computing resources, and is suitable for large-scale remote sensing data processing tasks; this method has strong adaptability and stability, and provides an efficient and accurate solution for the field of remote sensing semantic segmentation, with broad application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0056] Specific implementation method 1: Figure 1 As shown in Figure 1, a cross-model knowledge distillation method for remote sensing semantic segmentation tasks is presented. The specific steps include:

[0057] Step 1: Select a student model and a teacher model with complementary capabilities, select a data set to train the teacher model, and use the feature map output by the trained teacher model as knowledge input to the student model;

[0058] The specific process is:

[0059] Step 101: According to the characteristics and requirements of the target task, a complementary student model and teacher model are selected; according to the complexity of the task and actual needs, a suitable data set is selected to ensure the diversity and representativeness of the data set; the teacher model is trained using the data set to obtain high-quality feature representations; in the process of training the teacher model, the structure and parameters of the model are repeatedly adjusted to ensure that its performance on a given task is optimal; the training process of the teacher model includes multiple rounds of optimization to maximize its ability to represent the input data;

[0060] Step 102: The images in the processed data set Input into the trained teacher model;

[0061] The teacher model first uses its encoder to extract features and maps the original input image layer by layer into multi-scale feature maps.

[0062] Each layer of the encoder performs a higher level of abstraction on the input data to capture structural information, texture features, and spatial context information in the image;

[0063] Subsequently, the decoder of the teacher model gradually restores these feature maps to the same resolution as the original image through upsampling operations, generating the output Logit map The Logit graph contains the prediction results for each pixel and is the final output of the teacher model for the input image. In this step, a loss function is defined to measure the difference between the teacher model's prediction and the true label. The loss function includes cross entropy loss or mean square error loss. The form of the loss function is:

[0064]

[0065] In formula (1), H represents the original image height, W represents the original image width, σ represents the softmax function, and CE represents the cross entropy loss. represents the pixel-by-pixel value of the image under training, A pixel-by-pixel value representing the true value;

[0066] Step 103: In the process of knowledge distillation, the key is to align the output probability distribution between the teacher model and the student model. Specifically, the Logit graph generated by the teacher model is processed and used as a target to guide the training of the student model. In this process, the pixel-level alignment loss of the two is calculated to ensure that the student model can fully learn the rich feature representation in the teacher model. The loss function of the above task is defined as:

[0067]

[0068] In formula (2), KL represents the Kullback-Leibler divergence between two probabilities, α is a hyperparameter weight, and τ represents the temperature coefficient;

[0069] Step 104: Freeze all hyperparameters of the teacher model to ensure that the teacher model remains stable during the training process. The hyperparameters of the student model are continuously updated through back propagation. The training goal of the student model is to minimize the above alignment loss and task loss until the loss function converges or reaches the preset performance index.

[0070] Step 2: Weight the feature map input to the student model in step 1 by channel, assign different channel ratios to the attention mapping operation, assign corresponding weights to each layer of the feature map, and calculate the loss of step 2;

[0071] The specific process is:

[0072] Step 201: The feature map output by the teacher model in step 1 After the spatial average pooling operation, it is squeezed into a feature of length C, and then it is restricted to the range of (0, 1) through the normalization function to obtain the channel attention embedding The mathematical expression of this process is:

[0073]

[0074] In formula (3), represents the feature map height, represents the feature map width, δ represents the activation function, Feature map;

[0075] Step 202: Multiply the channel attention embedding obtained in step 201 by the original feature map channel by channel to generate a weighted channel feature map, and perform normalization on the weighted feature map. The mathematical expression of this process is:

[0076]

[0077] In formula (4), represents the normalized feature map, Represents a feature map;

[0078] Step 203: After obtaining the weighted channel feature map, the channel feature map is further processed by a spatial average pooling layer to obtain a spatial attention map. The mathematical expression of this step is as follows:

[0079]

[0080] In formula (5), C represents the feature map dimension;

[0081] Step 204, by calculating the spatial attention map between the two model outputs Distance, to ensure that the spatial representation of the student model is consistent with the teacher model. The loss function of this step is defined as follows:

[0082]

[0083] In formula (6), represents the spatial attention map of the teacher model, Represents the spatial attention map of the student model.

[0084] Step 3: Decouple the knowledge transferred by the teacher model in step 1 into the target area and the non-target area, perform prediction and knowledge distillation on these two independent target areas, and calculate the loss of step 3;

[0085] The specific process is:

[0086] Step 301: In step 1, the Logit graph output by the teacher model By applying the argmax function, the score of the target category is selected from N categories; this process converts the Logit map into a hard label, that is, assigning a clear category label to each pixel; after using the hard label as the true value of the student model input, the label smoothing regularization strategy is introduced. The mathematical representation of the loss function is as follows:

[0087]

[0088] In formula (7), Logit plot representing the student model, represents the Logit plot of the teacher model, ζ represents the label smoothing operation;

[0089] Step 302: For knowledge distillation of non-target categories, Kullback-Leibler divergence is used to adjust the difference in pixel-by-pixel non-target category probabilities between the teacher model and the student model, where the loss function is expressed as:

[0090]

[0091] In formula (8), represents the soft category of the teacher model, represents the soft category of the student model;

[0092] Step 303: Perform a weighted summation of the loss functions calculated in step 31 and step 32 to construct the total loss function of step 3. The total loss function of step 3 is expressed as follows:

[0093]

[0094] In formula (9), λ1λ2 represent two weight hyperparameters.

[0095] Step 4: Perform a weighted summation of the semantic segmentation loss of the student model and the losses of step 2 and step 3 to calculate the total loss. The parameters of the student model are updated by the total loss, and finally the student model is trained.

[0096] The final total loss function is:

[0097]

[0098] Finally, the training is completed and the student model after distillation is obtained.

[0099] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. A cross-model knowledge distillation method for remote sensing semantic segmentation tasks, characterized in that: The specific steps include: Step 1: Select a student model and a teacher model with complementary capabilities, select a data set to train the teacher model, and use the feature map output by the trained teacher model as knowledge input to the student model; Step 2: Weight the feature map input to the student model in step 1 by channel, assign different channel ratios to the attention mapping operation, assign corresponding weights to each layer of the feature map, and calculate the loss of step 2; Step 3: Decouple the knowledge transferred by the teacher model in step 1 into the target area and the non-target area, perform prediction and knowledge distillation on these two independent target areas, and calculate the loss of step 3; Step 4: Take the weighted sum of the semantic segmentation loss of the student model and the losses of steps 2 and 3 to calculate the total loss. Update the parameters of the student model through the total loss, and finally complete the training of the student model.

2. A cross-model knowledge distillation method for remote sensing semantic segmentation tasks according to claim 1, characterized in that: The specific process of step 1 is: Step 101: According to the characteristics and requirements of the target task, a complementary student model and teacher model are selected; according to the complexity of the task and actual needs, a suitable data set is selected to ensure the diversity and representativeness of the data set; the teacher model is trained using the data set to obtain high-quality feature representations; in the process of training the teacher model, the structure and parameters of the model are repeatedly adjusted to ensure that its performance on a given task is optimal; the training process of the teacher model includes multiple rounds of optimization to maximize its ability to represent the input data; Step 102: The images in the processed data set Input into the trained teacher model; The teacher model first uses its encoder to extract features and maps the original input image layer by layer into multi-scale feature maps. Each layer of the encoder performs a higher level of abstraction on the input data to capture structural information, texture features, and spatial context information in the image; Subsequently, the decoder of the teacher model gradually restores these feature maps to the same resolution as the original image through upsampling operations, generating the output Logit map The Logit graph contains the prediction results for each pixel and is the final output of the teacher model for the input image. In this step, a loss function is defined to measure the difference between the teacher model's prediction and the true label. The loss function includes cross entropy loss or mean square error loss. The form of the loss function is: In formula (1), H represents the original image height, W represents the original image width, σ represents the softmax function, and CE represents the cross entropy loss. represents the pixel-by-pixel value of the image under training, A pixel-by-pixel value representing the true value; Step 103: In the process of knowledge distillation, the key is to align the output probability distribution between the teacher model and the student model. Specifically, the Logit graph generated by the teacher model is processed and used as a target to guide the training of the student model. In this process, the pixel-level alignment loss of the two is calculated to ensure that the student model can fully learn the rich feature representation in the teacher model. The loss function of the above task is defined as: In formula (2), KL represents the Kullback-Leibler divergence between two probabilities, α is a hyperparameter weight, and τ represents the temperature coefficient; Step 104: Freeze all hyperparameters of the teacher model to ensure that the teacher model remains stable during the training process. The hyperparameters of the student model are continuously updated through back propagation. The training goal of the student model is to minimize the above alignment loss and task loss until the loss function converges or reaches the preset performance indicator.

3. The cross-model knowledge distillation method for remote sensing semantic segmentation tasks according to claim 1, characterized in that: The specific process of step 2 is: Step 201: The feature map output by the teacher model in step 1 After the spatial average pooling operation, it is squeezed into a feature of length C, and then it is restricted to the range of (0, 1) through the normalization function to obtain the channel attention embedding The mathematical expression of this process is: In formula (3), represents the feature map height, represents the feature map width, δ represents the activation function, Feature map; Step 202: Multiply the channel attention embedding obtained in step 201 by the original feature map channel by channel to generate a weighted channel feature map, and perform normalization on the weighted feature map. The mathematical expression of this process is: In formula (4), represents the normalized feature map, Represents a feature map; Step 203: After obtaining the weighted channel feature map, the channel feature map is further processed by a spatial average pooling layer to obtain a spatial attention map. The mathematical expression of this step is as follows: In formula (5), C represents the feature map dimension; Step 204, by calculating the spatial attention map between the two model outputs Distance, to ensure that the spatial representation of the student model is consistent with the teacher model. The loss function of this step is defined as follows: In formula (6), represents the spatial attention map of the teacher model, Represents the spatial attention map of the student model.

4. The cross-model knowledge distillation method for remote sensing semantic segmentation tasks according to claim 1, characterized in that: The specific process of step 3 is: Step 301: In step 1, the Logit graph output by the teacher model By applying the argmax function, the score of the target category is selected from N categories; this process converts the Logit map into a hard label, that is, assigning a clear category label to each pixel; after using the hard label as the true value of the student model input, the label smoothing regularization strategy is introduced. The mathematical representation of the loss function is as follows: In formula (7), Logit plot representing the student model, represents the Logit plot of the teacher model, ζ represents the label smoothing operation; Step 302: For knowledge distillation of non-target categories, Kullback-Leibler divergence is used to adjust the difference in pixel-by-pixel non-target category probabilities between the teacher model and the student model, where the loss function is expressed as: In formula (8), represents the soft category of the teacher model, represents the soft category of the student model; Step 303: Perform a weighted summation of the loss functions calculated in step 31 and step 32 to construct the total loss function of step 3. The total loss function of step 3 is expressed as follows: In formula (9), λ1λ2 represent two weight hyperparameters.

5. The cross-model knowledge distillation method for remote sensing semantic segmentation tasks according to claim 1, characterized in that: The final total loss function in step 4 is: Finally, the training is completed and the student model after distillation is obtained.

Images