Remote sensing image small target feature enhancement refined classification method based on dynamic mixed knowledge distillation

Abstract

The invention discloses a remote sensing image small target feature enhancement refined classification method based on dynamic mixed knowledge distillation, and the method comprises the steps: inputting a low-resolution remote sensing image of a to-be-classified small target into a pre-trained classification model, and obtaining the specific model information of the target; the classification model comprises a trunk low-resolution image student network branch, and in the training stage of the classification model, an auxiliary high-resolution image teacher network branch consistent with the student network branch in structure and a dynamic mixed knowledge distillation module are introduced, the dynamic mixed knowledge distillation module is deployed between a trunk low-resolution image student network branch and an auxiliary high-resolution image teacher network branch, and high-resolution image intermediate features and output features learned by the teacher network branch are used as priori knowledge to be distilled to the student network branch through the dynamic mixed knowledge distillation module; and training of the student network branches is guided, so that the feature representation capability of the student network branches is enhanced, and a trained classification model is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation. Background Art

[0002] The refined classification of targets such as airplanes and vehicles in visible light remote sensing images to identify and confirm the specific models of the targets can provide more detailed and valuable intelligence information. Compared with general classification tasks, refined classification tasks need to distinguish the local subtle differences between different model targets, which puts higher requirements on image quality and feature extraction algorithms. In recent years, using the powerful feature representation and extraction capabilities of deep learning algorithms, fruitful results have been achieved in the research on refined classification of high-resolution remote sensing image targets. Compared with high-resolution images, targets such as airplanes and vehicles in low-resolution images only occupy a small number of pixels and degenerate into small and weak targets with serious loss of identification detail information.

[0003] Using the high-quality and rich target detail information contained in high-resolution remote sensing images as prior knowledge to enhance the information of low-resolution remote sensing images is an important means to compensate for the lack of identification detail information of small and weak targets in low-resolution images. According to different information enhancement methods, the existing refined classification methods for small targets in low-resolution remote sensing images are mainly divided into two categories: super-resolution-based methods and knowledge distillation-based methods.

[0004] (1) Super-resolution-based methods

[0005] Super-resolution-based methods directly perform enhancement in the image domain and are widely used in remote sensing images. The low-resolution image is first reconstructed into a corresponding high-resolution image through super-resolution technology, and then refined classification is performed. This cascaded heavy model has a large amount of calculation and is difficult to optimize. The complicated super-resolution process leads to a reduction in overall efficiency, and its ill-posedness makes the detail reconstruction results diverse, which easily leads to inaccurate identification of target models for refined classification problems.

[0006] (2) Knowledge distillation-based methods

[0007] Knowledge distillation was originally proposed for model compression, and this knowledge transfer mechanism was later also applied to model performance improvement. Specifically, first, the teacher network is trained on high-resolution images; then, the prior knowledge obtained by the teacher network is distilled to guide the student network to train on low-resolution images. The student network acquires capabilities similar to the teacher network through the above-mentioned imitation learning. Knowledge distillation has become a mainstream method in matching tasks such as face verification and face recognition in low-resolution natural images. In the field of remote sensing images, it is mainly used for small target detection in low-resolution remote sensing images, and its potential in refined classification remains to be explored. Knowledge distillation can be divided into three categories according to the type of knowledge: output feature knowledge distillation, relationship feature knowledge distillation, and intermediate feature knowledge distillation. Output feature knowledge distillation constrains the probability distribution of the student network's final prediction to match that of the teacher. Relation feature knowledge distillation enables the student network to learn the relationship between multiple samples from the teacher network. Due to the gap between the apparent information content of targets in high-resolution and low-resolution remote sensing images, distilling output features and relationship features cannot effectively narrow the gap between the student network and the teacher network in target representation capabilities.

[0008] Distillation of intermediate feature knowledge minimizes the difference between the feature maps of the student and teacher networks in the early stage, which enables the low-resolution student network to directly imitate and learn the feature expression of the high-resolution teacher network. Subsequent methods further extract the attention maps of the student and teacher network features as intermediate features, and encourage the student network to focus on the spatial location that the teacher network focuses on by narrowing the difference between the student and teacher network attention maps. However, the attention map can only reflect the location information of the key discriminant area, and it is still difficult to compensate for the missing target detail identification features in low-resolution remote sensing images. As a result, this type of method has the problem that the intermediate features distilled to the student network have limited representation capabilities for the detail difference information between target categories. In addition, existing methods have the limitation of poor adaptability in the collaborative auxiliary mechanism between distillation tasks and classification tasks.

[0009] In summary, the current method for fine-grained classification of small targets in remote sensing images based on knowledge distillation has problems such as the limited representation ability of the intermediate features distilled to the student network for the detailed difference information between target categories and the poor auxiliary adaptability of the distillation task to the fine-grained classification task, which leads to limited improvement in the fine-grained classification performance. It is difficult to meet the needs of high-precision model identification of weak and small targets under low-resolution remote sensing observation conditions. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the prior art and propose a remote sensing image small target feature enhancement and refinement classification method based on dynamic hybrid knowledge distillation. This method can improve the refinement classification accuracy of small targets such as aircraft and vehicles under low-resolution remote sensing observation conditions, thereby enhancing the application value of remote sensing systems in military reconnaissance, urban traffic monitoring and other fields.

[0011] A refined classification method for enhancing small target features in remote sensing images based on dynamic hybrid knowledge distillation, including:

[0012] Input the low-resolution remote sensing image of the small target to be classified into a pre-trained classification model to obtain the specific model information of the target, that is, to achieve refined classification;

[0013] The classification model includes a backbone low-resolution image student network branch. During the training stage of the classification model, an auxiliary high-resolution image teacher network branch with the same structure as the student network branch and a dynamic hybrid knowledge distillation module are introduced. Among them, the dynamic hybrid knowledge distillation module is deployed between the backbone low-resolution image student network branch and the auxiliary high-resolution image teacher network branch. Through the dynamic hybrid knowledge distillation module, the intermediate features and output features of the high-resolution image learned by the teacher network branch are distilled as prior knowledge into the student network branch to guide the training of the student network branch, thereby enhancing the feature representation ability of the student network branch and obtaining a trained classification model.

[0014] Preferably, the backbone low-resolution image student network branch uses the ResNet-50 residual network as the basic network model structure, including a convolutional feature extraction module and a classifier. Among them,

[0015] The convolutional feature extraction module is used to perform convolutional feature extraction on the input remote sensing image I LR to generate a feature map C5;

[0016] The classifier is used to perform refined classification of the target according to the feature map C5 and output the prediction result Cls of the target model category.

[0017] Preferably, the convolutional feature extraction module includes: 1 basic convolutional layer Conv1 and 4 convolutional layers based on residual learning, Conv2_x, Conv3_x, Conv4_x, and Conv5_x; among them, Conv2_x includes 1 3×3 max-pooling layer and 3 residual blocks, Conv3_x includes 4 residual blocks, Conv4_x includes 6 residual blocks, and Conv5_x includes 3 residual blocks. Inside each residual block, the output of the input feature processed by the stacked convolutional layers is added to the input feature via a skip connection as the output feature of the residual block.

[0018] Preferably, the classifier includes 1 average pooling layer and 1 fully connected layer, and the number of neurons in the fully connected layer is the same as the total number of refined categories.

[0019] Preferably, the dynamic hybrid knowledge distillation module includes: an intermediate feature knowledge distillation unit, an output feature knowledge distillation unit, and a dynamic hybrid knowledge distillation unit; among them,

[0020] The intermediate feature knowledge distillation unit is used to generate channel and spatial weight masks for the intermediate feature maps f of the teacher network branch through an attention mechanism, so as to weight the intermediate feature differences between the student network branch and the teacher network branch, and construct an intermediate feature knowledge distillation loss function; HR , and generate channel and spatial weight masks through an attention mechanism to weight the intermediate feature differences between the student network branch and the teacher network branch, and construct an intermediate feature knowledge distillation loss function;

[0021] The output feature knowledge distillation unit is used to construct an output feature knowledge distillation loss function to guide the training of the student network branch;

[0022] The dynamic hybrid knowledge distillation unit is used to construct a dynamic hybrid knowledge distillation loss function to dynamically weight and calculate the intermediate feature and output feature knowledge distillation losses.

[0023] Preferably, the intermediate feature knowledge distillation loss function L feat is:

[0024]

[0025] where C, H, and W represent the number of channels, the height, and the width of the image respectively; represents the teacher network feature map of the k-th channel in the i-th row and j-th column, represents the student network feature map of the k-th channel in the i-th row and j-th column; represents the channel weight mask, represents the spatial weight mask.

[0026] Preferably, the output feature knowledge distillation loss function L logit is:

[0027] L logit = D KL [σ(z HR / τ), σ(z LR / τ)]

[0028] where D KL (·,·) represents the KL divergence, z HR and z LR are the logarithmic outputs of the teacher network branch and the student network branch respectively, τ is the distillation temperature, and σ(·) represents the softmax function.

[0029] Preferably, the dynamic hybrid knowledge distillation loss function is:

[0030]

[0031] where p represents the iteration stage of the current training, and are the intermediate feature knowledge distillation losses at the iteration stage p and the output feature knowledge distillation loss weight;

[0032]

[0033] wherein, represents gradient calculation, is the cross-entropy loss of the refined classification task itself.

[0034] Preferably, the method further includes the training process of the classification model, including:

[0035] Obtain the MAR20 remote sensing image military aircraft target recognition dataset, crop the aircraft targets in the images according to their horizontal frames to generate single-target image slices, and then obtain the refined categories including aircraft models according to the given division results of their airport images and assign them to the training set;

[0036] Take the images in the training set as high-resolution images, and then obtain the corresponding low-resolution images through bicubic interpolation, so as to form high-low resolution image pairs and corresponding target model type labels;

[0037] Set the batch data volume B, the total number of training iterations T, the learning rate strategy and the optimizer strategy;

[0038] Input the low-resolution images in the high-low resolution image pairs in the training set into the classification model introduced with the auxiliary high-resolution image teacher network branch and the dynamic hybrid knowledge distillation module for training until the training requirements are met, and obtain the trained backbone low-resolution image student network branch, that is, the classification model.

[0039] Preferably, the total loss function in the training process is:

[0040]

[0041] where p represents the current training iteration stage, is the cross-entropy loss of the refined classification task itself, is the dynamic hybrid knowledge distillation loss.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] (1) The present invention proposes a refined classification method for small target feature enhancement of remote sensing images based on dynamic hybrid knowledge distillation. The core innovation of this method lies in the proposal of a dynamic hybrid knowledge distillation method (DHKD) to enhance the features of the target remote sensing images, thereby improving the refined classification accuracy of small targets. Specifically, on the basis of the backbone low-resolution remote sensing image small target refined classification - student network branch, an auxiliary high-resolution image teacher network branch is introduced. By constructing a dynamic hybrid knowledge distillation module between the student and teacher network branches, the intermediate features and output features of the high-resolution images learned by the teacher network are distilled as prior knowledge into the student network (i.e., the target network) to guide its training, so as to enhance the feature representation ability of the student network and further improve its small target refined classification performance;

[0044] (2) The present invention designs an intermediate feature knowledge distillation module based on attention guidance. In the teacher network, the attention mechanism is used to focus on the target key identification feature regions in the intermediate features, generate channel and spatial weight masks, perform weighted processing on the differences between the intermediate features of the student and teacher networks, calculate the intermediate feature knowledge distillation loss, and guide the training of the student network, so as to enhance the ability of the intermediate features of the student network to represent the detailed difference information between target categories;

[0045] (3) The present invention proposes a dynamic hybrid knowledge distillation loss calculation method based on optimized direction consistency. The weight of the intermediate feature (output feature) knowledge distillation loss is measured by the degree of consistency between the intermediate feature (output feature) knowledge distillation and the refined classification optimization direction, and dynamic weighted processing is performed on the intermediate feature and output feature knowledge distillation losses to calculate the final knowledge distillation loss, so as to enhance the adaptability of the student network to the changes in knowledge requirements during the iterative training process and alleviate the problem that the distillation task of the existing method has poor auxiliary adaptability to the refined classification task.

[0046] (4) Through comparative experiments on the constructed MAR20 dataset, it is shown that the method of the present invention has a significant advantage in the refined classification accuracy of small targets of remote sensing images with absolute sizes of 32, 20, and 12 pixels compared with existing advanced knowledge distillation methods. Brief Description of the Drawings

[0047] Figure 1 is a schematic flow chart of the refined classification method for small target feature enhancement of remote sensing images based on dynamic hybrid knowledge distillation of the present invention;

[0048] Figure 2 is a structural diagram of the refined classification network model for small target feature enhancement of remote sensing images based on dynamic hybrid knowledge distillation of the present invention;

[0049] Figure 3 is a structural diagram of ResNet-50 adopted by the student network and teacher network of the present invention;

[0050] Figure 4 It is the network structure diagram of the intermediate feature knowledge distillation module based on attention guidance of the present invention. Detailed implementation manners

[0051] The present invention proposes a method for calculating the dynamic hybrid knowledge distillation loss based on the optimization direction consistency. By measuring the weight of the intermediate feature (output feature) knowledge distillation loss according to the degree of consistency between the intermediate feature (output feature) knowledge distillation and the refined classification optimization direction, the intermediate feature and output feature knowledge distillation losses are dynamically weighted to calculate the final knowledge distillation loss, so as to enhance the adaptability of the student network to the change of knowledge demand during the iterative training process.

[0052] The present invention discloses a method for enhancing the fine classification of small target features in remote sensing images based on dynamic hybrid knowledge distillation. On the basis of the backbone low-resolution remote sensing image small target fine classification - student network branch, an auxiliary high-resolution image teacher network branch is introduced. By constructing a dynamic hybrid knowledge distillation module between the student and teacher network branches, the intermediate features and output features of the high-resolution images learned by the teacher network are distilled into the student network as prior knowledge to guide its training, so as to enhance the feature representation ability of the student network and further improve its small target fine classification performance.

[0053] The technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0054] Embodiment 1

[0055] As Figure 1 shown, Embodiment 1 of the present invention proposes a method for enhancing the fine classification of small target features in remote sensing images based on dynamic hybrid knowledge distillation. Specifically, it includes: receiving the remote sensing image to be classified, inputting it into the pre-established and trained network model for enhancing the fine classification of small target features in remote sensing images based on dynamic hybrid knowledge distillation, and obtaining the target fine classification result, that is, the specific model information of the target;

[0056] This embodiment includes 4 steps: establishing a model for enhancing the fine classification of small target features in remote sensing images based on dynamic hybrid knowledge distillation, constructing a sample data set, training the model, and testing and verifying the model.

[0057] The first step: model establishment;

[0058] The model structure for enhancing the fine classification of small target features in remote sensing images based on dynamic hybrid knowledge distillation is as Figure 2As shown. The model specifically includes two network branches: the backbone low-resolution image student network and the auxiliary high-resolution image teacher network, as well as a dynamic hybrid knowledge distillation module between the two. The backbone low-resolution image student network branch uses ResNet-50 as the basic network model structure, which is specifically composed of a convolutional feature extraction module and a classifier. The auxiliary high-resolution image teacher network uses the same model structure as the backbone low-resolution image student network. A dynamic hybrid knowledge distillation module is constructed between the student network and the teacher network branch to calculate the dynamic hybrid knowledge distillation loss of intermediate features and output features, and distill the intermediate features and output features of high-resolution images learned by the teacher network into the student network as prior knowledge to guide its training, so as to enhance the feature representation ability of the student network and then improve its fine-grained classification performance for small targets.

[0059] In the present invention, it aims to predict the fine-grained classification information of the target, that is, the specific model information of the target, for the input target remote sensing image I.

[0060] The method specifically includes:

[0061] S1, the backbone low-resolution image student network branch

[0062] The backbone low-resolution image student network of the present invention is the core component of the entire fine-grained classification model for small target features of remote sensing images based on dynamic hybrid knowledge distillation. This student network aims to achieve the fine-grained classification ability for low-resolution images and is the target network of the present invention, running independently in the model inference stage.

[0063] The student network uses a residual network (Residual Network, ResNet). ResNet consists of a series of models with different depths, such as 18 layers, 34 layers, 50 layers, 101 layers, and 152 layers, providing choices for different application scenarios. Considering scale and performance comprehensively, the present invention selects the 50-layer model ResNet-50 with 25.6M parameters and 4.1G FLOPs as the student network. The overall structure of the ResNet-50 model is as Figure 3 shown, consisting of a convolutional feature extraction module and a classifier Classifier.

[0064] 1.1 Convolutional feature extraction module

[0065] Perform convolutional feature extraction on the input remote sensing image I LR to generate a feature map C5, as Figure 3 shown. Specifically, it includes:

[0066] The convolutional feature extraction module consists of 1 basic convolutional layer Conv1 and 4 convolutional layers Conv2_x, Conv3_x, Conv4_x, and Conv5_x based on residual learning. The basic convolutional layer Conv1 is composed of a 7×7 Conv2d with 64 output channels. The convolutional layers based on residual learning are mainly composed of residual blocks. Within each residual block, the output of the input feature processed by the stacked convolutional layers is added to the input feature via a skip connection as the output feature of the residual block. The calculation function of the residual block is defined as follows:

[0067]

[0068] where x and y represent the input and output features of the residual block respectively, represents the output of the stacked convolutional layers with layer parameters {W i}, and W s represents the linear mapping. In the skip connection, the input feature first passes through this linear mapping to make its dimension consistent with the dimension of the output of the convolutional layer, and then is added to the output of the convolutional layer. The skip connection of the residual block enables the network to learn the residual between the input and output, effectively ensuring the backpropagation of gradients and solving the problems of gradient disappearance and gradient explosion that hinder training when the network is deepened.

[0069] The specific structures of the convolutional layers based on residual learning are as follows: Conv2_x consists of 1 3×3 max pooling layer and 3 residual blocks. Each residual block is composed of a 1×1 Conv2d with 64 output channels, a 3×3 Conv2d with 64 output channels, and a 1×1 Conv2d with 256 output channels; Conv3_x consists of 4 residual blocks. Each residual block is composed of a 1×1 Conv2d with 128 output channels, a 3×3 Conv2d with 128 output channels, and a 1×1 Conv2d with 512 output channels; Conv4_x consists of 6 residual blocks. Each residual block is composed of a 1×1 Conv2d with 256 output channels, a 3×3 Conv2d with 256 output channels, and a 1×1 Conv2d with 1024 output channels; Conv5_x consists of 3 residual blocks. Each residual block is composed of a 1×1 Conv2d with 512 output channels, a 3×3 Conv2d with 512 output channels, and a 1×1 Conv2d with 2048 output channels.

[0070] The input of the convolutional feature extraction module is the remote sensing image I LR , and the output is the feature map C5, which can be expressed as:

[0071] C5 = Conv(I LR )

[0072] 1.2 Classifier

[0073] For the feature map C5 obtained by the above convolutional feature extraction module, target fine-grained classification is performed, and finally the predicted result Cls of the target model category is output, as Figure 3 shown; specifically including:

[0074] The classifier Classifier consists of 1 average pooling layer and 1 fully connected layer. The number of neurons in the fully connected layer is the same as the total number of fine-grained categories. The output of each neuron in the fully connected layer is mapped to a category probability with a value range of [0,1] through softmax. The category corresponding to the highest probability is the category predicted by the classifier. The classifier Classifier can be expressed as,

[0075] Cls = Classifier(C5) = max(softmax(FC(AvgPool(C5))))

[0076] where AvgPool(·) represents the average pooling layer and FC(·) represents the fully connected layer.

[0077] S2, Auxiliary high-resolution image teacher network branch

[0078] Based on the backbone low-resolution image student network branch of the present invention, an auxiliary high-resolution image teacher network branch is introduced. By distilling the intermediate features and output features of the high-resolution image learned by the teacher network as prior knowledge into the student network to guide its training, the feature representation ability of the student network is enhanced, and thus its target fine-grained classification performance is improved.

[0079] The model structure of the auxiliary high-resolution image teacher network is the same as that of the above backbone low-resolution image student network.

[0080] This module is only used in the training phase and does not participate in model inference.

[0081] S3, Dynamic hybrid knowledge distillation module

[0082] The present invention constructs a dynamic hybrid knowledge distillation module between the backbone low-resolution image student network branch and the auxiliary high-resolution image teacher network branch to calculate the dynamic hybrid knowledge distillation loss of the intermediate features and output features, as Figure 2As shown in the figure, the intermediate features and output features learned by the teacher network are distilled into the student network as prior knowledge to guide its training, so as to enhance the feature representation ability of the student network and then improve its fine-grained classification performance for small targets. First, a method for distilling intermediate feature knowledge guided by attention is proposed, and a corresponding intermediate feature knowledge distillation module is designed. In the teacher network, the attention mechanism is used to focus on the target key identification feature regions in the intermediate features, generate channel and spatial weight masks, perform weighted processing on the differences in the intermediate features between the student and teacher networks, and calculate the intermediate feature knowledge distillation loss. Second, an output feature knowledge distillation module based on soft targets is introduced. By constraining the probability distribution prediction of the final output of the student network to match that of the teacher network, a knowledge distillation loss function is constructed, and the output feature knowledge distillation loss is calculated. Finally, a dynamic hybrid knowledge distillation loss calculation method is proposed to perform dynamic weighted processing on the intermediate feature and output feature knowledge distillation losses based on the consistency of the optimization direction, and calculate the final knowledge distillation loss to flexibly adapt to the changes in the knowledge requirements of the student network during the iterative training process.

[0083] This module is only used in the training phase and does not participate in model inference.

[0084] The dynamic hybrid knowledge distillation module proposed by the present invention consists of three parts: intermediate feature knowledge distillation, output feature knowledge distillation, and dynamic hybrid knowledge distillation, and specifically includes:

[0085] 3.1. Intermediate feature knowledge distillation

[0086] The present invention designs an intermediate feature knowledge distillation module guided by attention, as Figure 4 shown in the figure. In the teacher network, the attention mechanism is used to focus on the target key identification feature regions in the intermediate features, generate channel and spatial weight masks, perform weighted processing on the differences in the intermediate features between the student and teacher networks, and calculate the intermediate feature knowledge distillation loss to guide the training of the student network, so as to enhance the representation ability of the intermediate features of the student network for the detailed difference information between target categories.

[0087] Given a pair of high- and low-resolution target image pairs (x HR , x LR ), the teacher network and the student network with the same structure respectively extract the features of the high-resolution image and the low-resolution image. We select the feature maps f HR and f LR output by the convolutional feature extraction modules of the teacher network and the student network as the basis for intermediate feature knowledge distillation. They have the same number of channels C, height H, and width W, and H = W in the selected feature maps.

[0088] For the intermediate feature map f HR, the channel and spatial weight masks are generated through the attention mechanism to weight the intermediate feature differences between the student and teacher networks, and an intermediate feature knowledge distillation loss function is constructed. The calculation methods of the channel and spatial weight masks for the intermediate feature map of the teacher network are as follows.

[0089] (1) Calculation of channel weight mask

[0090] The convolution calculation results corresponding to each channel of the feature map reflect the responses to different local patterns of the target. The refined classification teacher network trained on high-resolution images can accurately capture discriminative local patterns, corresponding to the feature map channels with higher response values. We perform global average pooling on the feature map f HR of the teacher network to obtain statistical information in the channel dimension, and then calculate the proportion of the pooling value of each channel in all channels as the weight of this channel to generate the channel weight mask M c The values of each dimension are calculated as follows.

[0091]

[0092] Among them, is the value of the q-th dimension of the channel weight mask, reflecting the weight of the q-th channel, and GAP(·) represents global average pooling.

[0093] (2) Calculation of spatial weight mask

[0094] The spatial positions with higher response values in each channel of the feature map reveal the receptive fields of the local patterns corresponding to this channel in the input image. For each channel of the feature map f HR of the teacher network, we calculate its following correlation matrices in the horizontal and vertical directions respectively:

[0095]

[0096] Among them, is the k-th channel of the teacher network feature map, and its horizontal and vertical correlation matrices are respectively represented as and

[0097] Then, according to the spatial dimension statistical information contained in the above correlation matrices, the horizontal spatial weight mask and the vertical spatial weight mask of each channel are generated as follows:

[0098]

[0099] Among them, represents the i-th row of represents The j-th column of Rows with a high degree of correlation reveal the receptive fields of the rows where key structures co-occur in the vertical direction. This area is a strip composed of multiple rows of pixels, which can accommodate a certain degree of deviation of these structures in the vertical direction caused by the target parking direction. Similarly, The columns of enhance the structures that co-occur in the horizontal direction.

[0100] The spatial weight masks of each channel of the feature map are the means of the above horizontal and vertical spatial weight masks. The final spatial weight mask is formed by concatenating the spatial weights of each channel as follows:

[0101]

[0102] where [·,·] represents the concatenation operation.

[0103] The channel and spatial weight masks generated through the above calculations are used to weight the intermediate feature differences between the student and teacher networks, and the intermediate feature knowledge distillation loss is calculated to guide the training of the student network.

[0104] The intermediate feature knowledge distillation loss function based on attention guidance is defined as follows:

[0105]

[0106] where C, H, and W represent the number of channels, the height, and the width of the image, respectively; represents the feature map of the teacher network at the k-th channel, the i-th row, and the j-th column, represents the feature map of the student network at the k-th channel, the i-th row, and the j-th column; represents the channel weight mask, represents the spatial weight mask.

[0107] 3.2. Output Feature Knowledge Distillation

[0108] Based on the above attention-guided intermediate feature knowledge distillation module, we further introduce an output feature knowledge distillation module based on soft targets. By constraining the probability distribution prediction of the final output of the student network to match that of the teacher network, an output feature knowledge distillation loss function is constructed to guide the training of the student network, so as to achieve a more comprehensive learning of the teacher network's capabilities by the student network. The output feature knowledge distillation loss function is defined as follows:

[0109] L logit = D KL [σ(z HR / τ), σ(z LR / τ)](9)

[0110] where DKL (·,·) represents the KL divergence, and z HR and z LR are the logarithmic outputs of the teacher and student networks respectively, τ is the distillation temperature, and σ(·) represents the softmax function.

[0111] 3.3. Dynamic Hybrid Knowledge Distillation

[0112] The present invention proposes a dynamic hybrid knowledge distillation method, as Figure 2 shown, dynamically weights and calculates the knowledge distillation losses of intermediate features and output features to flexibly adapt to the changes in the knowledge requirements of the student network during the iterative training process.

[0113] The dynamic hybrid knowledge distillation loss function is defined as follows:

[0114]

[0115] where p represents the iterative stage of the current training, and are the weights of the intermediate feature knowledge distillation loss and the output feature knowledge distillation loss respectively.

[0116] The weight of the intermediate feature (output feature) knowledge distillation loss is measured by the degree of consistency between the intermediate feature (output feature) knowledge distillation and the refined classification optimization direction. Specifically, it is calculated by the gradient cosine similarity between the intermediate feature (output feature) knowledge distillation loss and the refined classification task loss to enhance the adaptability of the student network to the changes in knowledge requirements during the iterative training process.

[0117] Specifically, during the p-th iteration of the current training, the weight of the intermediate feature (output feature) knowledge distillation loss is calculated as follows:

[0118]

[0119] where represents gradient calculation, is the cross-entropy loss of the refined classification task itself. If the cosine similarity of the loss function gradient is positive, it indicates that in the current iteration round, the knowledge distillation and the classification task have similar optimization directions, and then this cosine similarity is used as the weight of the knowledge distillation loss. If the cosine similarity is zero or negative, it indicates that in the current iteration round, the knowledge distillation and the classification task have orthogonal or opposite optimization directions, which is irrelevant to the improvement of classification performance or even has a negative effect, then the weight of the knowledge distillation loss is set to zero.

[0120] Step 2: Construct the sample data set;

[0121] The present invention constructs a training sample data set for the proposed refined classification network model for enhancing small target features in remote sensing images based on the existing publicly available MAR20 remote sensing image military aircraft target recognition data set.

[0122] The MAR20 data set is constructed from 3,842 high-resolution remote sensing image data collected at 60 military airports around the world by Google Earth, and contains 22,341 target instances of 20 aircraft models (i.e., refined categories). We crop the aircraft targets in the images according to their horizontal bounding boxes to generate single-target image slices, and then, according to the given division results of the airport images where they are located, assign them to the training set or the test set, obtaining a total of 7,870 training set images and 14,471 test set images, and then further randomly select 10% of the images of each category from the training set as the validation set. We use these single-target image slices as high-resolution images, and then obtain the corresponding low-resolution images through bicubic interpolation, thus forming high-low resolution image pairs.

[0123] The present invention follows the common method of defining small low-resolution targets based on pixels, and follows the sizes of small targets in remote sensing images adopted by previous work, generating small targets with absolute sizes of 32, 20, and 12 pixels respectively during the above degradation process. The absolute size of the target is calculated according to the width w and height h of its oriented bounding box.

[0124] Step 3: Model training;

[0125] S1, Initialize the refined classification network model for enhancing small target features in remote sensing images based on dynamic hybrid knowledge distillation, including setting the initialization parameters and training parameters of the network model, determining the loss function, and selecting the optimizer.

[0126] S2, Train the model. According to the above initialized network model, train the entire network model, and finally select the optimal network model.

[0127] In this embodiment, the random initialization method is used to initialize the weights of all convolutional layers of the entire refined classification network model;

[0128] The training parameters of the refined classification network model that need to be set in this embodiment mainly include: specifying the paths of the training data set and the validation data set, the batch data volume B, the total number of training iterations T, the learning rate strategy, and the optimizer strategy. In this embodiment, the batch data volume B is set to 64; the total number of training iterations T is set to 300; the learning rate strategy includes decaying by a factor of 10 every 60 iterations; the optimizer strategy includes using the stochastic gradient descent optimizer with weight decay, the initial learning rate is 0.001, the momentum is set to 0.9, and the weight decay is 5×10 -5Used to perform weight decay on all parameters to avoid overfitting.

[0129] The present invention designs a loss function to iteratively optimize the proposed refined classification network model for small target feature enhancement in remote sensing images based on dynamic hybrid knowledge distillation. The total loss of the target refined classification network model includes the refined classification loss and the dynamic hybrid knowledge distillation loss. The specific calculation function is defined as follows:

[0130]

[0131] Among them, p represents the current training iteration stage, is the cross-entropy loss of the refined classification task itself, is the dynamic hybrid knowledge distillation loss.

[0132] Step 4, model test and verification;

[0133] For the above-mentioned refined classification network model for small target feature enhancement in remote sensing images obtained by training, the test set in the above-mentioned constructed dataset is used to test and verify the performance of the model. A set of test data in the test set consists of 1 input low-resolution image and the corresponding target model category label. Input the input image in a set of test data into the above-mentioned trained model, and the target refined classification result can be predicted and output.

[0134] The present invention uses the overall accuracy as the evaluation index, which reflects the proportion of images with correct classification in all images of the test set.

[0135] On the above-mentioned constructed MAR20 dataset, the present invention conducts a performance comparison experiment on the proposed dynamic hybrid knowledge distillation method (DHKD) and various existing advanced knowledge distillation methods in the task of refined classification of small target features in remote sensing images. The specific comparison methods include SemCKD and ICKD based on intermediate feature knowledge distillation, RKD and CSKD based on relational feature knowledge distillation, DKD and LSKD based on output feature knowledge distillation, and SRRL that combines intermediate and output feature knowledge distillation.

[0136] Existing advanced knowledge distillation methods are mainly used for model compression tasks. We constructed a refined classification network model for enhancing small target features in remote sensing images based on dynamic hybrid knowledge distillation with reference to the present invention. A refined classification network model for enhancing small target features in remote sensing images based on knowledge distillation was constructed for the above-mentioned comparative knowledge distillation methods. For fair comparison, the configuration of the refined classification model corresponding to the comparative distillation method is the same as that of the model of the present invention. For example, we used ResNet-50 as the model structure of the teacher network and the student network in all refined classification models. During training, paired high- and low-resolution images of the same target are input into the teacher and student networks respectively; during inference, we only retain the student network for testing on low-resolution images.

[0137] The experimental performance comparison results are shown in Table 1, indicating that the proposed DHKD method of the present invention has a significant advantage in the refined classification accuracy of small targets in remote sensing images with absolute sizes of 32, 20, and 12 pixels compared with existing advanced knowledge distillation methods. For aircraft targets with absolute sizes of 32, 20, and 12 pixels respectively, the overall accuracy of the method of the present invention has increased by 2.29%, 2.95%, and 2.81% compared with other optimal methods.

[0138] Table 1

[0139]

[0140]

[0141] Example 2

[0142] Example 2 of the present invention provides a refined classification system for enhancing small target features in remote sensing images based on dynamic hybrid knowledge distillation, which is implemented based on the method of Example 1. The system includes:

[0143] A refined classification module for inputting a low-resolution remote sensing image of a small target to be classified into a pre-trained classification model to obtain the specific model information of the target, that is, to achieve refined classification;

[0144] The classification model includes a backbone low-resolution image student network branch. During the training stage of the classification model, an auxiliary high-resolution image teacher network branch with the same structure as the student network branch and a dynamic hybrid knowledge distillation module are introduced. The dynamic hybrid knowledge distillation module is deployed between the backbone low-resolution image student network branch and the auxiliary high-resolution image teacher network branch. Through the dynamic hybrid knowledge distillation module, the intermediate features and output features of the high-resolution image learned by the teacher network branch are distilled into the student network branch as prior knowledge to guide the training of the student network branch, thereby enhancing the feature representation ability of the student network branch and obtaining a trained classification model.

[0145] It should be noted that in the embodiments of the above system, the various modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional modules are only for the convenience of mutual distinction and are not used to limit the protection scope of the present invention.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the embodiments, those of ordinary skill in the art should understand that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A refined classification method for small target feature enhancement of remote sensing images based on dynamic hybrid knowledge distillation, comprising: Inputting the low-resolution remote sensing image of the small target to be classified into a pre-trained classification model to obtain the specific model information of the target, that is, to achieve refined classification; The classification model includes a backbone low-resolution image student network branch. In the training stage of the classification model, an auxiliary high-resolution image teacher network branch with the same structure as the student network branch and a dynamic hybrid knowledge distillation module are introduced. Among them, the dynamic hybrid knowledge distillation module is deployed between the backbone low-resolution image student network branch and the auxiliary high-resolution image teacher network branch. Through the dynamic hybrid knowledge distillation module, the intermediate features and output features of the high-resolution image learned by the teacher network branch are distilled into the student network branch as prior knowledge to guide the training of the student network branch, thereby enhancing the feature representation ability of the student network branch and obtaining a trained classification model.

2. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 1, wherein, The backbone low-resolution image student network branch uses the ResNet-50 residual network as the basic network model structure, including a convolutional feature extraction module and a classifier. Among them, The convolutional feature extraction module is used to perform convolutional feature extraction on the input remote sensing image I LR to generate a feature map C5; The classifier is used to perform refined classification of the target according to the feature map C5 and output the predicted result Cls of the target model category.

3. The refined classification method for small target feature enhancement of remote sensing images based on dynamic hybrid knowledge distillation according to claim 2, wherein, The convolutional feature extraction module includes: 1 basic convolutional layer Conv1 and 4 convolutional layers based on residual learning, namely Conv2_x, Conv3_x, Conv4_x, and Conv5_x; among them, Conv2_x includes 1 3×3 max-pooling layer and 3 residual blocks, Conv3_x includes 4 residual blocks, Conv4_x includes 6 residual blocks, and Conv5_x includes 3 residual blocks. Inside each residual block, the output of the input feature processed by the stacked convolutional layers is added to the input feature via a skip connection as the output feature of the residual block.

4. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 2, characterized in that The classifier includes 1 average pooling layer and 1 fully connected layer, and the number of neurons in the fully connected layer is the same as the total number of refined categories.

5. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 1, wherein The dynamic hybrid knowledge distillation module includes: an intermediate feature knowledge distillation unit, an output feature knowledge distillation unit, and a dynamic hybrid knowledge distillation unit; among them, The intermediate feature knowledge distillation unit is used to generate channel and spatial weight masks for the intermediate feature map f of the teacher network branch through an attention mechanism, so as to weight the intermediate feature differences between the student network branch and the teacher network branch, and construct an intermediate feature knowledge distillation loss function; HR ​ The output feature knowledge distillation unit is used to construct an output feature knowledge distillation loss function to guide the training of the student network branch; The dynamic hybrid knowledge distillation unit is used to construct a dynamic hybrid knowledge distillation loss function to perform dynamic weighted calculation on the intermediate feature and output feature knowledge distillation losses.

6. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 5, wherein The intermediate feature knowledge distillation loss function L feat is as follows: Among them, C, H, and W respectively represent the number of channels, the height, and the width of the image; represents the teacher network feature map of the k-th channel at the i-th row and j-th column, represents the student network feature map of the k-th channel at the i-th row and j-th column; represents the channel weight mask, represents the spatial weight mask.

7. The refined classification method for small target feature enhancement of remote sensing images based on dynamic hybrid knowledge distillation according to claim 5, wherein The output feature knowledge distillation loss function L logit is as follows: L logit = D KL [σ(z HR / τ), σ(z LR / τ)] Among them, D KL (·,·) represents the KL divergence, z HR and z LR are the logarithmic outputs of the teacher network branch and the student network branch respectively, τ is the distillation temperature, and σ(·) represents the softmax function.

8. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 5, wherein The dynamic hybrid knowledge distillation loss function is as follows: where p represents the current training iteration phase, and are the intermediate feature knowledge distillation loss and the output feature knowledge distillation loss weights, respectively; Among them, represents gradient calculation, is the cross-entropy loss of the refined classification task itself.

9. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 1, wherein, The method further includes the training process of the classification model, including: Obtaining the MAR20 remote sensing image military aircraft target recognition dataset, cropping the aircraft targets in the images according to their horizontal frames to generate single-target image slices, and then obtaining the refined categories including aircraft models according to the given division results of the airport images where they are located and attributing them to the training set; Taking the images in the training set as high-resolution images, and then obtaining the corresponding low-resolution images through bicubic interpolation, thereby forming a high-low resolution image pair and corresponding target model type labels; Setting the batch data volume B, the total number of training iterations T, the learning rate strategy, and the optimizer strategy; The low-resolution images in the high- and low-resolution image pairs in the training set are input into a classification model that introduces an auxiliary high-resolution image teacher network branch and a dynamic hybrid knowledge distillation module, and training is carried out until the training requirements are met, obtaining a trained backbone low-resolution image student network branch, that is, the classification model.

10. The refined classification method for enhancing small target features of remote sensing images based on dynamic hybrid knowledge distillation according to claim 8, wherein, Total loss function during training is as follows: where p represents the current training iteration phase, is the cross-entropy loss of the fine-grained classification task itself, is the dynamic mixed knowledge distillation loss.

Images