A knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection

Abstract

The present invention discloses a knowledge distillation algorithm for hybrid knowledge decoupling of remote sensing target detection, wherein the method comprises the following steps: S1, constructing a remote sensing target detection model as a teacher model for knowledge distillation; S2, lightweighting the model to form a student model for knowledge distillation; S3, predicting bounding box information and calculating target detection loss; S4, the teacher model guides the student model to decouple different types of knowledge; S5, distilling semantic knowledge from the output level, and calculating the loss function value of the output layer distillation; S6, cross-distilling semantic features and positioning features at different levels, and calculating the loss function value of cross-feature distillation; S7, calculating the total loss function value and optimizing the student model. The present invention solves the problem that remote sensing detection models with large and complex parameters are difficult to deploy to edge devices such as satellites. This method not only achieves the lightweighting of remote sensing detectors, but also improves the performance of detectors.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image detection, and in particular to a knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection. Background Art

[0002] Remote sensing image target detection is one of the basic tasks in satellite image processing, which aims to extract the category and location information of targets from remote sensing images.

[0003] In recent years, with the rapid development of deep learning technology, research on remote sensing target detection has achieved significant breakthroughs, significantly improving detection accuracy. However, high-precision detection algorithms rely on complex network models, and the enormous computational complexity and storage requirements hinder their deployment on edge devices such as satellites. Research on lightweight remote sensing target detection algorithms is of great practical significance. Lightweighting deep network models is an effective means to improve the feasibility of network model deployment.

[0004] While model lightweighting techniques can reduce model parameters and computational complexity, they can also affect model accuracy. In recent years, research on lightweighting techniques that balance model detection performance and inference speed has gradually emerged. Knowledge distillation, an emerging model lightweighting method, uses a teacher-student training structure to "distill" the feature representations learned by a complex, highly learning network and transfer them to a network with fewer parameters and weaker learning capabilities, resulting in a fast and powerful lightweight network.

[0005] Currently, knowledge distillation has been widely studied in fields such as image classification and target detection. However, knowledge distillation for remote sensing target detection has not been fully studied. Existing knowledge distillation methods can be divided into knowledge distillation of intermediate features (feature distillation) and knowledge distillation of predicted outputs (Logit knowledge distillation) according to the different stages of distillation. These two knowledge distillation modes transfer knowledge fairly to each distillation region, but in reality not all regions play an equal role in knowledge transfer. In addition, deep feature distillation often requires students to imitate the teacher features at the same level, which leads to the separation of feature transfer processes at different levels and ignores the guiding value of the teacher's shallow features to the student's deep features. Summary of the Invention

[0006] To solve the above problems, the present invention provides a knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection. The specific scheme is as follows:

[0007] A knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection includes the following steps:

[0008] S1. Build a remote sensing target detection model as a teacher model for knowledge distillation;

[0009] S2. Perform lightweight processing on the model in step S1 to form a student model for knowledge distillation;

[0010] S3, the student model in step S2 predicts the bounding box information and calculates the target detection loss;

[0011] S4. The teacher model guides the student model to decouple different types of knowledge. That is, under the guidance of real annotated data, the student model generates semantic-aware masks and location-aware masks for knowledge transfer based on the output of the teacher model.

[0012] S5, under the guidance of the positioning perception mask in step S4, distill the semantic knowledge from the output level and calculate the loss function value of the output layer distillation;

[0013] S6. Under the guidance of the semantic perception mask and positioning perception mask in step S4, cross-distill the semantic features and positioning features at different levels and calculate the loss function value of the cross-feature distillation; that is, adaptively fuse the feature distillation process between teachers and students at different levels to give full play to the guiding role of shallow teacher features on deep student features;

[0014] S7. Calculate the total loss function value and optimize the student model.

[0015] Preferably, in step S2, pruning technology is used to lightweight the remote sensing target detection model; the weight of the BN layer in the student model is used to judge the importance of the channel, and channel pruning is performed on all modules in the backbone network of the student model according to the channel importance.

[0016] Preferably, the lightweight processing steps specifically include:

[0017] S21. Construct the backbone network corresponding to the student model and load the model parameters;

[0018] S22, traverse all BN layers in the student backbone network, record their corresponding weights and number of channels; according to the manually set pruning rate θ, the number of channels after pruning is multiplied by θ to the original number, and record it as the number of channels after pruning;

[0019] S23. Traverse all BN layers and sort their weights in descending order; based on the number of channels after pruning, filter out the important weights that you want to retain and generate the pruning mask corresponding to each BN layer;

[0020] S24, traverse all modules in the student backbone network, and filter the weights corresponding to the convolutional layer, linear layer, and BN layer in a certain dimension according to the instructions of the pruning mask, and discard unwanted weights;

[0021] S25. Construct a lightweight backbone network based on the number of pruned channels; save the pruned weights to the network, and generate a student model file for initializing the student model.

[0022] Preferably, step S3 specifically includes the following steps:

[0023] S31, preprocessing the input image, including unifying the image size and normalizing it, and finally converting it into a tensor form;

[0024] S32, initialize the student model and load the pruned lightweight network as the initial backbone network;

[0025] S33. The image tensor is input into the student model. The backbone network and the neck network in the student model extract image features from shallow to deep to obtain image features of different granularities. The specific operation process is as shown in formula (1):

[0026]

[0027] Among them, (S1, S2, ..., S n ) represents the n stages of the student feature extraction network, n represents the number of features of different granularity, ° represents function nesting, and X represents the image tensor;

[0028] S34, multi-granularity student features F stu The bounding box information is predicted by the detection head in the student model to obtain the category prediction score and regression prediction value Specific calculation formula (2) and formula (3):

[0029]

[0030]

[0031] Among them, S cls and S reg They represent the category prediction layer and regression prediction layer in the student model respectively;

[0032] S35. Calculate the classification loss function value and the regression loss function value based on the true box label Y and the regression target Δ generated by the student detection head. The specific operations are as follows:

[0033]

[0034]

[0035] in, represents the classification loss function, represents the regression loss function.

[0036] Preferably, step S4 of generating the semantic-aware mask and the positioning-aware mask specifically includes the following steps:

[0037] S41. Load the pre-trained teacher model and set the gradients of all parameters of the teacher model not to be back-propagated.

[0038] S42. The backbone network and neck network of the teacher model extract image features from shallow to deep to obtain teacher image features of different granularities. The specific operation is as shown in formula (6):

[0039]

[0040] Among them, (T1, T2, ..., T n ) represents the n stages of the teacher feature extraction network;

[0041] S43. Multi-granularity teacher image feature F tea The bounding box information is predicted by the detection head in the teacher model to obtain the category prediction score and regression prediction value The specific operations are as follows:

[0042]

[0043]

[0044] Among them, T cls and T reg Represent the category prediction layer and regression prediction layer in the teacher model respectively;

[0045] S44. Use the bounding box information predicted by the teacher model to mine the boundary between semantic knowledge and positioning knowledge and generate semantically aware masks and location-aware masks To capture the sensitivity of the distilled area to semantic knowledge and positioning knowledge.

[0046] Preferably, the step S44 specifically includes the following steps:

[0047] S441. Teacher image features For each element in , calculate the maximum value of all category prediction scores and use it as a semantic perception mask The specific operation is as shown in formula (9):

[0048]

[0049] Among them, K is the total number of categories, (c1,…,ci ,…,c K ) represents all target categories;

[0050] S442, Teacher Characteristics Chart For each element in , the predicted value is Encode the anchor box A into the corresponding predicted regression box Then calculate the IoU between the predicted regression box and the real box GT and use it as the positioning perception mask The specific operations are as follows:

[0051]

[0052]

[0053] Among them, M represents the number of all prediction boxes, and decode is the prediction box encoding function.

[0054] Preferably, in the semantically aware mask Under the guidance of , the calculation of the output layer distillation loss function value in step S5 is as follows:

[0055]

[0056] Among them, H k 、W k represents the size of the k-layer feature map, Represents the probability vector at the position (i, j) of the k-layer teacher feature map, Represents the probability vector at the position (i, j) of the k-layer student feature map, represents the Logit distillation loss function, which is used to measure the closeness between the student prediction and the soft label. T is the smoothing factor. is the element value of the k-layer semantic perception mask (i, j), is the sum of all element values ​​of the k-layer semantic-aware mask.

[0057] Preferably, the cross-feature distillation process in step S6 includes the following steps:

[0058] S61, according to the size of the receptive field, the student features F of different granularities are stu Sort in descending order to get And iteratively merge and update in sequence;

[0059] Specifically, first, initialize the fusion features, Feature transformation Secondly, feature fusion is performed; at the tth iteration, due to the predecessor fusion feature The size and current features Therefore, the interpolation method is used to align the two features, and then the weighted sum is used to obtain the fusion feature of t iterations. After n-1 iterations, the feature order is reversed, and finally a multi-layer fusion feature is obtained. The specific operations are shown in formula (13) and formula (14):

[0060]

[0061]

[0062] Among them, φ is the feature transformation layer, and is the fusion weight of t iterations, is the interpolation function;

[0063] S62, introduce semantic-aware mask se and location-aware mask lo , feature distillation is performed on the semantic knowledge and positioning knowledge on the feature map respectively. In addition, the student features used in the distillation process are fused features H stu , the loss function is specifically shown in formula (15), formula (16) and formula (17):

[0064]

[0065]

[0066]

[0067] in, is the value of the k-layer semantic perception mask (i, j) element, The value of the element (i,j) of the location perception mask for layer k, is the sum of the element values ​​of the k-layer semantic perception mask, is the sum of the element values ​​of the k-layer positioning perception mask, is the traditional feature distillation loss function, W se and W lo They represent the coefficients corresponding to the semantic feature distillation and positioning feature distillation losses respectively.

[0068] Preferably, in step S7, the total loss function value includes the classification loss, regression loss, cross-feature distillation loss and output layer distillation loss of the remote sensing target detection task.

[0069] Preferably, the student model optimization in step S7 includes the following steps:

[0070] S71. Use the back-propagation mechanism to calculate the gradient of the total loss function with respect to the student model parameters.

[0071] S72, updating the student model parameters along the gradient direction;

[0072] The total loss function value is calculated as shown in formula (18):

[0073] L total =αL cls +βL reg +γL logit +λL feat #(18)

[0074] Among them, α, β, γ, and λ represent the coefficients corresponding to the losses of each part respectively.

[0075] The beneficial effects of the present invention are:

[0076] This method addresses the difficulty of deploying complex and parameter-intensive remote sensing detection models to edge devices such as satellites. By capturing the sensitivity of distilled regions to different types of knowledge and establishing connections between feature transfers at different levels, it ensures remote sensing detector accuracy while reducing model parameters, computational complexity, and inference time. This method not only reduces the weight of remote sensing detectors but also improves their performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 This is a flow chart of a knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection provided by the present invention.

[0079] Figure 2 is an input picture provided by an embodiment of the present invention.

[0080] Figure 3 This is a comparison chart of semantic information and positioning information perception.

[0081] Figure 4 This is the detection result before distillation provided by the embodiment of the present invention.

[0082] Figure 5 This is the test result after distillation provided by the embodiment of the present invention.

[0083] Note: Figure 4 and Figure 5 The rectangular frame in the figure is artificially added to facilitate the comparison of the test results before and after distillation. DETAILED DESCRIPTION

[0084] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0085] like Figure 1 As shown in FIG, a knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection includes the following steps:

[0086] S1. Build a remote sensing target detection model as a teacher model for knowledge distillation.

[0087] Among them, the teacher model can choose a variety of mainstream remote sensing target detection models, such as: single-stage method Rotated RetinaNet, Rotated ATSS, etc.

[0088] Specifically, the present invention uses RotatedRetinaNet as the basic model, the backbone network adopts ResNet50, and the teacher model and student model detection framework are consistent.

[0089] S2: Compared to the student model, the teacher model has stronger representation capabilities and a larger model size. To obtain a compact student model, the remote sensing target detection model in step S1 is lightweighted using pruning technology to form a student model for knowledge distillation. The weights of the batch normalization layer in the student model are used to determine the importance of channels. Based on the channel importance, channel pruning is performed on all modules in the student model backbone network.

[0090] The lightweight processing steps specifically include:

[0091] S21, construct the backbone network corresponding to the student model and load the model parameters. Specifically, construct the ResNet50 network R O , load the model parameters.

[0092] S22. Traverse all BN layers in the student backbone network and record their corresponding weights and number of channels; according to the manually set pruning rate θ, the number of channels after pruning is increased to θ times the original number, and record the number of channels after pruning.

[0093] Specifically, traverse R O All BN layers in the , record the weights corresponding to each layer and number of channels According to the artificially set pruning rate θ, the number of channels after pruning is multiplied by θ to the original number, and the number of channels after pruning is recorded. Where N is R O The number of BN layers contained, θ is 0.8.

[0094] S23. Traverse all BN layers and sort their weights in descending order; based on the number of channels after pruning, filter out the important weights that you want to retain and generate the pruning mask corresponding to each BN layer.

[0095] Traverse all BN layers, filter important channels, and generate pruning masks. For the i-th BN layer, sort its weights W in descending order. i BN Then filter out the top , generate a mask that indicates whether a channel is important As shown in formula (1'). After N traversals, we finally get a list of pruning masks.

[0096]

[0097] S24, traverse all modules in the student backbone network, specifically, traverse R O According to the instructions of the pruning mask, the weights corresponding to the convolutional layer, linear layer, and BN layer are filtered in a certain dimension and unwanted weights are discarded.

[0098] Specifically, for the i-th convolutional layer, its weight is The weight pruning process is as follows: the weight tensor is pruned in 0 dimension according to the pruning mask Index, in 1 dimension according to the mask Index, and finally get the size Tensor of . Among them, K i CONV Represents the convolution kernel size of the i convolution layer.

[0099] The linear layer in ResNet50 is the category prediction layer, and its weight is The weight pruning process is as follows: the weight tensor is pruned in 1 dimension according to the pruning mask Index to filter out unimportant linear layer weights. Where N_C represents the total number of categories in the dataset.

[0100] S25. Construct a lightweight backbone network based on the number of pruned channels; load the pruned weights into the network, and generate a student model file for initializing the student model.

[0101] Specifically, according to the number of channels CP Create a new ResNet50 network R P . O All weights are loaded into R P , and R P All weights are saved as model files for direct loading and use.

[0102] S3: The student model from step S2 predicts bounding box information and calculates the object detection loss. Specifically, the input image is first preprocessed and converted into a 3*1024*1024 tensor. Next, the image tensor is passed through the student model to obtain the class score and bounding box information. The classification loss function and regression loss function are then calculated based on the remote sensing object detection loss function.

[0103] The specific steps include:

[0104] S31, preprocessing the input image, including unifying the image size and normalizing it, and finally converting it into a tensor form;

[0105] Specifically, the main steps in preprocessing the input image include resizing it to 1024*1024*3, then normalizing it to a mean of (123.675, 116.28, 103.53) and a variance of (58.395, 57.12, 57.375), and converting it into a 3*1024*1024 tensor. To speed up processing, batch processing is introduced to expand the image tensor dimensions to Batch*3*1024*1024, where Batch represents the number of images processed in a batch.

[0106] S32, initialize the student model, load the pruned lightweight network as the initial backbone network; specifically, load the pruned R P The network serves as the initial backbone network.

[0107] S33. The image tensor is input into the student model. The backbone network and the neck network in the student model extract image features from shallow to deep, and obtain 5 image features of different granularities. The specific operation process is as shown in formula (1):

[0108]

[0109] Among them, (S1, S2, ..., S5) represents the five stages of the student feature extraction network, Indicates function nesting, X represents image tensor; Represents a tensor of size Batch*256*128*128, Represents a tensor of size Batch*256*64*64, Represents a tensor of size Batch*256*32*32, Represents a tensor of size Batch*256*16*16, Represents a tensor of size Batch*256*8*8.

[0110] S34, multi-granularity student features F stu The bounding box information is predicted by the detection head in the student model to obtain the category prediction score and regression prediction value Specific calculation formula (2) and formula (3):

[0111]

[0112]

[0113] Among them, S cls and S reg They represent the category prediction layer and regression prediction layer in the student model respectively; Represents a tensor of size Batch*135*128*128, Represents a tensor of size Batch*135*64*64, Represents a tensor of size Batch*135*32*32, Represents a tensor of size Batch*135*16*16, Represents a tensor of size Batch*135*8*8, Represents a tensor of size Batch*45*128*128, Represents a tensor of size Batch*45*64*64, Represents a tensor of size Batch*45*32*32, Represents a tensor of size Batch*45*16*16, Represents a tensor of size Batch*45*8*8.

[0114] S35. Calculate the classification loss function value and the regression loss function value based on the true box label Y and the regression target Δ generated by the student detection head. The specific operations are as follows:

[0115]

[0116]

[0117] in, Represents the classification loss function Focal Loss, Represents the regression loss function L1 Loss.

[0118] S4. The teacher model guides the student model to decouple different types of knowledge; that is, under the guidance of real labeled data, semantic-aware masks and positioning-aware masks for knowledge transfer are generated based on the output of the teacher model.

[0119] like Figure 3 As shown in the figure, since each element on the feature graph has different perception capabilities for different types of knowledge, it is necessary to distinguish the degree of contribution of each element in different knowledge transfer processes. A teacher model with strong representation and knowledge perception capabilities can guide the student model to decouple different types of knowledge. Therefore, this application designs a knowledge decoupling module that uses the category prediction scores and regression prediction values ​​of the teacher model to capture the semantic knowledge and positioning knowledge on the feature graph, thereby guiding the student model to transfer different types of knowledge in a targeted manner.

[0120] The step S4 of generating the semantic-aware mask and the positioning-aware mask specifically includes the following steps:

[0121] S41. Load the pre-trained teacher model and set the gradients of all parameters of the teacher model not to be back-propagated.

[0122] S42, the backbone network ResNet50 and the neck network FPN of the teacher model extract image features from shallow to deep to obtain teacher image features of different granularities The specific operation is as shown in formula (6):

[0123]

[0124] Among them, (T1, T2, ..., T5) represents the five stages of the teacher feature extraction layer, Represents a tensor of size Batch*256*128*128, Represents a tensor of size Batch*256*64*64, Represents a tensor of size Batch*256*32*32, Represents a tensor of size Batch*256*16*16, Represents a tensor of size Batch*256*8*8.

[0125] S43. Multi-granularity teacher image feature F tea The bounding box information is predicted by the detection head in the teacher model to obtain the category prediction score and regression prediction value The specific operations are as follows:

[0126]

[0127]

[0128] Among them, T cls and T reg Represent the category prediction layer and regression prediction layer in the teacher model respectively; Represents a tensor of size Batch*135*128*128, Represents a tensor of size Batch*135*64*64, Represents a tensor of size Batch*135*32*32, Represents a tensor of size Batch*135*16*16, Indicates the size is Batch*135*8*8, Represents a tensor of size Batch*45*128*128, Represents a tensor of size Batch*45*64*64, Represents a tensor of size Batch*45*32*32, Represents a tensor of size Batch*45*16*16, Represents a tensor of size Batch*45*8*8.

[0129] S44. Use the bounding box information predicted by the teacher model to mine the boundary between semantic knowledge and positioning knowledge and generate semantically aware masks and location-aware masks To capture the sensitivity of the distilled area to semantic knowledge and positioning knowledge.

[0130] The step S44 specifically includes the following steps:

[0131] S441. Teacher image features For each element in , calculate the maximum value of all category prediction scores and use it as a semantic perception mask The specific operation is as shown in formula (9):

[0132]

[0133] Among them, K is the total number of categories, (c1,…,c i ,…,c K ) represents all target categories;

[0134] S442, Teacher Characteristics Chart For each element in , the predicted value will be regressed Encoded into the corresponding prediction regression box Then calculate the IoU between the predicted regression box and the real box GT and use it as the positioning perception mask Specifically, the size is Batch*45*H i *W i Features The dimension is transformed into a size of Batch*(H i *W i *9)*5 tensor, and then predict the value based on the regression The anchor box A generated by the student detection head is decoded into a size of Batch*(H i *W i *9)*5 prediction box Since each image contains a different number of real boxes, we perform Batch iterations to calculate the IoU between the predicted box and the real box and find the maximum value. Each iteration will get a value of size H. i *W i Finally, these tensors are stacked into a Batch*H i *W i The specific operations are as follows:

[0135]

[0136]

[0137] Among them, M represents the number of all prediction boxes, and decode is the prediction box encoding function.

[0138] S5. Under the guidance of the positioning perception mask in step S4, the semantic knowledge is distilled from the output level and the loss function value of the output layer distillation is calculated.

[0139] The basic idea of ​​output layer knowledge distillation (Logit knowledge distillation) is to use the output of the teacher model as a kind of supervision information, and continuously optimize the distance between the output of the student model and the soft label provided by the teacher model during the distillation process. The traditional Logit knowledge distillation process is shown in formula (2'):

[0140]

[0141] Among them, H k 、W k represents the size of the k-layer feature map, Represents the probability vector at the position (i, j) of the k-layer teacher feature map, Represents the probability vector at the position (i, j) of the k-layer student feature map, represents the Logit distillation loss function, which is used to measure the closeness between the student prediction and the soft label, and T is the smoothing factor.

[0142] Specifically, (k∈{1,2,3,4,5}) represents the probability vector at position (i,j) of the k-layer teacher category probability map, (k∈{1,2,3,4,5}) represents the probability vector at position (i,j) of the k-layer student category probability map.

[0143] However, this knowledge distillation model often transfers semantic knowledge equally across each distilled region. In reality, not all regions contribute equally to semantic transfer. Therefore, to enhance semantically sensitive regions and suppress less sensitive regions, this patent introduces semantic masking, assigning different weights to each distilled region.

[0144] Based on the traditional Logit distillation paradigm, the semantic mask is introduced se , giving different weights to each element on the category probability map to give full play to its due distillation value.

[0145] Specifically, the calculation of the output layer distillation loss function value is as shown in formula (12):

[0146]

[0147] in, is the element value of the k-layer semantic perception mask (i, j), is the sum of all element values ​​of the k-layer semantic perception mask, is the KL divergence function, and the value of T is 10.

[0148] S6. Under the guidance of the semantic perception mask and positioning perception mask in step S4, cross-distill the semantic features and positioning features at different levels and calculate the loss function value of the cross-feature distillation; that is, adaptively fuse the feature distillation process between teachers and students at different levels to give full play to the guiding role of shallow teacher features on deep student features.

[0149] The basic idea of ​​feature distillation is that student features imitate teacher features. The gap between student features and teacher features is continuously optimized during the distillation process, as shown in formula (3'):

[0150]

[0151] in, (k∈{1,2,3,4,5}) represents the feature vector at the position (i,j) of the k-layer teacher feature map, (k∈{1,2,3,4,5}) represents the feature vector at position (i,j) of the k-layer student feature map. Represents the feature distillation loss, which is used to measure the similarity between teacher features and student features.

[0152] Traditional feature distillation requires the student to mimic the teacher's features at the same granularity, which isolates the transfer between features of different granularities. In reality, the shallow features of the teacher model also provide guidance for the deep features of the student model. Furthermore, feature distillation is essentially the transfer of knowledge between corresponding elements in the student and teacher feature maps, and each element plays an equal role in the knowledge transfer process, which obscures the contribution of different distillation regions to feature distillation. Moreover, because remote sensing target detection is a combination of classification and regression tasks, each distillation region contributes differently to semantic knowledge transfer and positioning knowledge transfer.

[0153] Therefore, this patent designs a multi-layer feature interaction module to iteratively fuse student features of different granularities to obtain fused features that enrich information for the feature distillation stage to achieve cross-feature distillation. Semantic masks and localization masks generated from the teacher's prediction results are introduced to distinguish the sensitivity of each distillation region to semantic knowledge and localization knowledge, thereby achieving targeted semantic feature distillation and localization feature distillation.

[0154] Specifically, the cross-feature distillation process includes the following steps:

[0155] S61, according to the size of the receptive field, the student features F of different granularities are stu Sort in descending order to get And iteratively merge and update in sequence;

[0156] First, initialize the fusion features. Feature transformation Secondly, feature fusion is performed; at the tth iteration, due to the predecessor fusion feature The size and current features Therefore, the interpolation method is used to align the two features, and then the weighted sum is used to obtain the fusion feature of t iterations. After 5-1 iterations, the feature order is reversed, and finally multi-layer fusion features are obtained The specific operations are shown in formula (13) and formula (14):

[0157]

[0158]

[0159] Among them, φ is the feature transformation layer, and is the fusion weight of t iterations, is the bilinear interpolation function, The dimension is Batch*256*H t-1 *Wt-1 , The dimension is Batch*256*H t *W t , The dimensions are Bach*256*H t *W t .

[0160] S62. Based on the traditional distillation paradigm, semantically aware mask is introduced se and location-aware mask lo , feature distillation is performed on the semantic knowledge and positioning knowledge on the feature map respectively. In addition, the student features used in the distillation process are fused features H stu , the loss function is specifically shown in formula (15), formula (16) and formula (17):

[0161]

[0162]

[0163]

[0164] in, is the value of the k-layer semantic perception mask (i, j) element, The value of the element (i,j) of the location perception mask for layer k, is the sum of the element values ​​of the k-layer semantic perception mask, is the sum of the element values ​​of the k-layer positioning perception mask, is L1Loss, W se and W lo They represent the coefficients corresponding to the semantic feature distillation and positioning feature distillation losses, respectively, and both are 1.

[0165] S7. Calculate the total loss function value and optimize the student model. The total loss function value is calculated as shown in formula (18):

[0166] L total =αL cls +βL reg +γL logit +λL feat #(18)

[0167] Here, α, β, γ, and λ represent the coefficients corresponding to the losses of each part, and their corresponding values ​​are 1, 1, 1, 0.01, and 0.01 respectively.

[0168] Specifically, the student model optimization includes the following steps:

[0169] S71. Use the back-propagation mechanism to calculate the gradient of the total loss function with respect to the student model parameters.

[0170] S72. Update the student model parameters along the gradient direction.

[0171] The knowledge distillation method proposed in this paper addresses the difficulty of deploying complex, parameter-heavy remote sensing detection models to edge devices such as satellites. This method not only reduces the weight of remote sensing detectors but also improves their performance.

[0172] The above is a detailed introduction to the knowledge distillation method for hybrid knowledge decoupling for remote sensing target detection provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

[0173] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0174] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection, characterized by: The following steps are involved: S1. Build a remote sensing target detection model as a teacher model for knowledge distillation; S2. Perform lightweight processing on the model in step S1 to form a student model for knowledge distillation; S3, the student model in step S2 predicts the bounding box information and calculates the target detection loss; S4. The teacher model guides the student model to decouple different types of knowledge. That is, under the guidance of real annotated data, the student model generates semantic-aware masks and location-aware masks for knowledge transfer based on the output of the teacher model. S5, under the guidance of the positioning perception mask in step S4, distill the semantic knowledge from the output level and calculate the loss function value of the output layer distillation; S6. Under the guidance of the semantic perception mask and positioning perception mask in step S4, cross-distill the semantic features and positioning features at different levels and calculate the loss function value of the cross-feature distillation. That is, the feature distillation process between teachers and students at different levels is adaptively integrated to give full play to the guiding role of shallow teacher features on deep student features. Specifically, the cross-feature distillation process includes the following steps: S61, according to the size of the receptive field, the student features F of different granularities are stu Sort in descending order to get And iteratively merge and update in sequence; Specifically, first, initialize the fusion features, Feature transformation Secondly, feature fusion is performed; at the tth iteration, due to the predecessor fusion feature The size and current features Therefore, the interpolation method is used to align the two features, and then the weighted sum is used to obtain the fusion feature of t iterations. After n-1 iterations, the feature order is reversed, and finally a multi-layer fusion feature is obtained. The specific operations are shown in formula (13) and formula (14): Among them, φ is the feature transformation layer, and is the fusion weight of t iterations, is the interpolation function; S62, introduce semantic-aware mask se and location-aware mask lo , feature distillation is performed on the semantic knowledge and positioning knowledge on the feature map respectively. In addition, the student features used in the distillation process are fused features H stu , the loss function is shown in formula (15), formula (16) and formula (17): in, is the value of the k-layer semantic perception mask (i, j) element, The value of the element (i,j) of the location perception mask for layer k, is the sum of the element values ​​of the k-layer semantic perception mask, is the sum of the element values ​​of the k-layer positioning perception mask, is the traditional feature distillation loss function, W se and W lo Represent the coefficients corresponding to the semantic feature distillation and positioning feature distillation losses respectively; S7. Calculate the total loss function value and optimize the student model.

2. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 1 is characterized by: In step S2, pruning technology is used to lightweight the remote sensing target detection model; the weight of the BN layer in the student model is used to judge the importance of the channel, and channel pruning is performed on all modules in the backbone network of the student model according to the channel importance.

3. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 2 is characterized in that: The steps of lightweight processing include: S21. Construct the backbone network corresponding to the student model and load the model parameters; S22, traverse all BN layers in the student backbone network, record their corresponding weights and number of channels; according to the manually set pruning rate θ, the number of channels after pruning is multiplied by θ to the original number, and record the number of channels after pruning; S23. Traverse all BN layers and sort their weights in descending order; based on the number of channels after pruning, filter out the important weights that you want to retain and generate the pruning mask corresponding to each BN layer; S24, traverse all modules in the student backbone network, and filter the weights corresponding to the convolutional layer, linear layer, and BN layer in a certain dimension according to the instructions of the pruning mask, and discard unwanted weights; S25. Construct a lightweight backbone network based on the number of pruned channels; save the pruned weights to the network, and generate a student model file for initializing the student model.

4. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 1 is characterized in that: Step S3 specifically includes the following steps: S31, preprocessing the input image, including unifying the image size and normalizing it, and finally converting it into a tensor form; S32, initialize the student model and load the pruned lightweight network as the initial backbone network; S33. The image tensor is input into the student model. The backbone network and the neck network in the student model extract image features from shallow to deep to obtain image features of different granularities. The specific operation process is as shown in formula (1): Among them, (S1, S2, ..., S n ) represents the n stages of the student feature extraction network, n represents the number of features of different granularity, ° represents function nesting, and X represents the image tensor; S34, multi-granularity student features F stu The bounding box information is predicted by the detection head in the student model to obtain the category prediction score and regression prediction value Specific calculation formula (2) and formula (3): Among them, S cls and S reg They represent the category prediction layer and regression prediction layer in the student model respectively; S35. Calculate the classification loss function value and the regression loss function value based on the true box label Y and the regression target Δ generated by the student detection head. The specific operations are as follows: in, represents the classification loss function, represents the regression loss function.

5. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 1 is characterized in that: Step S4 generates a semantic-aware mask and a positioning-aware mask, specifically comprising the following steps: S41. Load the pre-trained teacher model and set the gradients of all parameters of the teacher model not to be back-propagated. S42. The backbone network and neck network of the teacher model extract image features from shallow to deep to obtain teacher image features of different granularities. The specific operation is as shown in formula (6): Among them, (T1, T2, ..., T n ) represents the n stages of the teacher feature extraction network; S43. Multi-granularity teacher image feature F tea The bounding box information is predicted by the detection head in the teacher model to obtain the category prediction score and regression prediction value The specific operations are as follows: Among them, T cls and T reg Represent the category prediction layer and regression prediction layer in the teacher model respectively; S44. Use the bounding box information predicted by the teacher model to mine the boundary between semantic knowledge and positioning knowledge and generate semantically aware masks and location-aware masks To capture the sensitivity of the distilled area to semantic knowledge and positioning knowledge.

6. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 5 is characterized in that: The step S44 specifically includes the following steps: S441. Teacher image features For each element in , calculate the maximum value of all category prediction scores and use it as a semantic perception mask The specific operation is as shown in formula (9): Among them, K is the total number of categories, (c1,…,c i ,…,c K ) represents all target categories; S442, Teacher Characteristics Chart For each element in , the predicted value is Encode the anchor box A into the corresponding predicted regression box Then calculate the IoU between the predicted regression box and the real box GT and use it as the positioning perception mask The specific operations are as follows: Among them, M represents the number of all prediction boxes, and decode is the prediction box encoding function.

7. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 1 is characterized in that: In semantically aware masking Under the guidance of , the calculation of the output layer distillation loss function value in step S5 is as follows: Among them, H k 、W k represents the size of the k-layer feature map, Represents the probability vector at the position (i, j) of the k-layer teacher feature map, Represents the probability vector at the position (i, j) of the k-layer student feature map, represents the Logit distillation loss function, which is used to measure the closeness between the student prediction and the soft label. T is the smoothing factor. is the element value of the k-layer semantic perception mask (i, j), is the sum of all element values ​​of the k-layer semantic-aware mask.

8. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 1 is characterized by: In step S7, the total loss function value includes the classification loss, regression loss, cross-feature distillation loss, and output layer distillation loss of the remote sensing target detection task.

9. The knowledge distillation algorithm for hybrid knowledge decoupling for remote sensing target detection according to claim 1 is characterized in that: The student model optimization in step S7 includes the following steps: S71. Use the back-propagation mechanism to calculate the gradient of the total loss function with respect to the student model parameters. S72, updating the student model parameters along the gradient direction; The total loss function value is calculated as shown in formula (18): L total =αL cls +βL reg +γL logit +λL feat #(18) Among them, α, β, γ, and λ represent the coefficients corresponding to the losses of each part respectively.

Images