Remote sensing target detection method and device based on multi-level knowledge distillation

Abstract

The invention relates to the technical field of remote sensing target detection, in particular to a remote sensing target detection method and device based on multi-level knowledge distillation, and the method comprises the steps: inputting remote sensing images into a teacher model and a student model, and extracting multi-level features; the low-level features are distilled through a shift local feature collaborative distillation module, and the recognition capability of a student model on a small target is enhanced by utilizing local spatial feature modeling; a deep pixel-by-pixel distillation module is used for carrying out pixel-by-pixel distillation on the high-level features, so that the classification capability is improved; the generative adversarial distillation is performed on the angle prediction of the teacher network and the student network, so that the angle prediction precision is improved; and finally, optimizing the student model in combination with various losses. According to the student model obtained through training, the prediction precision of the remote sensing target is remarkably improved, meanwhile, high prediction efficiency is kept, and the actual requirements of remote sensing application scenes are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing target detection, in particular to a remote sensing target detection method and device based on multi-level knowledge distillation. BACKGROUND

[0002] Remote sensing target detection is a technology that automatically identifies and locates specific targets (such as airplanes, vehicles, buildings, ships, etc.) from remote sensing images. It combines image processing, pattern recognition, and deep learning methods to achieve class judgment and spatial location (such as bounding boxes) labeling of targets, and is widely used in urban planning, military reconnaissance, environmental monitoring, disaster assessment, and other fields. It is a key link in remote sensing information extraction and application.

[0003] With the continuous development of deep learning technology, remote sensing target detection models can achieve high detection accuracy due to their strong expression ability. However, as the accuracy improves, the model often has problems such as large number of model parameters and high computational complexity, making it difficult to deploy in practical applications, especially in some computing resource-limited scenarios (such as drones, edge devices, etc.).

[0004] Knowledge distillation technology is an effective model compression and knowledge transfer method. Its core idea is to transfer the knowledge learned by a mature complex "teacher model" to a "student model" with simpler structure and fewer parameters, so that the student model can maintain close to the performance of the teacher model while significantly reducing computational cost and storage requirements, facilitating deployment on resource-limited devices.

[0005] However, traditional knowledge distillation methods are mostly designed for natural images, while remote sensing targets have their unique characteristics, leading to many challenges when applying traditional knowledge distillation methods to remote sensing target detection tasks. Specifically, these challenges include: (1) Multi-scale target distribution: The size of targets in remote sensing images varies greatly, with a high proportion of small targets. Traditional knowledge distillation methods have a single feature alignment method, focusing only on global features or simple feature matching, lacking effective modeling of local spatial structures and detailed features in remote sensing images, making it difficult to achieve local precise alignment of features between teacher and student models, resulting in the student model after migration being unable to accurately capture multi-scale targets, especially small targets; (2) Spatial position sensitivity: In remote sensing target detection, spatial position information is crucial. Traditional distillation methods lack in transferring positioning information, which may ignore spatial structure information of features, making it difficult to effectively improve the positioning accuracy of the student model; (3) Angle information: In the remote sensing target detection task, the angle information of the target is also an important detection dimension, and the traditional knowledge distillation method often lacks an effective distillation mechanism for angle prediction and other direction information, resulting in low prediction accuracy of the student model in the angle information.

[0006] Therefore, how to apply the knowledge distillation method to the remote sensing target detection task to improve the detection efficiency while effectively improving the accuracy of target detection is a problem to be solved in the prior art. SUMMARY

[0007] To this end, the technical problem to be solved by the present application is to overcome the problem that the knowledge distillation method is applied to the remote sensing target detection task in the prior art, and it is difficult to effectively improve the accuracy of target detection.

[0008] To solve the above technical problems, the present application provides a remote sensing target detection method based on multi-level knowledge distillation, comprising: inputting the remote sensing image into the trained teacher model and student model respectively, and outputting five levels of teacher features and five levels of student features; inputting the low-level features in the teacher features and student features into the shift local feature cooperative distillation module, calculating the shift local feature loss, comprising: cutting each low-level feature into a patch-level feature, and calculating the first spatial feature of each low-level feature according to the patch-level feature; cutting each low-level feature into a shift patch-level feature after shifting, calculating the shift local spatial feature of each low-level feature according to the shift patch-level feature, and then obtaining the second spatial feature of each low-level feature by restoring the position after reversing the shift local spatial feature; fusing the first spatial feature and the second spatial feature of each low-level feature to obtain the fused spatial feature of each low-level feature; calculating the local feature distillation loss according to the teacher fused spatial feature and the student fused spatial feature of the corresponding level; inputting the high-level features in the teacher features and student features into the deep pixel-by-pixel distillation module, and calculating the pixel-by-pixel distillation loss according to the distance between all corresponding pixels between the teacher features and student features of the corresponding level; inputting the teacher features and student features into the detection head to obtain the predicted classification, predicted position and predicted angle of the target corresponding to the teacher model and student model respectively; taking the predicted angle of the teacher model as the real angle and taking the predicted angle of the student model as the generated angle to calculate the generative adversarial distillation loss; training the student model based on the total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss and the supervision loss; The remote sensing image to be detected is input into the student model completed training to obtain a predicted classification, a predicted position and a predicted angle of the target.

[0009] Preferably, the teacher model is an FCOS model based on Swin-T, and the student model is an FCOS model based on Swin-T-T.

[0010] Preferably, the low-level features are features of the first and second levels, and the high-level features are features of the fourth and fifth levels.

[0011] Preferably, each low-level feature is respectively cropped into a patch-level feature, and the formula is: ; wherein, and are a teacher feature and a student feature of the i-th level, respectively, and are a teacher patch-level feature and a student patch-level feature of the i-th level, respectively, is a cropping operation, ; A first spatial feature of each low-level feature is calculated according to the patch-level feature, and the formula is: ; ; ; ; wherein, and respectively represent an operation of averaging each patch along the y-axis direction and the x-axis direction and then mapping back to the patch size, and are a teacher local spatial feature and a student local spatial feature of the i-th level in the y-axis direction, respectively, and are a teacher local spatial feature and a student local spatial feature of the i-th level in the x-axis direction, respectively, is a concatenation operation along the channel dimension, and are a teacher first spatial feature and a student first spatial feature of the i-th level, respectively. ;

[0012] Preferably, each low-level feature is respectively cropped into a patch-level feature, and the formula is: ; ; wherein, ​​​and are the teacher feature and the student feature of the i-th level respectively are the teacher feature and the student feature of the i-th level respectively is a shift operation, and are the teacher feature and the student feature of the i-th level respectively are the teacher feature and the student feature of the i-th level respectively is a crop operation, and are the teacher feature and the student feature of the i-th level respectively are the teacher shift patch-level feature and the student shift patch-level feature of the i-th level respectively ; The shift local spatial feature of each low-level feature is calculated according to the shift patch-level feature, and the formula is: ; ; wherein, and respectively represent the operation of averaging each patch along the y-axis direction and the x-axis direction and then mapping back to the patch size, and are the teacher shift local spatial feature and the student shift local spatial feature of the i-th level respectively are the teacher shift local spatial feature and the student shift local spatial feature of the i-th level respectively are the teacher shift local spatial feature and the student shift local spatial feature of the i-th level respectively are the teacher shift local spatial feature and the student shift local spatial feature of the i-th level respectively are the teacher shift local spatial feature and the student shift local spatial feature of the i-th level respectively The second spatial feature of each low-level feature is obtained by recovering the position after the inverse shift, and the formula is: ; ; ; wherein, and are the teacher local spatial feature of the i-th level after recovering the position in the y-axis direction and the x-axis direction respectively and are the student local spatial feature of the i-th level after recovering the position in the y-axis direction and the x-axis direction respectively is an inverse shift operation; is a concatenation operation along the channel dimension, and are the teacher second spatial feature and the student second spatial feature of the i-th level respectively are the teacher second spatial feature and the student second spatial feature of the i-th level respectively

[0013] ​​Preferably, the step length of the shift operation is half of the step length of the cropping operation, the feature is shifted right down by the shift operation, and the part exceeding the feature size range after the shift is filled to the left up; After the shifted feature is cropped to the patch-level feature by the cropping operation, the patch block containing the filled part in the left up is cropped again according to the filled edge, and the target shifted patch-level feature is obtained.

[0014] Preferably, after the fusion feature of each low-level feature is obtained, the method further comprises: The fusion feature is normalized, and the formula is: ; wherein, and are the teacher fusion feature and the student fusion feature of the i-th level, are the normalized teacher fusion feature and the normalized student fusion feature of the i-th level.

[0015] Preferably, the local feature distillation loss is calculated according to the teacher fusion feature and the student fusion feature of the corresponding level, and the formula is: ; wherein, and are the normalized teacher fusion feature and the normalized student fusion feature of the i-th level, is the local feature distillation loss.

[0016] Preferably, the high-level features in the teacher feature and the student feature are input into a deep pixel-by-pixel distillation module, the pixel-by-pixel distillation loss is calculated according to the distance between all corresponding pixels of the teacher feature and the student feature of the corresponding level, and the formula is: ; ; wherein, is the pixel-by-pixel distillation loss, and are the teacher feature and the student feature of the i-th level, is a flattening operation, and are the teacher flattened vector and the student flattened vector of the i-th level, and are the i-th pixel value of and , and , is the total number of pixels,​​​​ .

[0017] The application also provides a remote sensing target detection device based on multi-level knowledge distillation, comprising: A feature extraction module is configured to input a remote sensing image into a trained teacher model and a student model respectively, and output five levels of teacher features and five levels of student features; A local feature distillation loss calculation module is configured to input low-level features in the teacher features and the student features into a shift local feature cooperative distillation module, calculate a shift local feature loss, and comprise: Each low-level feature is respectively cropped into a patch-level feature, and a first spatial feature of each low-level feature is calculated according to the patch-level feature; Each low-level feature is respectively shifted and then cropped into a shifted patch-level feature, a shift local spatial feature of each low-level feature is calculated according to the shifted patch-level feature, and a second spatial feature of each low-level feature is obtained by restoring the position through reverse shifting of the shift local spatial feature; The first spatial feature and the second spatial feature of each low-level feature are fused to obtain a fusion spatial feature of each low-level feature; A local feature distillation loss is calculated according to the teacher fusion spatial feature and the student fusion spatial feature of the corresponding level; A pixel-by-pixel distillation loss calculation module is configured to input high-level features in the teacher features and the student features into a deep pixel-by-pixel distillation module, and calculate a pixel-by-pixel distillation loss according to the distance between all corresponding pixels between the teacher features and the student features of the corresponding level; A detection module is configured to input the teacher features and the student features into a detection head to obtain a predicted classification, a predicted position and a predicted angle of a target corresponding to the teacher model and the student model respectively; A discriminant generation loss calculation module is configured to take the predicted angle of the teacher model as a real angle, take the predicted angle of the student model as a generated angle, and calculate a generative adversarial distillation loss; A training module is configured to train the student model with a total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss and a supervision loss; An inference module is configured to input a remote sensing image to be detected into the trained student model to obtain a predicted classification, a predicted position and a predicted angle of a target.

[0018] The above technical solutions of the application have the following beneficial effects compared with the prior art: The remote sensing target detection method based on multi-level knowledge distillation provided by the application inputs shallow features output by a teacher model and a student model into a shift local feature collaborative distillation module, divides the shallow features into multiple local regions and calculates local spatial features, enhances the modeling capability for local spatial structures and detailed features in a remote sensing image, and more accurately realizes shallow feature alignment and distillation of the teacher network and the student network by using a shift operation, effectively solves the problem of lack of local modeling in traditional knowledge distillation methods, realizes effective migration of the spatial structure of the shallow features, and enhances the recognition capability for small targets in the remote sensing image. Furthermore, the application inputs deep features output by the teacher model and the student model into a deep pixel-by-pixel distillation module to realize effective migration of semantic information of the deep features and improve the classification capability of the model. Moreover, the application constructs a generative adversarial distillation loss based on the prediction angle of the teacher model and the student model, realizes effective migration of angle prediction knowledge, and improves the accuracy of the student model in predicting the angle of the target in the remote sensing image. The student model trained by the application significantly improves the prediction accuracy of the remote sensing target while maintaining high prediction efficiency, and meets the actual needs of remote sensing application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to make the content of the application more easily understood, the application will be further described in detail below according to specific embodiments of the application and in conjunction with the drawings, in which: Figure 1 is a flowchart of the remote sensing target detection method based on multi-level knowledge distillation of the application; Figure 2 is a structure diagram of the whole distillation; Figure 3 is a schematic diagram of the shift local feature collaborative distillation module, wherein Figure 3 (a) in is an original feature map, Figure 3 (b) in is a feature map after shifting, Figure 3 (c) in is a feature map after recovering the position after reverse shifting; Figure 4 is a comparison result diagram of the method of the application and a baseline model, wherein Figure 4 (a) and (c) in are target detection result example diagrams of the student model trained by the method of the application, Figure 4 (b) and (d) in are target detection result example diagrams of the baseline student model. DETAILED DESCRIPTION

[0020] The application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the application and implement it, but the embodiments are not limiting to the application.

[0021] Reference is made to Figure 1 andFigure 2 As shown, the present application provides a remote sensing target detection method based on multi-level knowledge distillation, comprising: S1: inputting the remote sensing image into the trained teacher model and student model respectively, and outputting five levels of teacher features and five levels of student features; S2: inputting the low-level features in the teacher features and student features into the shift local feature collaborative distillation module, and calculating the shift local feature loss; S3: inputting the high-level features in the teacher features and student features into the deep pixel-by-pixel distillation module, and calculating the pixel-by-pixel distillation loss according to the distance between all corresponding pixels between the teacher features and student features of the corresponding level; S4: inputting the teacher features and student features into the detection head, and obtaining the predicted classification, predicted position and predicted angle of the target corresponding to the teacher model and student model respectively; S5: taking the predicted angle of the teacher model as the real angle, and taking the predicted angle of the student model as the generated angle, and calculating the generative adversarial distillation loss; S6: training the student model with the total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss and the supervision loss; S7: inputting the remote sensing image to be detected into the trained student model, and obtaining the predicted classification, predicted position and predicted angle of the target.

[0022] Specifically, in S1, the teacher model is an FCOS (Fully Convolutional One-Stage Object Detection) model based on Swin-T, and the student model is an FCOS model based on Swin-T-T. The network parameters of the teacher model and the student model are shown in Table 1.

[0023] Table 1, network parameter table

[0024] The Swin-T-T structure is a simplified Swin-T structure. The Swin-T structure includes four stages, and the number of transformer blocks set in each stage of the Swin-T-T structure is 1, 1, 2 and 1 respectively, and the number of attention heads set in each stage is 2, 4, 8 and 8 respectively. The Swin-T-T structure has fewer parameters than the Swin-T structure as a whole, with a parameter amount of 13.63M.

[0025] The data set used for training in this embodiment is the DOTA1.0 data set disclosed in the prior art, and the test set is used as the validation set. The remote sensing image is outputted as five levels of teacher features after passing through the teacher model, denoted as ; and is outputted as five levels of student features after passing through the student model, denoted as wherein .

[0026] Preferably, the low-level features are defined as the features of the first and second levels, and the high-level features are defined as the features of the fourth and fifth levels.

[0027] Specifically, in S2, the low-level features in the teacher features and the student features are input into the shift local feature collaborative distillation module to calculate a shift local feature loss, including: S21: each low-level feature is respectively cropped into a patch-level feature, and a first spatial feature of each low-level feature is calculated according to the patch-level feature.

[0028] Each low-level feature is respectively cropped into a patch-level feature, and the formula is: ; wherein, and are the teacher features and the student features of the first level, respectively, and are the teacher patch-level features and the student patch-level features of the first level, respectively, is a cropping operation, .

[0029] Preferably, the step size of the cropping operation is set to 8, and the size of each patch in the obtained teacher patch-level feature and the student patch-level feature is 8x8. is the nth patch in the teacher patch-level feature of the first level, is the nth patch in the student patch-level feature of the first level, and n is the total number of patches.

[0030] The first spatial feature of each low-level feature is calculated according to the patch-level feature, and the formula is: ; ; ; wherein, and respectively represent an operation of averaging each patch along the y-axis direction and the x-axis direction and then mapping back to the patch size, and are the teacher local spatial features and the student local spatial features of the first level y-axis direction, respectively, is the teacher local spatial feature of the first The local spatial features of the nth patch in the y-axis direction of the teacher's local spatial features in the hierarchical y-axis direction. For the first The local spatial features of the nth patch in the y-axis direction of the student's local spatial features in the hierarchical y-axis direction; and The first Local spatial features of teachers and students along the x-axis of the hierarchy. For the first The local spatial features of the nth patch in the x-axis direction of the teacher's local spatial features in the hierarchical x-axis direction. For the first The local spatial features of the nth patch in the x-axis direction of the student local spatial features in the hierarchical x-axis direction; This is a splicing operation along the channel dimension. and The first The hierarchical characteristics of the teacher's first space and the characteristics of the student's first space. Let n be the first-space feature of the nth patch in the first-space features of the teacher. The first-space feature of the nth patch in the student's first-space features; and The number of channels are respectively and Twice the size of the space dimension, with the same dimensions.

[0031] S22: After shifting each low-level feature, crop it into shift patch level features. Calculate the shifted local spatial features of each low-level feature based on the shift patch level features. Then, restore the position of each low-level feature by reverse shifting the shifted local spatial features to obtain the second spatial features of each low-level feature.

[0032] Specifically, each low-level feature is shifted and then cropped into shifted patch-level features, as shown in the formula: ; ; in, and The first The characteristics of teachers and students at different levels. For shift operations, and These are the shifted numbers. The characteristics of teachers and students at different levels. For the cropping operation, and The first Teacher-level and student-level shift patch features at different levels. .

[0033] Preferably, the step size of the shift operation is half the step size of the trimming operation; in this embodiment, the step size of the shift operation is 4. The feature is shifted to the lower right by the shift operation, and the portion exceeding the feature size range after shifting is filled to the upper left. Therefore... and The size of the space is respectively with and Maintain consistency; After cropping the shifted feature into a patch-level feature using the cropping operation, crop it again according to the filling edge in the patch block containing the filling part in the upper left corner to obtain the target shifted patch-level feature.

[0034] Reference Figure 3 As shown, where Figure 3 In the image, (a) is the original feature map. Figure 3 (b) in the image is the shifted feature map. Figure 3 In the diagram, (c) represents the feature map after the position is restored following the reverse shift. The red box represents the patch obtained after cropping the feature map. By cropping the original feature map into multiple patches, the spatial features of small targets can be extracted more effectively. Since the relative positions of images at the edges will be disrupted after the shift, it is important to avoid placing images from different regions in the same patch when cropping patches for the edge areas.

[0035] Taking the top-left corner of the original feature as the origin, the shift operation is specifically represented as follows: ; in, The characteristics after shifting, The original feature is represented by x and y, which are the x and y coordinates of the pixel, respectively, and H and W are the height and width of the original feature, respectively.

[0036] The shifted local spatial features of each low-level feature are calculated based on the shifted patch-level features, using the following formula: ; ; in, and These represent the operations of averaging each patch along the y-axis and x-axis and then mapping it back to the patch size. and The first Local spatial characteristics of teacher and student displacement along the y-axis of the hierarchy. and The first Local spatial features of teacher and student displacement along the x-axis of the hierarchy.

[0037] Because there are patches of different sizes in the shifted patch-level features, the size of the shifted local feature will differ from the size of the original local feature.

[0038] After restoring the positions through inverse shifting, the second spatial features of each low-level feature are obtained, as shown in the formula: ; ; ; in, and The first Local spatial features of the teacher after restoring their position in the hierarchical y-axis and x-axis directions. and The first Local spatial features of students after restoring their positions along the hierarchical y-axis and x-axis. This is a reverse shift operation; This is a splicing operation along the channel dimension. and The first The characteristics of the second space of teachers and students at different levels.

[0039] Specifically, the reverse shift operation is represented as: ; in, To restore the features after the position is restored, This represents the shifted feature. Through the reverse shift operation, the feature that was filled in the upper left corner during the shift operation will return to its original position, so that the feature after restoration maintains the same relative position as the original feature before shift.

[0040] S23: The first spatial feature and the second spatial feature of each low-level feature are fused to obtain the fused spatial feature of each low-level feature.

[0041] Specifically, no. The characteristics of teacher integration at different levels are , No. The characteristics of student integration at different levels are .

[0042] This invention integrates the spatial features obtained without shifting and the spatial features obtained after shifting, which not only allows attention to the local features of small targets, but also breaks the independent state between local patches, thereby improving the model's ability to identify small targets.

[0043] S24: calculating a local feature distillation loss according to the teacher fusion spatial feature and the student fusion spatial feature of the corresponding level.

[0044] Preferably, after obtaining the fusion feature of each low-level feature, the method further comprises: normalizing the fusion feature, and the formula is: ; wherein, and are a teacher fusion feature and a student fusion feature of the i-th level, respectively, and are a normalized teacher fusion feature and a normalized student fusion feature of the i-th level, respectively. calculating a local feature distillation loss according to the teacher fusion feature and the student fusion feature of the corresponding level, and the formula is:

[0045] ; wherein, and are a normalized teacher fusion feature and a normalized student fusion feature of the i-th level, respectively, is a local feature distillation loss.

[0046] Specifically, in S3, the high-level features in the teacher feature and the student feature are input into a deep pixel-by-pixel distillation module, and a pixel-by-pixel distillation loss is calculated according to the distance between all corresponding pixels of the teacher feature and the student feature of the corresponding level, and the formula is: ; ; wherein, is a pixel-by-pixel distillation loss, and are a teacher feature and a student feature of the i-th level, respectively, is a flattening operation, and are a teacher flattened vector and a student flattened vector of the i-th level, respectively, and are i-th pixel values of and , respectively, and , is a total number of pixels, .

[0047] Specifically, in S5, the predicted angle of the teacher model is taken as the real angle, and the predicted angle of the student model is taken as the generated angle, and a generative adversarial distillation loss is calculated, comprising:​​​ Let the predicted angle of the teacher model be denoted as , and the predicted angle of the student model be denoted as , the input of the discriminator, and the respective discrimination results are obtained, and the formula is: ; ; wherein, is the discrimination result of the predicted angle of the teacher model, is the discrimination result of the predicted angle of the student model; is a discriminator composed of a multilayer perceptron, used to distinguish the predicted angle of the teacher model and the predicted angle of the student model. The greater the value of the discrimination result, the greater the probability that the predicted angle comes from the teacher model, and vice versa. The generative adversarial distillation loss includes a discrimination loss and a generation loss, wherein the formula of the discrimination loss is: ; wherein, is the discrimination loss, is the mean operation.

[0048] The formula of the generation loss is: ; wherein, is the generation loss.

[0049] By minimizing the discrimination loss and the generation loss, the angle prediction result of the student model can be made closer to the angle prediction result of the teacher model, so as to improve the prediction ability of the student model for the target angle.

[0050] Specifically, in S6, the local feature distillation loss, the pixel-by-pixel distillation loss, the discrimination loss, the generation loss and the supervision loss constitute the total loss, and the formula is: ; wherein, is the total loss, and is the supervision loss; , and are loss weights, and is the optimal effect.

[0051] The supervision loss refers to the loss between the real label of the input remote sensing image and the predicted classification, predicted position and predicted angle output by the student model, for example, FcoalLoss and RotatedIoULoss can be selected as the classification loss and position loss. By training the student model, the local spatial features contained in the low-level features, the high-level semantic knowledge of the high-level features and the angle prediction priori knowledge in the teacher model can be transferred to the student model.

[0052] To verify the effectiveness of the method of the present application, this embodiment counts the accuracy of each target class when the shift local feature cooperative distillation module, the deep pixel-by-pixel distillation and the generative adversarial distillation are used alone, as shown in Table 2.

[0053] Table 2, accuracy of each target class when different distillation modules are used alone

[0054] From Table 2, it can be seen that the shift local feature cooperative distillation module has outstanding accuracy improvement in the detection tasks of small vehicles, aircrafts and ships, and is suitable for the “small target, dense distribution” category in remote sensing target detection.

[0055] Table 3 is the accuracy improvement result when the shift local feature cooperative distillation module, the deep pixel-by-pixel distillation and the generative adversarial distillation are applied to the training of the student model. It can be seen that after the three modules work together, the average precision is improved by 3.7 compared with the baseline model.

[0056] Table 3, results of each module application

[0057] Figure 4 The comparison results between the method of the present application and the baseline model are shown in the following figures, wherein Figure 4 (a) and (c) in Figure 1 are examples of target detection results of the student model trained by the method of the present application, Figure 4 (b) and (d) in Figure 1 are examples of target detection results of the baseline student model. It can be seen that the method of the present application significantly improves the detection accuracy of the student model in remote sensing small target detection.

[0058] To sum up, the remote sensing target detection method based on multi-level knowledge distillation provided by the application inputs the shallow features output by the teacher model and the student model into a shift local feature cooperative distillation module, divides the shallow features into multiple local regions and calculates local spatial features, enhances the modeling capability for local spatial structures and detailed features in the remote sensing image, and more accurately aligns and distills the shallow features of the teacher network and the student network through the shift operation, effectively solves the problem of lack of local modeling in the traditional knowledge distillation method, and effectively migrates the spatial structure of the shallow features. Furthermore, the application further inputs the deep features output by the teacher model and the student model into a deep pixel-by-pixel distillation module to effectively migrate the semantic information of the deep features. Moreover, the application constructs a generative adversarial distillation loss based on the prediction angles of the teacher model and the student model to effectively migrate the angle prediction knowledge and improve the accuracy of the student model in predicting the angles of the targets in the remote sensing image. The student model obtained by migration significantly improves the prediction accuracy of the remote sensing targets while maintaining high prediction efficiency, and meets the actual needs of the remote sensing application scenarios.

[0059] Based on the remote sensing target detection method based on multi-level knowledge distillation, the application further provides a remote sensing target detection device based on multi-level knowledge distillation, comprising: a feature extraction module for inputting a remote sensing image into a trained teacher model and a student model respectively, and outputting five levels of teacher features and five levels of student features; a local feature distillation loss calculation module for inputting low-level features in the teacher features and the student features into a shift local feature cooperative distillation module, and calculating a shift local feature loss, comprising: cutting each low-level feature into a patch-level feature, and calculating a first spatial feature of each low-level feature according to the patch-level feature; cutting each low-level feature into a shift patch-level feature after shifting, calculating a shift local spatial feature of each low-level feature according to the shift patch-level feature, and then obtaining a second spatial feature of each low-level feature by restoring the position of the shift local spatial feature after reverse shifting; fusing the first spatial feature and the second spatial feature of each low-level feature to obtain a fused spatial feature of each low-level feature; calculating a local feature distillation loss according to the teacher fused spatial feature and the student fused spatial feature of the corresponding level; a pixel-by-pixel distillation loss calculation module for inputting high-level features in the teacher features and the student features into a deep pixel-by-pixel distillation module, and calculating a pixel-by-pixel distillation loss according to the distance between all corresponding pixels between the teacher features and the student features of the corresponding level; The detection module is configured to input the teacher feature and the student feature into a detection head to obtain a predicted classification, a predicted position and a predicted angle of the target corresponding to the teacher model and the student model respectively. The discriminant generation loss calculation module is configured to take the predicted angle of the teacher model as a real angle and take the predicted angle of the student model as a generated angle to calculate a generative adversarial distillation loss. The training module is configured to train the student model based on a total loss composed of the local feature distillation loss, the pixel-wise distillation loss, the generative adversarial distillation loss and the supervision loss. The inference module is configured to input a remote sensing image to be detected into the trained student model to obtain a predicted classification, a predicted position and a predicted angle of the target.

[0060] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0061] The present application is described with reference to flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0062] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0063] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing devices provide steps for implementing the function specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0064] Obviously, the above-described embodiments are only examples for clearly illustrating the present application, and are not intended to limit the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments are not required to be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A remote sensing target detection method based on multi-level knowledge distillation, characterized in that, include: The remote sensing images are input into the trained teacher model and student model respectively, and the output consists of five levels of teacher features and five levels of student features. The low-level features from teacher and student features are input into the shifted local feature co-distillation module to calculate the shifted local feature loss, including: Each low-level feature is cropped into a patch-level feature, and the first spatial feature of each low-level feature is calculated based on the patch-level features. Each low-level feature is shifted and cropped into shift patch-level features. The shifted local spatial features of each low-level feature are calculated based on the shift patch-level features. The second spatial features of each low-level feature are obtained by inverse shifting the shifted local spatial features to restore their positions. By fusing the first and second spatial features of each low-level feature, a fused spatial feature of each low-level feature is obtained. Calculate the local feature distillation loss based on the teacher integration space characteristics and student integration space characteristics at the corresponding level; The high-level features from the teacher and student features are input into the deep pixel-by-pixel distillation module, and the pixel-by-pixel distillation loss is calculated based on the distance between all corresponding pixels between the teacher and student features at the corresponding level. By inputting teacher and student features into the detection head, the predicted classification, predicted location, and predicted angle of the target corresponding to the teacher model and student model are obtained respectively. The prediction angle of the teacher model is used as the true angle, and the prediction angle of the student model is used as the generation angle to calculate the generation adversarial distillation loss. The student model is trained using a total loss consisting of local feature distillation loss, pixel-wise distillation loss, generative adversarial distillation loss, and supervisory loss. The remote sensing image to be detected is input into the trained student model to obtain the predicted classification, predicted location, and predicted angle of the target.

2. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, The teacher model is an FCOS model based on Swin-T, and the student model is an FCOS model based on Swin-TT.

3. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, Low-level features are the features of the first and second levels, while high-level features are the features of the fourth and fifth levels.

4. The remote sensing target detection method based on multi-level knowledge distillation according to claim 3, characterized in that, Each low-level feature is cropped into a patch-level feature, using the following formula: ； in, and The first The characteristics of teachers and students at different levels. and The first Teacher-level patch features and student-level patch features at different levels. For the cropping operation, ; The first spatial feature of each low-level feature is calculated based on the patch-level features, using the following formula: ； ； ； in, and These represent the operations of averaging each patch along the y-axis and x-axis and then mapping it back to the patch size. and The first Local spatial features of teachers and students along the y-axis of the hierarchy. and The first Local spatial features of teachers and students along the x-axis of the hierarchy. This is a splicing operation along the channel dimension. and The first The first-space characteristics of teachers and students at different levels.

5. The remote sensing target detection method based on multi-level knowledge distillation according to claim 3, characterized in that, Each low-level feature is shifted and then cropped into shifted patch-level features, as shown in the formula: ； ； in, and The first The characteristics of teachers and students at different levels. For shift operations, and These are the shifted numbers. The characteristics of teachers and students at different levels. For the cropping operation, and The first Teacher-level and student-level shift patch features at different levels. ; The shifted local spatial features of each low-level feature are calculated based on the shifted patch-level features, using the following formula: ； ； in, and These represent the operations of averaging each patch along the y-axis and x-axis and then mapping it back to the patch size. and The first Local spatial characteristics of teacher and student displacement along the y-axis of the hierarchy. and The first Local spatial characteristics of teacher and student displacement along the x-axis of the hierarchy; After restoring the positions through inverse shifting, the second spatial features of each low-level feature are obtained, as shown in the formula: ； ； ； in, and The first Local spatial features of the teacher after restoring their position in the hierarchical y-axis and x-axis directions. and The first Local spatial features of students after restoring their positions along the hierarchical y-axis and x-axis. This is a reverse shift operation; This is a splicing operation along the channel dimension. and The first The characteristics of the second space of teachers and students at different levels.

6. The remote sensing target detection method based on multi-level knowledge distillation according to claim 5, characterized in that, The step size of the shift operation is half the step size of the trim operation. The shift operation moves the feature to the lower right, and the part that exceeds the feature size range after the shift is filled to the upper left. After cropping the shifted feature into a patch-level feature using the cropping operation, crop it again according to the filling edge in the patch block containing the filling part in the upper left corner to obtain the target shifted patch-level feature.

7. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, After obtaining the fused features of each low-level feature, it also includes: The fusion features are normalized using the following formula: ； in, and The first The characteristics of teacher integration and student integration at different levels. and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization.

8. The remote sensing target detection method based on multi-level knowledge distillation according to claim 7, characterized in that, The local feature distillation loss is calculated based on the teacher fusion characteristics and student fusion characteristics at the corresponding level, using the following formula: ； in, and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization This refers to localized distillation losses.

9. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, The high-level features from the teacher and student features are input into the deep pixel-by-pixel distillation module. The pixel-by-pixel distillation loss is calculated based on the distance between all corresponding pixels of the teacher and student features at the corresponding level. The formula is as follows: ； ； in, For pixel-by-pixel distillation loss, and The first The characteristics of teachers and students at different levels. For flattening operation, and The first The teacher expansion vector and student expansion vector at each level. and They are respectively and The value of the i-th pixel, Total number of pixels .

10. A remote sensing target detection device based on multi-level knowledge distillation, characterized in that, include: The feature extraction module is used to input remote sensing images into the trained teacher model and student model respectively, and output five levels of teacher features and five levels of student features. The local feature distillation loss calculation module is used to input low-level features from teacher and student features into the shifted local feature co-distillation module to calculate the shifted local feature loss, including: Each low-level feature is cropped into a patch-level feature, and the first spatial feature of each low-level feature is calculated based on the patch-level features. Each low-level feature is shifted and cropped into shift patch-level features. The shifted local spatial features of each low-level feature are calculated based on the shift patch-level features. The second spatial features of each low-level feature are obtained by inverse shifting the shifted local spatial features to restore their positions. By fusing the first and second spatial features of each low-level feature, a fused spatial feature of each low-level feature is obtained. Calculate the local feature distillation loss based on the teacher integration space characteristics and student integration space characteristics at the corresponding level; The pixel-by-pixel distillation loss calculation module is used to input high-level features from teacher features and student features into the deep pixel-by-pixel distillation module, and calculate the pixel-by-pixel distillation loss based on the distance between all corresponding pixels between the teacher features and student features at the corresponding level. The detection module is used to input teacher features and student features into the detection head, and obtain the predicted classification, predicted position and predicted angle of the target corresponding to the teacher model and the student model, respectively. The discriminant generation loss calculation module is used to calculate the generation adversarial distillation loss by using the prediction angle of the teacher model as the true angle and the prediction angle of the student model as the generation angle. The training module is used to train the student model with a total loss consisting of local feature distillation loss, pixel-wise distillation loss, generative adversarial distillation loss, and supervisory loss. The inference module is used to input the remote sensing image to be detected into the trained student model to obtain the predicted classification, predicted location, and predicted angle of the target.

Images