A remote sensing target detection method and device based on multi-level knowledge distillation

Abstract

The present application relates to remote sensing target detection technical field, especially in kind, a kind of remote sensing target detection method and device based on multilevel knowledge distillation, comprising: the remote sensing image is input respectively teacher model and student model, extracts multilevel feature;With shift local feature collaborative distillation module, low-level feature is distilled, and the identification ability of student model to small target is enhanced using local space feature modeling;With deep layer pixel-by-pixel distillation module, high-level feature is distilled pixel by pixel, and classification ability is improved;The angle prediction of teacher network and student network is generated and is antagonized and distilled, and the precision of angle prediction is improved;Finally, the student model is optimized in combination with each loss.The student model trained by the present application significantly improves the prediction accuracy of remote sensing target, while maintaining high prediction efficiency, adapting to the actual needs of remote sensing application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing target detection, in particular to a remote sensing target detection method and device based on multi-level knowledge distillation. BACKGROUND

[0002] Remote sensing target detection is a technology that automatically identifies and locates specific targets (such as airplanes, vehicles, buildings, ships, etc.) from remote sensing images. It combines image processing, pattern recognition, and deep learning methods to achieve class judgment and spatial location (such as bounding boxes) labeling of targets, and is widely used in urban planning, military reconnaissance, environmental monitoring, disaster assessment, and other fields. It is a key link in remote sensing information extraction and application.

[0003] With the continuous development of deep learning technology, remote sensing target detection models can achieve high detection accuracy due to their strong expression ability. However, as the accuracy improves, the model often has problems such as large number of model parameters and high computational complexity, making it difficult to deploy in practical applications, especially in some computing resource limited scenarios (such as drones, edge devices, etc.).

[0004] Knowledge distillation technology is an effective model compression and knowledge transfer method. Its core idea is to transfer the knowledge learned by a mature complex "teacher model" to a simpler "student model" with fewer parameters, so that the student model can maintain close to the performance of the teacher model while significantly reducing computational cost and storage requirements, facilitating deployment on resource-constrained devices.

[0005] However, traditional knowledge distillation methods are mostly designed for natural images, while remote sensing targets have their unique characteristics, leading to many challenges when applying traditional knowledge distillation methods to remote sensing target detection tasks. Specifically, these challenges include:

[0006] (1) Multi-scale target distribution: The size of targets in remote sensing images varies greatly, with a high proportion of small targets. Traditional knowledge distillation methods have a single feature alignment method, focusing only on global features or simple feature matching, lacking effective modeling of local spatial structures and detailed features in remote sensing images, making it difficult to achieve local precise alignment of features between teacher and student models, resulting in the inability of the transferred student model to accurately capture multi-scale targets, especially small targets;

[0007] (2) Spatial position sensitivity: In remote sensing target detection, spatial position information is crucial. Traditional distillation methods lack in transferring positioning information, which may ignore spatial structure information of features, making it difficult to effectively improve the positioning accuracy of the student model;

[0008] (3) has angle information: in the remote sensing target detection task, the angle information of the target is also an important detection dimension, and the traditional knowledge distillation method often lacks effective distillation mechanism for angle prediction and other direction information, resulting in low prediction accuracy of the student model in the angle information.

[0009] Therefore, how to apply the knowledge distillation method to the remote sensing target detection task to improve the detection efficiency and effectively improve the accuracy of target detection is a problem to be solved in the prior art. SUMMARY

[0010] Therefore, the technical problem to be solved by the present application is to overcome the problem that the knowledge distillation method is applied to the remote sensing target detection task in the prior art, and the accuracy of target detection cannot be effectively improved.

[0011] To solve the above technical problems, the present application provides a remote sensing target detection method based on multi-level knowledge distillation, comprising:

[0012] The remote sensing image is input into the trained teacher model and student model, and five levels of teacher features and five levels of student features are output;

[0013] The low-level features in the teacher features and student features are input into the shift local feature cooperative distillation module, and the shift local feature loss is calculated, comprising:

[0014] Each low-level feature is cut into a patch-level feature, and the first spatial feature of each low-level feature is calculated according to the patch-level feature;

[0015] Each low-level feature is cut into a shift patch-level feature after being shifted, and the shift local spatial feature of each low-level feature is calculated according to the shift patch-level feature, and then the second spatial feature of each low-level feature is obtained by restoring the position after the shift local spatial feature is shifted back;

[0016] The first spatial feature and the second spatial feature of each low-level feature are fused to obtain the fusion spatial feature of each low-level feature;

[0017] The local feature distillation loss is calculated according to the teacher fusion spatial feature and the student fusion spatial feature of the corresponding level;

[0018] The high-level features in the teacher features and student features are input into the deep pixel-by-pixel distillation module, and the pixel-by-pixel distillation loss is calculated according to the distance between all corresponding pixels between the teacher features and student features of the corresponding level;

[0019] The teacher features and student features are input into the detection head to obtain the predicted classification, predicted position and predicted angle of the target corresponding to the teacher model and student model, respectively;

[0020] The predicted angle of the teacher model is taken as the real angle, and the predicted angle of the student model is taken as the generated angle to calculate the generative adversarial distillation loss.

[0021] The student model is trained by a total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss and the supervision loss.

[0022] The remote sensing image to be detected is input into the trained student model to obtain the predicted classification, the predicted position and the predicted angle of the target.

[0023] Preferably, the teacher model is an FCOS model based on Swin-T, and the student model is an FCOS model based on Swin-T-T.

[0024] Preferably, the low-level features are the features of the first and second levels, and the high-level features are the features of the fourth and fifth levels.

[0025] Preferably, each low-level feature is cropped into a patch-level feature, and the formula is:

[0026] ;

[0027] wherein, and are the teacher feature and the student feature of the i-th level, respectively, and are the teacher patch-level feature and the student patch-level feature of the i-th level, respectively, is a cropping operation, ; According to the patch-level feature, the first spatial feature of each low-level feature is calculated, and the formula is:

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] wherein, and represent an operation of averaging each patch along the y-axis direction and the x-axis direction and then mapping back to the patch size, and are the teacher local spatial feature and the student local spatial feature of the i-th level along the y-axis direction, respectively, and are the teacher local spatial feature and the student local spatial feature of the i-th level along the x-axis direction, respectively, ​​Local spatial features of teachers and students along the x-axis of the hierarchy. This is a splicing operation along the channel dimension. and The first The first-space characteristics of teachers and students at different levels.

[0033] Preferably, each low-level feature is shifted and then cropped into shifted patch-level features, as shown in the formula:

[0034] ;

[0035] ;

[0036] in, and The first The characteristics of teachers and students at different levels. For shift operations, and These are the shifted numbers. The characteristics of teachers and students at different levels. For the cropping operation, and The first Teacher-level and student-level shift patch features at different levels. ;

[0037] The shifted local spatial features of each low-level feature are calculated based on the shifted patch-level features, using the following formula:

[0038] ;

[0039] ;

[0040] in, and These represent the operations of averaging each patch along the y-axis and x-axis and then mapping it back to the patch size. and The first Local spatial characteristics of teacher and student displacement along the y-axis of the hierarchy. and The first Local spatial characteristics of teacher and student displacement along the x-axis of the hierarchy;

[0041] After restoring the positions through inverse shifting, the second spatial features of each low-level feature are obtained, as shown in the formula:

[0042] ;

[0043] ;

[0044] ;

[0045] in, and The first Local spatial features of the teacher after restoring their position in the hierarchical y-axis and x-axis directions. and The first Local spatial features of students after restoring their positions along the hierarchical y-axis and x-axis. This is a reverse shift operation; This is a splicing operation along the channel dimension. and The first The characteristics of the second space of teachers and students at different levels.

[0046] Preferably, the step size of the shift operation is half the step size of the trimming operation. The feature is shifted to the lower right by the shift operation, and the part that exceeds the feature size range after the shift is filled to the upper left.

[0047] After cropping the shifted feature into a patch-level feature using the cropping operation, crop it again according to the filling edge in the patch block containing the filling part in the upper left corner to obtain the target shifted patch-level feature.

[0048] Preferably, after obtaining the fused features of each low-level feature, the method further includes:

[0049] The fusion features are normalized using the following formula:

[0050] ;

[0051] in, and The first The characteristics of teacher integration and student integration at different levels. and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization.

[0052] Preferably, the local feature distillation loss is calculated based on the teacher fusion characteristics and student fusion characteristics at the corresponding level, using the following formula:

[0053] ;

[0054] in, and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization This refers to localized distillation losses.

[0055] Preferably, high-level features from teacher and student features are input into the deep pixel-by-pixel distillation module. The pixel-by-pixel distillation loss is calculated based on the distance between all corresponding pixels of the teacher and student features at the corresponding level, using the following formula:

[0056] ;

[0057] ;

[0058] in, For pixel-by-pixel distillation loss, and The first The characteristics of teachers and students at different levels. For flattening operation, and The first The teacher flattening vector and student flattening vector at each level. and They are respectively and The value of the i-th pixel, Total number of pixels .

[0059] The present invention also provides a remote sensing target detection device based on multi-level knowledge distillation, comprising:

[0060] The feature extraction module is used to input remote sensing images into the trained teacher model and student model respectively, and output five levels of teacher features and five levels of student features.

[0061] The local feature distillation loss calculation module is used to input low-level features from teacher and student features into the shifted local feature co-distillation module to calculate the shifted local feature loss, including:

[0062] Each low-level feature is cropped into a patch-level feature, and the first spatial feature of each low-level feature is calculated based on the patch-level features.

[0063] Each low-level feature is shifted and cropped into shift patch-level features. The shifted local spatial features of each low-level feature are calculated based on the shift patch-level features. The second spatial features of each low-level feature are obtained by inverse shifting the shifted local spatial features to restore their positions.

[0064] By fusing the first and second spatial features of each low-level feature, a fused spatial feature of each low-level feature is obtained.

[0065] The local feature distillation loss is calculated according to the teacher fusion space feature and the student fusion space feature of the corresponding level;

[0066] The pixel-by-pixel distillation loss calculation module is configured to input the high-level features in the teacher feature and the student feature into a deep pixel-by-pixel distillation module, and calculate a pixel-by-pixel distillation loss according to the distance between all corresponding pixels between the teacher feature and the student feature of the corresponding level;

[0067] The detection module is configured to input the teacher feature and the student feature into a detection head to obtain the predicted classification, the predicted position and the predicted angle of the target corresponding to the teacher model and the student model, respectively.

[0068] The discriminative generation loss calculation module is configured to take the predicted angle of the teacher model as the real angle and take the predicted angle of the student model as the generated angle, and calculate the generative adversarial distillation loss.

[0069] The training module is configured to train the student model by using the total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss and the supervision loss.

[0070] The inference module is configured to input the remote sensing image to be detected into the student model after training to obtain the predicted classification, the predicted position and the predicted angle of the target.

[0071] The above technical solutions of the present application have the following beneficial effects compared with the prior art:

[0072] The remote sensing target detection method based on multi-level knowledge distillation disclosed in the present application inputs the shallow features output by the teacher model and the student model into a shift local feature cooperative distillation module, divides the shallow features into multiple local regions and calculates local space features, thereby enhancing the modeling capability for local space structures and detailed features in the remote sensing image, and more accurately realizing the alignment and distillation of the shallow features of the teacher network and the student network by using the shift operation, effectively solving the problem of lack of local modeling in the traditional knowledge distillation method, realizing the effective migration of the spatial structure of the shallow features, and enhancing the recognition capability for small targets in the remote sensing image. Furthermore, the present application inputs the deep features output by the teacher model and the student model into a deep pixel-by-pixel distillation module to realize the effective migration of the semantic information of the deep features and improve the classification capability of the model. Moreover, the present application constructs a generative adversarial distillation loss based on the predicted angles of the teacher model and the student model, realizes the effective migration of the angle prediction knowledge, and improves the accuracy of the student model in predicting the angle of the target in the remote sensing image. The student model obtained by training in the present application significantly improves the prediction accuracy of the remote sensing target while maintaining a high prediction efficiency, and meets the actual needs of the remote sensing application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0073] ​In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the accompanying drawings, in which:

[0074] Figure 1 is a flow chart of a remote sensing target detection method based on multi-level knowledge distillation of the present application;

[0075] Figure 2 is a structure diagram of the overall distillation;

[0076] Figure 3 is a schematic diagram of the shift local feature collaborative distillation module, wherein Figure 3 (a) in is the original feature map, Figure 3 (b) in is the feature map after shifting, Figure 3 (c) in is the feature map after recovering the position after reverse shifting;

[0077] Figure 4 is a comparison result diagram of the method of the present application and the baseline model, wherein Figure 4 (a) and (c) in are example diagrams of target detection results of the student model trained by the method of the present application, Figure 4 (b) and (d) in are example diagrams of target detection results of the baseline student model. DETAILED DESCRIPTION

[0078] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting to the present application.

[0079] Referring to Figure 1 and Figure 2 , the present application provides a remote sensing target detection method based on multi-level knowledge distillation, comprising:

[0080] S1: inputting the remote sensing image into the trained teacher model and student model respectively, and outputting five levels of teacher features and five levels of student features;

[0081] S2: inputting the low-level features in the teacher features and student features into the shift local feature collaborative distillation module, and calculating the shift local feature loss;

[0082] S3: inputting the high-level features in the teacher features and student features into the deep pixel-by-pixel distillation module, and calculating the pixel-by-pixel distillation loss according to the distance between all corresponding pixels between the corresponding level of teacher features and student features;

[0083] S4: inputting the teacher features and student features into the detection head, and respectively obtaining the predicted classification, predicted position and predicted angle of the target corresponding to the teacher model and student model;

[0084] S5: taking the predicted angle of the teacher model as the real angle and the predicted angle of the student model as the generated angle, calculating a generative adversarial distillation loss;

[0085] S6: training the student model based on a total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss, and the supervision loss;

[0086] S7: inputting the remote sensing image to be detected into the trained student model to obtain the predicted classification, the predicted position, and the predicted angle of the target.

[0087] Specifically, in S1, the teacher model is an FCOS (Fully Convolutional One-Stage Object Detection) model based on Swin-T, and the student model is an FCOS model based on Swin-T-T. The network parameters of the teacher model and the student model are shown in Table 1.

[0088] Table 1, network parameter table

[0089]

[0090] The Swin-T-T structure is a simplified Swin-T structure. The Swin-T structure includes four stages, and the number of transformer blocks set in each stage of the Swin-T-T structure is 1, 1, 2, and 1, respectively, and the number of attention heads set in each stage is 2, 4, 8, and 8, respectively. The Swin-T-T structure has a smaller parameter amount of 13.63M than the Swin-T structure as a whole.

[0091] The data set used for training in this embodiment is the currently disclosed DOTA1.0 data set, and the test set is used as the validation set. The remote sensing image is outputted as five levels of teacher features after passing through the teacher model, denoted as , and is outputted as five levels of student features after passing through the student model, denoted as , wherein .

[0092] Preferably, the low-level features are defined as the features of the first and second levels, and the high-level features are defined as the features of the fourth and fifth levels.

[0093] Specifically, in S2, the low-level features in the teacher features and the student features are inputted into the shift local feature collaborative distillation module to calculate the shift local feature loss, including:

[0094] S21: each low-level feature is respectively cropped into a patch-level feature, and a first spatial feature of each low-level feature is calculated according to the patch-level feature.

[0095] Each low-level feature is respectively cropped into patch-level feature, formula is:

[0096] ;

[0097] Among them, and respectively the teacher feature and student feature of the first level, and respectively the teacher patch-level feature and student patch-level feature of the first level, is the cropping operation, .

[0098] Preferably, the step size of the cropping operation is set to 8, and the size of each patch in the obtained teacher patch-level feature and student patch-level feature is 8x8, is the nth patch in the teacher patch-level feature of the first level, is the nth patch in the student patch-level feature of the first level, and n is the total number of patches.

[0099] According to the patch-level feature, the first spatial feature of each low-level feature is calculated, formula is:

[0100] ;

[0101] ;

[0102] ;

[0103] Among them, and respectively represent the operation of mapping back to the patch size after averaging each patch along the y-axis direction and the x-axis direction, and respectively the teacher local spatial feature and student local spatial feature in the y-axis direction of the first level, is the y-axis direction local spatial feature of the nth patch in the teacher local spatial feature in the y-axis direction of the first level, is the y-axis direction local spatial feature of the nth patch in the student local spatial feature in the y-axis direction of the first level; and respectively the teacher local spatial feature and student local spatial feature in the x-axis direction of the first level, is the x-axis direction local spatial feature of the nth patch in the teacher local spatial feature in the x-axis direction of the first the x-axis direction local spatial feature of the n-th patch in the x-axis direction local spatial feature of the teacher of the layer level, the first spatial feature of the n-th patch in the first spatial feature of the teacher of the layer level, the x-axis direction local spatial feature of the n-th patch in the x-axis direction local spatial feature of the student of the layer level; the concatenation operation along the channel dimension, and the first spatial feature of the n-th patch in the first spatial feature of the teacher of the layer level, the first spatial feature of the n-th patch in the first spatial feature of the student of the layer level, the first spatial feature of the n-th patch in the first spatial feature of the teacher of the layer level, the first spatial feature of the n-th patch in the first spatial feature of the student of the layer level; and the number of channels of the teacher feature and the student feature of the layer level, and the number of channels of the teacher feature and the student feature of the layer level.

[0104] S22: each low-level feature is shifted and cropped to obtain a shifted patch-level feature, the shifted local spatial feature of each low-level feature is calculated according to the shifted patch-level feature, and the second spatial feature of each low-level feature is obtained by restoring the position through inverse shifting of the shifted local spatial feature.

[0105] Specifically, each low-level feature is shifted and cropped to obtain a shifted patch-level feature, and the formula is:

[0106]

[0107]

[0108] wherein, and the teacher feature and the student feature of the layer level, the shifting operation, and the teacher feature and the student feature of the layer level after shifting, the cropping operation, and the teacher shifted patch-level feature and the student shifted patch-level feature of the layer level,

[0109] Preferably, the step size of the shifting operation is half of the step size of the cropping operation, and in the present embodiment, the step size of the shifting operation is 4; through the shifting operation, the feature is shifted to the lower right, and the part exceeding the feature size range after shifting is filled to the upper left; therefore and ​​​​​​The spatial sizes of the features are respectively consistent with and ;

[0110] After the shifted features are cropped into patch-level features by the cropping operation, the target shifted patch-level features are obtained by cropping the patch block containing the padding part in the upper left corner again according to the padding edge.

[0111] Referring to Figure 3 , where Figure 3 (a) in (b) is the shifted feature map, Figure 3 (c) is the feature map after the inverse shift and the position recovery. The red box is the patch obtained after cropping the feature map. By cropping the original feature map into multiple patches, the spatial features of small targets can be better extracted. Since the relative positions of the pictures at the edge will be disturbed after the shift, the pictures in different areas should be avoided in the same patch when cropping the patch at the edge. Figure 3 Taking the upper left corner of the original feature as the origin, the shift operation is specifically represented as:

[0112]

[0113] ;

[0114] wherein, is the shifted feature, is the original feature, x and y are the x and y coordinates of the pixel, and H and W are the height and width of the original feature, respectively.

[0115] According to the shifted patch-level features, the shifted local spatial features of each low-level feature are calculated, and the formula is:

[0116] ;

[0117] ;

[0118] wherein, and respectively represent the operation of mapping back to the patch size after averaging each patch along the y-axis and the x-axis, and are the teacher shifted local spatial features and the student shifted local spatial features of the y-axis of the level, and are the teacher shifted local spatial features and the student shifted local spatial features of the x-axis of the level.

[0119] Since there are patches of different sizes in the shifted patch-level features, the size of the shifted local features will be different from the size of the original local features.​

[0120] After recovering the position by inverse displacement, the second spatial features of each low-level feature are obtained, and the formula is:

[0121]

[0122]

[0123]

[0124] wherein, and are the recovered position of the teacher local spatial features in the y-axis direction and the x-axis direction of the first level y respectively, and are the recovered position of the student local spatial features in the y-axis direction and the x-axis direction of the first level y respectively, is an inverse displacement operation; is a concatenation operation along the channel dimension, and are the second spatial features of the teacher and the student of the first level y respectively. Specifically, the inverse displacement operation is represented as:

[0125]

[0126]

[0127] wherein, is the feature after recovering the position, is the feature after displacement. Through the inverse displacement operation, the feature filled into the upper left in the displacement operation will return to its original position, so that the feature after recovering the position and the original feature before displacement keep the relative position unchanged.

[0128] S23: fuse the first spatial features and the second spatial features of each low-level feature to obtain the fused spatial features of each low-level feature.

[0129] Specifically, the teacher fused feature of the first level y is , and the student fused feature of the first level y is . The present application can not only pay attention to the local features of small targets, but also break the independent state between local patches by fusing the spatial features without displacement and the spatial features obtained after displacement, so as to better improve the recognition ability of the model to small targets.

[0130] The present application can not only pay attention to the local features of small targets, but also break the independent state between local patches by fusing the spatial features without displacement and the spatial features obtained after displacement, so as to better improve the recognition ability of the model to small targets.

[0131] ​​​​​​​​S24: Calculate the local feature distillation loss based on the teacher fusion space characteristics and student fusion space characteristics of the corresponding level.

[0132] Preferably, after obtaining the fused features of each low-level feature, the method further includes:

[0133] The fusion features are normalized using the following formula:

[0134] ;

[0135] in, and The first The characteristics of teacher integration and student integration at different levels. and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization.

[0136] The local feature distillation loss is calculated based on the teacher fusion characteristics and student fusion characteristics at the corresponding level, using the following formula:

[0137] ;

[0138] in, and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization This refers to localized distillation losses.

[0139] Specifically, in S3, high-level features from teacher and student features are input into the deep pixel-by-pixel distillation module. The pixel-by-pixel distillation loss is calculated based on the distance between all corresponding pixels of the teacher and student features at the corresponding level. The formula is as follows:

[0140] ;

[0141] ;

[0142] in, For pixel-by-pixel distillation loss, and The first The characteristics of teachers and students at different levels. For flattening operation, and The first The teacher expansion vector and student expansion vector at each level. and They are respectively and The value of the i-th pixel, Total number of pixels .

[0143] Specifically, in S5, the predicted angle of the teacher model is taken as the real angle, and the predicted angle of the student model is taken as the generated angle, and the generative adversarial distillation loss is calculated, including:

[0144] Let the predicted angle of the teacher model be , and the predicted angle of the student model be , input the discriminator to obtain the respective discrimination results, and the formula is:

[0145] ;

[0146] ;

[0147] wherein, is the discrimination result of the predicted angle of the teacher model, is the discrimination result of the predicted angle of the student model; is a discriminator composed of a multi-layer perceptron, used to distinguish the predicted angle of the teacher model and the predicted angle of the student model, and the greater the value of the discrimination result, the greater the probability that the predicted angle comes from the teacher model, and vice versa.

[0148] The generative adversarial distillation loss includes a discrimination loss and a generation loss, wherein the formula of the discrimination loss is:

[0149] ;

[0150] wherein, is the discrimination loss, is a mean operation.

[0151] The formula of the generation loss is:

[0152] ;

[0153] wherein, is the generation loss.

[0154] By minimizing the discrimination loss and the generation loss, the angle prediction result of the student model can be made closer to the angle prediction result of the teacher model, so as to improve the prediction ability of the student model for the target angle.

[0155] Specifically, in S6, the local feature distillation loss, the pixel-by-pixel distillation loss, the discrimination loss, the generation loss and the supervision loss constitute a total loss, and the formula is:

[0156] ;

[0157] wherein, is the total loss, to supervise the loss; , and are loss weights, and is optimal.

[0158] The supervision loss refers to the loss between the true label of the input remote sensing image and the predicted classification, predicted position and predicted angle output by the student model, for example, FcoalLoss and RotatedIoULoss can be selected as the classification loss and position loss. By training the student model, the local spatial features contained in the low-level features, the high-level semantic knowledge of the high-level features and the angle prediction priori knowledge in the teacher model can be transferred to the student model.

[0159] To verify the effectiveness of the method of the application, this embodiment counts the accuracy of each target class when the shift local feature cooperative distillation module, the deep pixel-by-pixel distillation and the generative adversarial distillation three distillation modules act alone, as shown in Table 2.

[0160] Table 2, accuracy of each target class when different distillation modules act alone

[0161]

[0162] As can be seen from Table 2, the shift local feature cooperative distillation module has outstanding accuracy improvement in the detection task of small vehicles, aircraft and ships and other targets, and is suitable for the “small target, dense distribution” category in remote sensing target detection.

[0163] Table 3 is the accuracy improvement result when the shift local feature cooperative distillation module, the deep pixel-by-pixel distillation and the generative adversarial distillation are applied to the training of the student model. It can be seen that after the three modules act together, the average precision is improved by 3.7 compared with the baseline model.

[0164] Table 3, results of application of each module

[0165]

[0166] Figure 4 Fig. 1 is a comparison result diagram of the method of the application and the baseline model, wherein Figure 4 (a) and (c) in Fig. 1 are target detection result example diagrams of the student model trained by the method of the application, Figure 4 (b) and (d) in Fig. 1 are target detection result example diagrams of the baseline student model. It can be seen that the method of the application significantly improves the detection accuracy of the student model in remote sensing small target detection.

[0167] In summary, the remote sensing target detection method based on multi-level knowledge distillation provided by the present application inputs the shallow features output by the teacher model and the student model into the shift local feature collaborative distillation module, divides the shallow features into multiple local regions and calculates the local spatial features, thereby enhancing the modeling capability for the local spatial structure and the detailed features in the remote sensing image, and using the shift operation to more accurately align and distill the shallow features of the teacher network and the student network, effectively solving the problem of lack of local modeling in the traditional knowledge distillation method and realizing effective migration of the spatial structure of the shallow features. Furthermore, the present application inputs the deep features output by the teacher model and the student model into the deep pixel-by-pixel distillation module to realize effective migration of the semantic information of the deep features. Moreover, the present application constructs the generative adversarial distillation loss based on the prediction angles of the teacher model and the student model to realize effective migration of the angle prediction knowledge and improve the accuracy of the student model in predicting the angles of the targets in the remote sensing image. The student model obtained by migration of the present application significantly improves the prediction accuracy of the remote sensing targets while maintaining high prediction efficiency, thereby meeting the actual needs of the remote sensing application scenarios.

[0168] Based on the remote sensing target detection method based on multi-level knowledge distillation described above, the present application further provides a remote sensing target detection device based on multi-level knowledge distillation, comprising:

[0169] a feature extraction module configured to input a remote sensing image into a trained teacher model and a student model respectively, and output five levels of teacher features and five levels of student features;

[0170] a local feature distillation loss calculation module configured to input the low-level features in the teacher features and the student features into a shift local feature collaborative distillation module, and calculate a shift local feature loss, comprising:

[0171] cutting each low-level feature into a patch-level feature, and calculating a first spatial feature of each low-level feature according to the patch-level feature;

[0172] cutting each low-level feature into a shifted patch-level feature after shifting, calculating a shifted local spatial feature of each low-level feature according to the shifted patch-level feature, and then obtaining a second spatial feature of each low-level feature by restoring the position of the shifted local spatial feature after reverse shifting;

[0173] fusing the first spatial feature and the second spatial feature of each low-level feature to obtain a fused spatial feature of each low-level feature;

[0174] calculating a local feature distillation loss according to the teacher fused spatial feature and the student fused spatial feature of the corresponding level;

[0175] The pixel-by-pixel distillation loss calculation module is configured to input high-level features in the teacher features and the student features into a deep pixel-by-pixel distillation module, and calculate a pixel-by-pixel distillation loss according to distances between all corresponding pixels between the teacher features and the student features of the corresponding level.

[0176] The detection module is configured to input the teacher features and the student features into a detection head to obtain a predicted classification, a predicted position, and a predicted angle of the target corresponding to the teacher model and the student model, respectively.

[0177] The discriminative generation loss calculation module is configured to take the predicted angle of the teacher model as a real angle and take the predicted angle of the student model as a generated angle to calculate a generative adversarial distillation loss.

[0178] The training module is configured to train the student model based on a total loss composed of the local feature distillation loss, the pixel-by-pixel distillation loss, the generative adversarial distillation loss, and the supervision loss.

[0179] The inference module is configured to input a remote sensing image to be detected into the student model after training to obtain a predicted classification, a predicted position, and a predicted angle of the target.

[0180] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0181] The present application is described with reference to flowcharts and / or block diagrams according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0182] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0183] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0184] Obviously, the above embodiments are only examples for clearly illustrating, not limiting the embodiments. For those skilled in the art, on the basis of the above description, other different forms of changes or variations can also be made. Here, it is not necessary and also impossible to enumerate all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A remote sensing target detection method based on multi-level knowledge distillation, characterized in that, include: The remote sensing images are input into the trained teacher model and student model respectively, and the output consists of five levels of teacher features and five levels of student features. The low-level features from teacher and student features are input into the shifted local feature co-distillation module to calculate the shifted local feature loss, including: Each low-level feature is cropped into a patch-level feature, and the first spatial feature of each low-level feature is calculated based on the patch-level features. Each low-level feature is shifted and cropped into shift patch-level features. The shifted local spatial features of each low-level feature are calculated based on the shift patch-level features. The second spatial features of each low-level feature are obtained by inverse shifting the shifted local spatial features to restore their positions. By fusing the first and second spatial features of each low-level feature, a fused spatial feature of each low-level feature is obtained. Calculate the local feature distillation loss based on the teacher integration space characteristics and student integration space characteristics at the corresponding level; The high-level features from the teacher and student features are input into the deep pixel-by-pixel distillation module, and the pixel-by-pixel distillation loss is calculated based on the distance between all corresponding pixels between the teacher and student features at the corresponding level. By inputting teacher and student features into the detection head, the predicted classification, predicted location, and predicted angle of the target corresponding to the teacher model and student model are obtained respectively. The prediction angle of the teacher model is used as the true angle, and the prediction angle of the student model is used as the generation angle to calculate the generation adversarial distillation loss. The student model is trained using a total loss consisting of local feature distillation loss, pixel-wise distillation loss, generative adversarial distillation loss, and supervisory loss. The remote sensing image to be detected is input into the trained student model to obtain the predicted classification, predicted location, and predicted angle of the target.

2. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, The teacher model is an FCOS model based on Swin-T, and the student model is an FCOS model based on Swin-TT.

3. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, Low-level features are the features of the first and second levels, while high-level features are the features of the fourth and fifth levels.

4. The remote sensing target detection method based on multi-level knowledge distillation according to claim 3, characterized in that, Each low-level feature is cropped into a patch-level feature, using the following formula: ； in, and The first The characteristics of teachers and students at different levels. and The first Teacher-level patch features and student-level patch features at different levels. For the cropping operation, ; The first spatial feature of each low-level feature is calculated based on the patch-level features, using the following formula: ； ； ； in, and These represent the operations of averaging each patch along the y-axis and x-axis and then mapping it back to the patch size. and The first Local spatial features of teachers and students along the y-axis of the hierarchy. and The first Local spatial features of teachers and students along the x-axis of the hierarchy. This is a splicing operation along the channel dimension. and The first The first-space characteristics of teachers and students at different levels.

5. The remote sensing target detection method based on multi-level knowledge distillation according to claim 3, characterized in that, Each low-level feature is shifted and then cropped into shifted patch-level features, as shown in the formula: ； ； in, and The first The characteristics of teachers and students at different levels. For shift operations, and These are the shifted numbers. The characteristics of teachers and students at different levels. For the cropping operation, and The first Teacher-level and student-level shift patch features at different levels. ; The shifted local spatial features of each low-level feature are calculated based on the shifted patch-level features, using the following formula: ； ； in, and These represent the operations of averaging each patch along the y-axis and x-axis and then mapping it back to the patch size. and The first Local spatial characteristics of teacher and student displacement along the y-axis of the hierarchy. and The first Local spatial characteristics of teacher and student displacement along the x-axis of the hierarchy; After restoring the positions through inverse shifting, the second spatial features of each low-level feature are obtained, as shown in the formula: ； ； ； in, and The first Local spatial features of the teacher after restoring their position in the hierarchical y-axis and x-axis directions. and The first Local spatial features of students after restoring their positions along the hierarchical y-axis and x-axis. This is a reverse shift operation; This is a splicing operation along the channel dimension. and The first The characteristics of the second space of teachers and students at different levels.

6. The remote sensing target detection method based on multi-level knowledge distillation according to claim 5, characterized in that, The step size of the shift operation is half the step size of the trim operation. The shift operation moves the feature to the lower right, and the part that exceeds the feature size range after the shift is filled to the upper left. After cropping the shifted feature into a patch-level feature using the cropping operation, crop it again according to the filling edge in the patch block containing the filling part in the upper left corner to obtain the target shifted patch-level feature.

7. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, After obtaining the fused features of each low-level feature, it also includes: The fusion features are normalized using the following formula: ； in, and The first The characteristics of teacher integration and student integration at different levels. and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization.

8. The remote sensing target detection method based on multi-level knowledge distillation according to claim 7, characterized in that, The local feature distillation loss is calculated based on the teacher fusion characteristics and student fusion characteristics at the corresponding level, using the following formula: ； in, and The first Teacher integration characteristics and student integration characteristics after hierarchical normalization This refers to localized distillation losses.

9. The remote sensing target detection method based on multi-level knowledge distillation according to claim 1, characterized in that, The high-level features from the teacher and student features are input into the deep pixel-by-pixel distillation module. The pixel-by-pixel distillation loss is calculated based on the distance between all corresponding pixels of the teacher and student features at the corresponding level. The formula is as follows: ； ； in, For pixel-by-pixel distillation loss, and The first The characteristics of teachers and students at different levels. For flattening operation, and The first The teacher flattening vector and student flattening vector at each level. and They are respectively and The value of the i-th pixel, Total number of pixels .

10. A remote sensing target detection device based on multi-level knowledge distillation, characterized in that, include: The feature extraction module is used to input remote sensing images into the trained teacher model and student model respectively, and output five levels of teacher features and five levels of student features. The local feature distillation loss calculation module is used to input low-level features from teacher and student features into the shifted local feature co-distillation module to calculate the shifted local feature loss, including: Each low-level feature is cropped into a patch-level feature, and the first spatial feature of each low-level feature is calculated based on the patch-level features. Each low-level feature is shifted and cropped into shift patch-level features. The shifted local spatial features of each low-level feature are calculated based on the shift patch-level features. The second spatial features of each low-level feature are obtained by inverse shifting the shifted local spatial features to restore their positions. By fusing the first and second spatial features of each low-level feature, a fused spatial feature of each low-level feature is obtained. Calculate the local feature distillation loss based on the teacher integration space characteristics and student integration space characteristics at the corresponding level; The pixel-by-pixel distillation loss calculation module is used to input high-level features from teacher features and student features into the deep pixel-by-pixel distillation module, and calculate the pixel-by-pixel distillation loss based on the distance between all corresponding pixels between the teacher features and student features at the corresponding level. The detection module is used to input teacher features and student features into the detection head, and obtain the predicted classification, predicted position and predicted angle of the target corresponding to the teacher model and the student model, respectively. The discriminant generation loss calculation module is used to calculate the generation adversarial distillation loss by using the prediction angle of the teacher model as the true angle and the prediction angle of the student model as the generation angle. The training module is used to train the student model with a total loss consisting of local feature distillation loss, pixel-wise distillation loss, generative adversarial distillation loss, and supervisory loss. The inference module is used to input the remote sensing image to be detected into the trained student model to obtain the predicted classification, predicted location, and predicted angle of the target.

Images