Rotating small target detection method and system based on heterogeneous task model knowledge distillation

Abstract

The invention discloses a rotating small target detection method and system based on heterogeneous task model knowledge distillation, and the method specifically comprises the steps: firstly, selecting a rotating target detection algorithm as a reference rotating target detector; constructing a lightweight feature map embedding super-resolution module, and embedding the module between a backbone network and a neck network of the reference rotating target detector to obtain a super-resolution rotating target detector; then constructing a feature coding module, a feature decoding module and a feature up-sampling module, and combining the modules as parallel super-resolution network branches; aligning output features of the two types of heterogeneous models by using a feature affine layer and feature alignment loss, adopting end-to-end joint training and optimizing a rotating target detector and a parallel super-resolution network branch of an embedded feature map embedded super-resolution module at the same time, and realizing knowledge distillation between heterogeneous task models; and finally, executing a detection task by using the trained super-resolution rotating target detector. According to the invention, the detection precision and the detection speed of the rotating small target are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent interpretation of remote sensing images, intelligent perception of unmanned aerial vehicles and target detection, and in particular to a rotating small target detection method and system based on knowledge distillation of heterogeneous task models. Background Art

[0002] As two subtasks of target detection, rotated target detection and small target detection have received extensive attention and application in remote sensing images, scene text and industrial detection in recent years. Different from traditional target detection methods, rotated target detection uses a rotated bounding box (RBB) instead of a conventional horizontal bounding box (HBB) to achieve accurate representation of the target. In addition, due to the long imaging distance, there are a large number of small targets in remote sensing images and drone aerial images. Traditional target detection algorithms pay more attention to medium-sized and large-sized targets, and have poor detection performance for small targets. Rotated small target detection is closer to the real application scenarios of remote sensing images and drone aerial images, but this detection task also combines the difficulties of rotated target detection and small target detection, making it more challenging.

[0003] At present, there are few studies specifically targeting the detection of rotating small targets. In the field of small target detection in remote sensing images, Rabbi et al. proposed using the Laplacian operator to extract image edge features to enhance the super-resolution network SR to reconstruct high-resolution images. HR proposed a small target detection method MFL based on EESRGAN, which improves the performance of the detection network OD for small target detection; Bashir et al. proposed a super-resolution network SR framework that introduced a cyclic generative adversarial network GAN and residual feature aggregation, and used YOLO as the detection network OD. The performance of the detection network OD for small targets is improved; Bashir et al. proposed a small target detection method SRCGAN-RFA that introduced a cyclic generative adversarial network GAN and residual feature aggregation, and used YOLO as the detection network OD; Yang et al. proposed a small target detection method MFL based on EESRGAN by adding a feedback path so that the results of the region generation network RPN can be input into the discriminator of SRGAN to form a closed-loop structure to enhance SR's attention to the target area; In the above work, the problem of small target detection in OD when the low-resolution image LR is directly used as input is solved by introducing pre-SR, but the use of pre-SR will increase the additional computational burden and seriously affect the reasoning speed of OD, making it difficult for such methods to be implemented in actual industrial production environments. If SR can be removed in the reasoning stage, the reasoning performance can be guaranteed not to be greatly affected. For this reason, Zhang et al. proposed a SuperYOLO small target detection method that uses EDSR as an auxiliary network and learns to restore HR images through the fusion features of the YOLOv5 backbone network. However, this method lacks the necessary information exchange between the two heterogeneous task models of OD and SR, and is still independently trained and optimized. The efficiency and accuracy of rotated small target detection need to be improved. Summary of the invention

[0004] The purpose of the present invention is to provide a rotating small target detection method and system based on knowledge distillation of heterogeneous task models with high detection accuracy, strong performance and high efficiency.

[0005] The technical solution to achieve the purpose of the present invention is: a rotating small target detection method based on knowledge distillation of heterogeneous task models, comprising the following steps:

[0006] Step 1, determining a reference rotating target detector;

[0007] Step 2: Use a lightweight feature map embedding super-resolution module ESRM and embed it between the backbone network and the neck network of the reference rotation object detector to obtain a super-resolution rotation object detector;

[0008] Step 3: Use a lightweight network structure that is consistent with the first three layers of the super-resolution rotation object detector backbone network stemnetwork to form a feature encoding module FEM; then divide the classic SR model into a feature decoding module FDM and a feature upsampling module FUM, and combine FEM, FDM and FUM together as a super-resolution network branch PSRB parallel to the super-resolution rotation object detector, which is used to reconstruct the low-resolution image of the input reference rotation object detector into a high-resolution image;

[0009] Step 4: Use the feature affine layer FAL to assist the information interaction between the super-resolution rotation object detector and the super-resolution network branch, adopt the feature alignment loss function to minimize the similarity difference between the output features of the super-resolution rotation object detector and the super-resolution network branch, and use an end-to-end joint training method to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch to achieve knowledge distillation between heterogeneous task models;

[0010] Step 5: In the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task.

[0011] A rotating small target detection system based on heterogeneous task model knowledge distillation, the system is used to implement the rotating small target detection method based on heterogeneous task model knowledge distillation, the system includes the first module to the fifth module, wherein:

[0012] The first module determines a reference rotating target detector;

[0013] The second module uses a lightweight feature map embedding super-resolution module ESRM and embeds it between the backbone network and the neck network of the baseline rotation object detector to obtain a super-resolution rotation object detector;

[0014] The third module uses a lightweight network structure that is consistent with the first three layers of the super-resolution rotation object detector backbone network stemnetwork to form a feature encoding module FEM; then the classic SR model is divided into a feature decoding module FDM and a feature upsampling module FUM, and the combination of FEM, FDM and FUM is used as a super-resolution network branch PSRB parallel to the super-resolution rotation object detector, which is used to reconstruct the low-resolution image of the input reference rotation object detector into a high-resolution image;

[0015] The fourth module uses the feature affine layer FAL to assist the information interaction between the super-resolution rotation object detector and the super-resolution network branch, adopts the feature alignment loss function to minimize the similarity difference between the output features of the super-resolution rotation object detector and the super-resolution network branch, and adopts an end-to-end joint training method to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch to achieve knowledge distillation between heterogeneous task models;

[0016] In the fifth module, during the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task.

[0017] Compared with the prior art, the present invention has the following significant advantages: (1) it can directly perform target detection on low-resolution images. Unlike the super-resolution network in PSRB, which does not participate in the reasoning of the detection model, it significantly improves the reasoning speed of the detection model and improves the target detection efficiency without losing almost any detection accuracy; (2) it adopts a super-resolution rotating target detector and a super-resolution network branch to improve the target detection accuracy; (3) it can detect small rotating targets and achieve a practical small rotating target detection performance level, which is more in line with the needs of actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the rotating small target detection method based on knowledge distillation of heterogeneous task models in the present invention.

[0019] Figure 2 Schematic diagram of the structure of the overall deep neural network model in the model training stage in an embodiment of the present invention.

[0020] Figure 3 It is a visualized thermal comparison diagram of the neck network P2 feature map of the baseline rotation target detector and the super-resolution rotation target detector in an embodiment of the present invention, wherein (a) is a visualized thermal map of the neck network P2 feature map of the baseline rotation target detector, and (b) is a visualized thermal map of the neck network P2 feature map of the super-resolution rotation target detector.

[0021] Figure 4It is a schematic diagram of some detection results of RTMDet-ER in an embodiment of the present invention on a low-resolution DOTA dataset. DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0023] Combination Figure 1 The present invention provides a rotating small target detection method based on knowledge distillation of heterogeneous task models, comprising the following steps:

[0024] Step 1, determining a reference rotating target detector;

[0025] Step 2: Use a lightweight feature map embedding super-resolution module ESRM and embed it between the backbone network and the neck network of the reference rotation object detector to obtain a super-resolution rotation object detector;

[0026] Step 3: Use a lightweight network structure that is consistent with the first three layers of the super-resolution rotation object detector backbone network stemnetwork to form a feature encoding module FEM; then divide the classic SR model into a feature decoding module FDM and a feature upsampling module FUM, and combine FEM, FDM and FUM together as a super-resolution network branch PSRB parallel to the super-resolution rotation object detector, which is used to reconstruct the low-resolution image of the input reference rotation object detector into a high-resolution image;

[0027] Step 4: Use the feature affine layer FAL to assist the information interaction between the super-resolution rotation object detector and the super-resolution network branch, adopt the feature alignment loss function to minimize the similarity difference between the output features of the super-resolution rotation object detector and the super-resolution network branch, and use an end-to-end joint training method to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch to achieve knowledge distillation between heterogeneous task models;

[0028] Step 5: In the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task.

[0029] As a specific example, the determination of the reference rotating target detector in step 1 includes the following two methods:

[0030] The first approach uses a deep learning-based rotating object detection algorithm as a baseline rotating object detector;

[0031] The second method is to improve the general object detector by adding an angle regression branch or an output channel of the regression convolution, transforming the general object detector into a rotation object detector and using it as a baseline rotation object detector.

[0032] As a specific example, the conversion of the general object detector into the rotation object detector is specifically as follows:

[0033] The general object detector is added with an angle regression branch consisting of a 3×3 Conv with a single-channel output, or the output channel of the regression convolution is changed from 4 to 5, so as to transform the general object detector into a rotation object detector and realize the prediction of the object rotation angle.

[0034] As a specific example, in step 2, the network structure of the lightweight feature map embedding super-resolution module ESRM is as follows:

[0035] The first layer consists of a 3×3Conv followed by a SiLU activation function, the second layer consists of a 5×5 depthwise separable large kernel convolution followed by a SiLU activation function, the third layer consists of a 1×1Conv followed by a SiLU activation function, a separate 1×1Conv is used as the fourth layer, and the upsampling convolution layer consists of a 3×3Conv followed by a PixelShuffle sub-pixel convolution. The first, second, third, and fourth layers are connected to the upsampling convolution layer in sequence to form a lightweight feature map embedding super-resolution module ESRM.

[0036] As a specific example, in step 3, a lightweight network structure consistent with the first three layers of the super-resolution rotation object detector backbone network is used to form a feature encoding module FEM, and then the classic SR model is divided into a feature decoding module FDM and a feature upsampling module FUM, as follows:

[0037] The feature encoding module FEM consists of a 3×3 Conv with a step size of 2 followed by a BN layer and a ReLU activation function to form the first layer, a 3×3 Conv followed by a BN layer and a ReLU activation function to form the second and third layers, and then the first, second, and third layers are connected in sequence; the feature decoding module FDM is composed of the network structure before the sub-pixel convolution that performs feature upsampling in the EDSR network, using only four layers of stacked residual blocks; the feature upsampling module FUM consists of a 3 × 3Conv followed by a PixelShuffle sub-pixel convolution followed by a 3 × 3Conv composition.

[0038] As a specific example, the feature affine layer FAL in step 4 is composed of a 1×1 Conv.

[0039] As a specific example, the mathematical expression of the feature alignment loss function in step 4 is:

[0040]

[0041]

[0042] in i=1,2,...,C is the input feature map The spatial dimension compression feature map The corresponding row vector of ;|||| 2 Indicates the second norm of the vector, that is, the Euclidean distance; G i,j is the element in the normalized Gram matrix; F 1 and F 2 represents the input feature map whose similarity needs to be calculated; ⊙ represents the element-by-element multiplication of the matrix;

[0043] W roi The regional interest weight, which serves as a weighting parameter, is calculated as follows: first, the channel dimension of the output feature map of the classification branch is reduced to 1 dimension by taking the maximum value; then a threshold hyperparameter is set, and the values ​​in the output feature map of the classification branch that are greater than the threshold are set to 1, and the rest are set to 0, to obtain an attention mask; finally, the size of the attention mask is interpolated and scaled to the same size as F. 1 and F 2 The same size, and multiply each element of the attention mask by 4 plus 1 to get the regional interest weight W roi .

[0044] As a specific example, in step 4, an end-to-end joint training method is used to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch, as follows:

[0045] First calculate the total loss function of the model training The formula is:

[0046]

[0047] in is the training loss function of the super-resolution rotation object detector, is the training loss function of the super-resolution network branch PSRB, λ is the distillation balance hyperparameter, is the output feature O of the feature affine layer FAL SR The feature alignment loss between the output feature C2′ of the feature map embedding super-resolution module ESRM, is the output feature E of the feature encoding module FEM SR The feature alignment loss between the output feature C0 of the first three layers of the stem network of the super-resolution rotation object detector backbone network;

[0048] Then, the gradient descent algorithm is used to jointly optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch according to the total loss function.

[0049] As a specific example, in step 5, during the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task, as follows:

[0050] Step 5.1, delete the entire super-resolution network branch PSRB used for training;

[0051] Step 5.2: Take low-resolution image I LR As the input of the super-resolution rotation object detector, perform a model forward inference of the super-resolution rotation object detector to obtain the detection result;

[0052] Step 5.3: Use the rotation non-maximum suppression algorithm NMS for post-processing to obtain the final inference result.

[0053] The present invention also provides a rotating small target detection system based on heterogeneous task model knowledge distillation, which is used to implement the rotating small target detection method based on heterogeneous task model knowledge distillation. The system includes a first module to a fifth module, wherein:

[0054] The first module determines a reference rotating target detector;

[0055] The second module uses a lightweight feature map embedding super-resolution module ESRM and embeds it between the backbone network and the neck network of the baseline rotation object detector to obtain a super-resolution rotation object detector;

[0056] The third module uses a lightweight network structure that is consistent with the first three layers of the super-resolution rotation object detector backbone network stemnetwork to form a feature encoding module FEM; then the classic SR model is divided into a feature decoding module FDM and a feature upsampling module FUM, and the combination of FEM, FDM and FUM is used as a super-resolution network branch PSRB parallel to the super-resolution rotation object detector, which is used to reconstruct the low-resolution image of the input reference rotation object detector into a high-resolution image;

[0057] The fourth module uses the feature affine layer FAL to assist the information interaction between the super-resolution rotation object detector and the super-resolution network branch, adopts the feature alignment loss function to minimize the similarity difference between the output features of the super-resolution rotation object detector and the super-resolution network branch, and adopts an end-to-end joint training method to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch to achieve knowledge distillation between heterogeneous task models;

[0058] In the fifth module, during the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task.

[0059] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Example

[0061] This embodiment uses the lightweight RTMDet-R rotating target detection algorithm as the benchmark rotating target detector, and elaborates on the network model architecture of the parallel super-resolution network branch and the feature map embedded in the super-resolution module, the mathematical expression of the feature alignment loss, and the specific end-to-end joint training process, that is, the knowledge distillation process between heterogeneous task models. The RTMDet-R rotating target detection model enhanced by the method of the present invention is referred to as the RTMDet-ER rotating small target detection model. The overall deep neural network model architecture is shown in the attached figure. Figure 2 shown.

[0062] Following the conventional training process of the deep learning target detection model, the data set is processed first. This embodiment is not targeted at a specific data set, so it can be applied to public aerial image rotation target detection data sets represented by DOTA, UCAS-AOD and HRSC, and any data set that uses RBB as an annotation. In particular, when using a self-annotated private data set for a specific task field, the data set is usually divided into a training set, a validation set and a test set in a ratio of 5:2:3.

[0063] First, take the open source rotating object detection library MMRotate as an example, for the training set and validation set divided from the dataset, read the images in the specified folder through the Python language interface. HR The GT annotation is stored in memory and processed into common Numpy data types. In particular, the GT RBB annotation adopts the common long edge 90 degree rotation representation scheme;

[0064] Secondly, implement data enhancement strategies to avoid overfitting problems in deep learning model training. Usually, random flip enhancement and random rotation enhancement are used by default. Stronger data enhancement strategies are also applicable.

[0065] Then, the original image is downsampled by bicubic to obtain an I with a resolution attenuated by 2 times LR At the same time, the corresponding GT annotation is reduced by 2 times to obtain the corresponding GT of the low-resolution image;

[0066] Finally, I HR ,I LRThe GT annotations corresponding to the low-resolution images are converted from Numpy data types to Pytorch-specific tensor data types, and processed through the dataloader interface to output only one training tensor of a specified batch size each time. The above steps are all implemented in MMRotate, and the interface is packaged to be more advanced and easy to use, but the present invention is not limited to the specific implementation in MMRotate, and can also be easily implemented using other target detection libraries or other types of deep learning frameworks.

[0067] After the data set is processed, the rotating small target detection method based on heterogeneous task model knowledge distillation provided by the present invention is used to perform the rotating small target detection in this embodiment. Figure 1 , including the following steps:

[0068] Step 1: Specific implementation of the baseline rotation target detector. The present invention is not targeted at a specific rotation target detector, but can be combined with target detection models such as Faster R-CNN of the dual-stage anchor box paradigm, RetinaNet of the single-stage anchor box paradigm, FCOS of the single-stage anchor-free paradigm, and lightweight RTMDet. Taking RTMDet as an example, the specific implementation steps of the rotation target detector based on deep learning are briefly introduced.

[0069] Step 1.1, implementation of the backbone network. The backbone network used by the RTMDet-R model is CSPNetXt, which is a convolutional neural network. The main network structure is similar to the CSPDarknet network of the YOLOX model. The large-core convolution standard block constructed by the deep separable large-core convolution is used to replace the standard convolution block in the original CSPDarknet. Specifically, the large-core convolution standard block of CSPNetXt consists of a layer of 3×3Conv plus a layer of BN layer plus SiLU activation function to form the first layer of classic convolution block, and then a 5×5Depthwise Conv plus a layer of 1×1PointwiseConv plus a layer of BN layer plus SiLU activation function to form the second layer of deep separable large-core convolution layer. Finally, the first layer of classic convolution block and the second layer of deep separable large-core convolution layer are connected in series, and residual connection is added.

[0070] Step 1.2: Implementation of the neck network. The neck network used by the RTMDet-R model is the CSPNetXtPAN network. Specifically, the classic CSPPAN neck network is improved by combining a deep separable large kernel convolutional layer.

[0071] Step 1.3, implementation of the detection head network. In order to realize the prediction of the rotation angle, the present invention needs to add an additional angle prediction branch to the detection head network. Specifically, it is composed of a 3×3Conv with an output channel number of 1, which is used to output the prediction of the target rotation angle. Similarly, for the RetinaNet and FCOS models, it is also possible to complete the transition from the general target detection model to the rotation target detection model by adding only one 3×3Conv with a single-channel output to form an angle prediction branch, and the regression branches and classification branches of the other backbone networks, neck networks and detection heads remain unchanged.

[0072] Step 2: Specific network structure and implementation of super-resolution rotation object detector. The feature map embedding super-resolution module ESRM is embedded between the backbone network and the neck network of the baseline rotation object detector. In order to ensure that the final OD model does not introduce too much additional computational burden, a lightweight network design is required. The specific implementation steps are as follows:

[0073] Step 2.1, refer to the residual block structure in the EDSR network and the core large kernel convolution structure in CSPNetXt to construct the residual network module in ESRM. Specifically, a 3×3Conv connected to a SiLU activation function constitutes the first layer, a 5×5 depth-separable large kernel convolution connected to a SiLU activation function constitutes the second layer, a 1×1Conv connected to a SiLU activation function constitutes the third layer, and a single 1×1Conv is used as the fourth layer. Then, the first, second, third, and fourth layers are connected in sequence, and a residual connection is drawn from the input to add the output of the fourth layer to the input features of the module as the final module output.

[0074] Step 2.2: Use the sub-pixel convolution layer in the EDSR network as the upsampling convolution network module in ESRM. Specifically, it is implemented by a 3×3 Conv followed by a PixelShuffle sub-pixel convolution.

[0075] Step 2.3, sequentially connect the residual network module and the upsampling convolutional network module to form ESRM. In addition, the output of the third layer in the residual network module is taken as the residual output feature map of ESRM, which is used to align with the output feature map of PSRB.

[0076] Step 3: Specific network structure and implementation of parallel super-resolution network branches. This embodiment uses the classic EDSR super-resolution network structure as the basis for the parallel super-resolution network branches. The detailed network architecture can be found in the attached Figure 2 The specific implementation steps are as follows:

[0077] Step 3.1: Use the structure of the backbone network stem network to construct FEM. Specifically, the first layer is composed of a 3×3 Conv with a step size of 2, a Batch Normal (BN) layer and a ReLU activation function, the second and third layers are composed of a 3×3 Conv with a BN layer and a ReLU activation function, and then the first, second and third layers are connected in sequence to form FEM.

[0078] Step 3.2: Based on the EDSR network, the network structure before the sub-pixel convolution for feature upsampling is used as FDM, and the sub-pixel convolution and the last convolution layer constitute FUM. In particular, the number of channels of the first convolution layer of FDM is modified to the number of output channels of FEM, and only four layers of stacked residual blocks are used in FDM.

[0079] Step 3.3: FEM, FDM and FUM are connected in sequence to form the parallel super-resolution network branch of the present invention. In addition, the output feature map of FDM is extracted to be aligned with the feature map of the ROD model embedded in ESRM later.

[0080] Step 4: Implementation of the overall network architecture for knowledge distillation of heterogeneous task models. Following the previous steps, the RTMDet-R detector, PSRB and ESRM modules have been constructed. Figure 2 The overall network structure shown connects the various modules together to obtain the overall network architecture for knowledge distillation of heterogeneous task models. The specific implementation steps are as follows:

[0081] Step 4.1. Add three independent ESRMs after the C2, C3, and C4 output levels of the backbone network, and record the output feature maps of these three ESRMs as C2′, C3′, and C4′.

[0082] Step 4.2: Use a 1×1 Conv as the feature affine layer FAL to process the output features of PSRB after 4 times downsampling, and calculate the feature alignment loss between the output features of FAL and C2′.

[0083] Step 4.3: The feature alignment loss used is to measure the similarity difference between the output features of the two heterogeneous models. Therefore, the normalized Gram matrix is ​​first used to calculate the similarity between different channels of each feature map. The specific calculation formula is as follows:

[0084]

[0085] in i=1,2,...,C is the input feature map The spatial dimension compression feature map The corresponding row vector of , and |||| 2Indicates the second norm of the vector (i.e., Euclidean distance), G i,j is the element in the normalized Gram matrix;

[0086] Based on the normalized Gram matrix of each feature map, the weighted Euclidean distance is used to measure the structural relationship differences between different feature maps, that is, the similarity differences, to obtain the feature alignment loss. The specific calculation formula is as follows:

[0087]

[0088] where F 1 and F 2 represents the input feature map whose similarity needs to be calculated, ⊙ represents the element-by-element multiplication of the matrix, W roi is the regional interest weight which serves as a weighting parameter.

[0089] Step 4.4: Regional interest weight W roi In order to achieve PSRB's attention to the image area where the target may exist, so that the low-level structural features of the corresponding area in the detector are more obvious, the output feature map based on the classification branch of the detector is designed to calculate the regional interest weight W roi Specifically, first, the channel dimension of the output feature map of the classification branch is reduced to 1 dimension by taking the maximum value. Then, a threshold hyperparameter is set, and the values ​​in the output feature map of the classification branch that are greater than the threshold are set to 1 and the rest are set to 0 to obtain an attention mask. Then, the size of the mask is interpolated and scaled to the same size as F. 1 and F 2 The same size, and multiply each element of the mask by 4 plus 1 to get W roi .

[0090] Step 4.5: Use I LR As the input image of RTMDet-R and PSRB. First, perform forward calculations on RTMDet-R and PSRB respectively to obtain the output results of the regression branch and classification branch of each level of the detector and the output image of the super-resolution network; then, calculate the regional interest weight W roi , calculate the output feature O of FAL SR Feature alignment loss with C2′ At the same time, it is also necessary to calculate the output characteristic map E of FEM SR The feature alignment loss between the output feature map C0 of the detector stem network Finally, the training loss function of the detector and super-resolution network is calculated and The gradient descent algorithm is used to jointly optimize the parameters of the RTMDet-R and PSRB models according to the total loss function. Specifically, the total loss function calculation formula is as follows:

[0091]

[0092] where λ is the distillation equilibrium hyperparameter.

[0093] Step 5: Model reasoning process of rotating small target detector. The specific implementation steps are as follows:

[0094] Step 5.1: After training these two heterogeneous models according to the specified training epoch, the rotating small target detector RTMDet-R also synchronously completes the distillation of super-resolution knowledge. At this time, the ESRM embedded in the detector has better feature map super-resolution capability and can pay more attention to the area where small targets may exist. For the convenience of distinction, the detector model at this time is called RTMDet-ER. The visual heat map comparison of the P2 output features of RTMDet-ER and RTMDet-R is shown in the figure. Figure 3 As shown, from Figure 3 From (a) and (b), we can see that RTMDet-ER pays more attention to small objects in the image.

[0095] Step 5.2: Delete the entire PSRB network in the overall model architecture during the inference phase of RTMDet-ER and still use I LR As the input of the model, a forward inference is then performed to obtain the output of the detector, and the final inference result is obtained by post-processing through rotational NMS. Figure 4 The figure shows some detection results of RTMDet-ER in the embodiment of the present invention on the low-resolution DOTA dataset. It can be seen from the figure that compared with the rotating target detection algorithm using high-resolution images as input, the present invention can not only maintain a comparable detection accuracy, but also significantly improve the detection speed.

[0096] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A rotating small target detection method based on knowledge distillation of heterogeneous task models, characterized in that: The following steps are involved: Step 1, determining a reference rotating target detector; Step 2: Use a lightweight feature map embedding super-resolution module ESRM and embed it between the backbone network and the neck network of the reference rotation object detector to obtain a super-resolution rotation object detector; Step 3: Use a lightweight network structure that is consistent with the first three layers of the super-resolution rotation object detector backbone network. A feature encoding module FEM is formed; then the classic SR model is divided into a feature decoding module FDM and a feature upsampling module FUM, and the combination of FEM, FDM and FUM is used as a super-resolution network branch PSRB parallel to the super-resolution rotation object detector, which is used to reconstruct the low-resolution image of the input reference rotation object detector into a high-resolution image; Step 4: Use the feature affine layer FAL to assist the information interaction between the super-resolution rotation object detector and the super-resolution network branch, adopt the feature alignment loss function to minimize the similarity difference between the output features of the super-resolution rotation object detector and the super-resolution network branch, and use an end-to-end joint training method to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch to achieve knowledge distillation between heterogeneous task models; Step 5: In the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task.

2. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 1 is characterized in that: The determination of the reference rotating target detector described in step 1 includes the following two methods: The first approach uses a deep learning-based rotating object detection algorithm as a baseline rotating object detector; The second method is to improve the general object detector by adding an angle regression branch or an output channel of the regression convolution, transforming the general object detector into a rotation object detector and using it as a baseline rotation object detector.

3. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 2 is characterized in that: The conversion of the general object detector into a rotation object detector is specifically as follows: The general object detector is added with an angle regression branch consisting of a 3×3 Conv with a single-channel output, or the output channel of the regression convolution is changed from 4 to 5, so as to transform the general object detector into a rotation object detector and realize the prediction of the object rotation angle.

4. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 1 is characterized in that: In step 2, the network structure of the lightweight feature map embedding super-resolution module ESRM is as follows: The first layer consists of a 3×3Conv followed by a SiLU activation function, the second layer consists of a 5×5 depthwise separable large kernel convolution followed by a SiLU activation function, the third layer consists of a 1×1Conv followed by a SiLU activation function, a separate 1×1Conv is used as the fourth layer, and the upsampling convolution layer consists of a 3×3Conv followed by a PixelShuffle sub-pixel convolution. The first, second, third, and fourth layers are connected to the upsampling convolution layer in sequence to form a lightweight feature map embedding super-resolution module ESRM.

5. The rotating small target detection method based on knowledge distillation of heterogeneous task models according to claim 1 is characterized in that: In step 3, a lightweight network structure consistent with the first three layers of the super-resolution rotation object detector backbone network is used. The feature encoding module FEM is composed, and then the classic SR model is divided into the feature decoding module FDM and the feature upsampling module FUM, as follows: The feature encoding module FEM consists of a 3×3 Conv with a step size of 2, a BN layer and a ReLU activation function to form the first layer, a 3×3 Conv with a BN layer and a ReLU activation function to form the second and third layers, and then the first, second and third layers are connected in sequence; the feature decoding module FDM is composed of the network structure before the sub-pixel convolution that performs feature upsampling in the EDSR network, and only four layers of stacked residual blocks are used; the feature upsampling module FUM is composed of a 3×3 Conv, a PixelShuffle sub-pixel convolution and then a 3×3 Conv.

6. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 1 is characterized in that: The feature affine layer FAL in step 4 consists of a 1×1 Conv.

7. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 1 is characterized in that: The mathematical expression of the feature alignment loss function in step 4 is: in i=1,2,...,C is the input feature map The spatial dimension compression feature map The corresponding row vector of G; ||||2 means to find the vector's binorm, i.e., the Euclidean distance; i,j is the element in the normalized Gram matrix; F1 and F2 represent the input feature graphs for which similarity needs to be calculated; ⊙ represents the element-by-element multiplication of the matrix; W roi The regional interest weight, which serves as a weighting parameter, is calculated as follows: first, the channel dimension of the output feature map of the classification branch is reduced to 1 dimension by taking the maximum value; then a threshold hyperparameter is set, and the values ​​in the output feature map of the classification branch that are greater than the threshold are set to 1, and the rest are set to 0 to obtain an attention mask; finally, the size of the attention mask is interpolated and scaled to the same size as F1 and F2, and each element of the attention mask is multiplied by 4 and added by 1 to obtain the regional interest weight W. roi .

8. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 1 is characterized in that: In step 4, an end-to-end joint training approach is used to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch, as follows: First calculate the total loss function of the model training The formula is: in is the training loss function of the super-resolution rotation object detector, is the training loss function of the super-resolution network branch PSRB, λ is the distillation balance hyperparameter, is the output feature O of the feature affine layer FAL SR The feature alignment loss between the output feature C2′ of the feature map embedding super-resolution module ESRM, is the output feature E of the feature encoding module FEM SR The feature alignment loss between the output feature C0 of the first three layers of the stem network of the super-resolution rotation object detector backbone network; Then, the gradient descent algorithm is used to jointly optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch according to the total loss function.

9. The rotating small target detection method based on heterogeneous task model knowledge distillation according to claim 1 is characterized in that: In step 5, during the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task as follows: Step 5.1, delete the entire super-resolution network branch PSRB used for training; Step 5.2: Take low-resolution image I LR As the input of the super-resolution rotation object detector, perform a model forward inference of the super-resolution rotation object detector to obtain the detection result; Step 5.3: Use the rotation non-maximum suppression algorithm NMS for post-processing to obtain the final inference result.

10. A rotating small target detection system based on knowledge distillation of heterogeneous task models, characterized in that: The system is used to implement the rotating small target detection method based on heterogeneous task model knowledge distillation according to any one of claims 1 to 9, and the system includes the first module to the fifth module, wherein: The first module determines a reference rotating target detector; The second module uses a lightweight feature map embedding super-resolution module ESRM and embeds it between the backbone network and the neck network of the baseline rotation object detector to obtain a super-resolution rotation object detector; The third module uses a lightweight network structure that is consistent with the first three layers of the super-resolution rotation object detector backbone network stemnetwork to form a feature encoding module FEM; then the classic SR model is divided into a feature decoding module FDM and a feature upsampling module FUM, and the combination of FEM, FDM and FUM is used as a super-resolution network branch PSRB parallel to the super-resolution rotation object detector, which is used to reconstruct the low-resolution image of the input reference rotation object detector into a high-resolution image; The fourth module uses the feature affine layer FAL to assist the information interaction between the super-resolution rotation object detector and the super-resolution network branch, adopts the feature alignment loss function to minimize the similarity difference between the output features of the super-resolution rotation object detector and the super-resolution network branch, and adopts an end-to-end joint training method to simultaneously optimize the parameters of the super-resolution rotation object detector and the super-resolution network branch to achieve knowledge distillation between heterogeneous task models; In the fifth module, during the inference phase of the model, the trained super-resolution rotation object detector is used to perform the detection task.

Images