An optical remote sensing target detection light-weight method based on a knowledge distillation network

Abstract

The application provides an optical remote sensing target detection lightening method based on a knowledge distillation network, adopts a model compression method of knowledge distillation, reduces the network size, generates a lightening child network for target detection conforming to on-board load deployment, namely, on the basis of a general deep learning framework, a reasonable lightening network structure is designed according to a limited on-board GPU of a computing resource, so that the model keeps a high detection accuracy, and fast, accurate and flexible remote sensing image target detection is realized, on-orbit real-time calculation of key frame remote sensing images is realized, and the method can be applied to an on-board GPU platform with limited computing resources.

Description

Technical Field

[0001] This invention belongs to the field of intelligent image processing, and in particular relates to a lightweight method for optical remote sensing target detection based on knowledge distillation networks. Background Technology

[0002] As information processing technology advances towards intelligence, the importance of deep convolutional neural networks in the field of target detection in remote sensing images is becoming increasingly prominent. Due to the massive scale of optical remote sensing images acquired by satellite payloads, real-time processing of these images using satellite downlink data is difficult. Therefore, on-orbit processing technology has become an important development direction for target detection in optical remote sensing images. Deep convolutional neural network models often suffer from problems such as large number of model parameters and network node redundancy, resulting in significant memory overhead and thus limiting their deployment on computationally limited onboard GPUs.

[0003] Furthermore, traditional satellite remote sensing image target detection technology uses data downlink to perform target detection at the ground level, and the results often lack real-time processing capabilities. Therefore, traditional space optical remote sensing technology is no longer adequate for the precise, flexible, and rapid response requirements of disaster relief, intelligence reconnaissance, and target surveillance. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a lightweight optical remote sensing target detection method based on knowledge distillation networks. This method enables real-time, rapid, accurate, and flexible detection of keyframe images acquired by satellite optical payloads and can be deployed on hardware platforms with limited computing resources, such as satellite on-orbit processing.

[0005] A lightweight method for optical remote sensing target detection based on knowledge distillation networks is disclosed. This method employs a knowledge distillation-based target detection neural network model to detect targets in real-time satellite-captured optical remote sensing images. The method for obtaining the target detection neural network model includes the following steps:

[0006] S1: Obtain the target region and background region of each optical remote sensing image, and use the optical remote sensing image and target region as the input and output of the CSPNet v5 network respectively to train the CSPNet v5 network and obtain the teacher network.

[0007] S2: Using the teacher network as input to the distillation network, the distillation network is trained under the guidance of the teacher network's network parameters to obtain the student network;

[0008] S3: Determine whether the distillation loss between the student network and the teacher network is less than a set threshold. If yes, the current student network is the target detection neural network model. If no, the distillation loss is used for backpropagation of the distillation network, and the rotation IoU loss is also used for backpropagation of the CSPNet v5 network to adjust the distillation network and the CSPNet v5 network until the distillation loss is less than the set threshold.

[0009] Furthermore, the rotational IoU loss The definition is as follows:

[0010]

[0011] Where N represents the corner coordinates of the target's true detection box, generated using the 180° five-parameter representation method. The corner coordinates are generated using the 180° five-parameter representation method to characterize the target prediction box after distillation. The corner coordinates include the box width, box height, x-coordinate of the target center point, y-coordinate of the target center point, and the rotation angle of the box.

[0012] Furthermore, the distillation loss L tea-stu (X,Y) is defined as follows:

[0013]

[0014] Where W, H, and S represent the length, width, and number of channels of the feature map generated during the training of the distillation network, respectively, and F feature f represents the distillation loss relationship between the teacher network and the student network. stu (x,y,z) represents the feature map obtained by extracting features from the student network, f tea (x,y,z) represents the feature map obtained by the teacher network after feature extraction, ω is the set balance parameter, and M is the feature map f. tea The sum of the number of non-zero pixels in (x,y,z).

[0015] Furthermore, the CSPNet v5 network uses the YOLOX framework, with depth-separable convolutional layers and skip connections between network nodes.

[0016] Furthermore, the convolution calculation process of the CSPNet v5 network and the distillation network is divided into depthwise convolution and pointwise convolution operations. The number of output channels of the Depthwise layer is set to 128, and the kernel size is 3*3*1. The number of output channels of the Pointwise layer is set to be consistent with that of the Depthwise layer, and the kernel size is 1*1*1.

[0017] Furthermore, the optical remote sensing images described in step S1 are extracted from the DIOR dataset.

[0018] Furthermore, after filtering and denoising each optical remote sensing image and uniformly cropping it into pixel blocks of a set size, the pixel blocks are divided into target pixel blocks and background pixel blocks. Finally, the target pixel blocks and background pixel blocks are used as inputs to the CSPNet v5 network.

[0019] Beneficial effects:

[0020] 1. This invention provides a lightweight optical remote sensing target detection method based on knowledge distillation networks. It employs a knowledge distillation model compression method to reduce the network size and generate a lightweight student network for target detection that is suitable for on-board payload deployment. In other words, this invention, based on a general deep learning framework, designs a reasonable lightweight network structure for onboard GPUs with limited computing resources. This allows the model to maintain high detection accuracy while achieving fast, accurate, and flexible remote sensing image target detection. It enables real-time on-orbit computation of keyframe remote sensing images and can be applied to onboard GPU platforms with limited computing resources.

[0021] 2. This invention provides a lightweight optical remote sensing target detection method based on knowledge distillation network. It improves the ordinary convolutional layer by using a depth-separable convolution method, and divides the convolution calculation process into depthwise convolution and pointwise convolution operations, which can reduce the space complexity of model calculation.

[0022] 3. This invention provides a lightweight optical remote sensing target detection method based on knowledge distillation network. The parameters of the student network are updated through backpropagation, and the weights of the teacher network nodes are updated in the downstream task through forward propagation, thereby achieving the effect of fitting a large model with a small model and reducing the number of network parameters.

[0023] 4. This invention provides a lightweight optical remote sensing target detection method based on knowledge distillation network. The network nodes adopt a skip connection method, that is, a residual network is introduced between the convolutional layers of the network framework, which alleviates the gradient vanishing and gradient explosion problems that occur in deep neural networks, and makes the context information propagation more smoothly in the forward and backward directions. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the model structure;

[0025] Figure 2 This demonstrates the application of depthwise separable convolution in remote sensing image target detection.

[0026] Figure 3 Platform deployment test diagram for detecting aircraft targets using a lightweight model. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0028] Common methods for lightweighting neural networks include network pruning, model quantization, low-rank decomposition, and knowledge distillation. Knowledge distillation, in particular, can fit complex models by compressing them, mimicking the functionality of complex models. In recent years, researchers have verified the effectiveness of knowledge distillation in compressing model size and improving detection speed. Against this backdrop, to achieve real-time processing of remote sensing images on spaceborne platforms, a lightweight target detection model for optical remote sensing images based on knowledge distillation is designed. This is of great significance for deployment and application on embedded platforms with limited computing power, such as real-time on-orbit detection of satellites.

[0029] A lightweight method for optical remote sensing target detection based on knowledge distillation networks is proposed. This method employs a knowledge distillation-based target detection neural network model to perform target detection on real-time optical remote sensing images captured by satellites. For example... Figure 1 As shown, the method for obtaining the target detection neural network model includes the following steps:

[0030] S1: Obtain the target region and background region of each optical remote sensing image, and use the optical remote sensing image and target region as the input and output of the CSPNet v5 network respectively to train the CSPNet v5 network and obtain the teacher network.

[0031] Furthermore, this invention filters and denoises each optical remote sensing image and uniformly crops it into 2880*2160 pixel blocks. These pixel blocks are then divided into target pixel blocks and background pixel blocks, which are finally used as input to the CSPNet v5 network. Additionally, after obtaining the satellite remote sensing image data stream from the DIOR dataset, this invention extracts keyframes from the data stream, filters and denoises the keyframes, and uniformly crops them into 2880*2160 pixel blocks before inputting them into the subsequent convolutional neural network.

[0032] It should be noted that, based on previous comparative experiments, the YOLOX framework outperforms other deep learning frameworks in detecting small targets in space. It demonstrates superior performance on the basis of the CSPNet v5 backbone network. Therefore, this invention uses the CSPNet v5 network to replace the Darknet-53 backbone network of the YOLOX framework.

[0033] Furthermore, to achieve efficient convolution calculation methods and reduce the space complexity of model computation, such as... Figure 2As shown, this invention improves ordinary convolutional layers by using depthwise separable convolution to perform convolution calculations between network layers, reducing the number of intermediate parameters in the network model and lowering the space complexity of the algorithm. This invention also divides the convolution calculation process into depthwise convolution and pointwise convolution operations. At the same time, the number of output channels of the Depthwise layer is set to 128, and the kernel size is 3*3*1. The number of output channels of the Pointwise layer is set to be consistent with that of the Depthwise layer in the same module, and the kernel size is 1*1*1.

[0034] S2: Using the teacher network as input to the distillation network, the distillation network is trained under the guidance of the teacher network's network parameters to obtain the student network.

[0035] It should be noted that the teacher network is already the output of CSPNetv5, and as a complete detection model, it can output well-detected remote sensing images. However, to achieve lightweight design, this invention directly feeds the teacher network into a distillation network. Without affecting detection accuracy, some low-weight node connections in the teacher network are distilled out, resulting in the student network. In other words, the input to the distillation network is the teacher network, and the output is the student network. Furthermore, the network parameters of the teacher network typically include the weights between network nodes and the convolution kernel function.

[0036] It should be noted that at this point, the student network has already distilled some low-weight node connections. For knowledge distillation of the network model, this invention divides the distillation process into three steps: 1) Extending from the top-level output features to intermediate-level features, allowing more knowledge to be transferred to the student network and outputting refined feature maps. 2) Designing a distillation loss function to constrain the training process of the student and teacher networks, assigning high weight values ​​to key target regions in the remote sensing image and low weight values ​​to background information regions. 3) Updating the parameters of the student network through backpropagation and updating the weights of the teacher network nodes in downstream tasks through forward propagation, thereby achieving the effect of fitting a large model with a small model, reducing the number of network parameters. The improved network model can be named CKD-YOLOX.

[0037] Furthermore, this invention employs a residual network structure to improve fully connected layers, forming a skip connection. Introducing residual network modules between convolutional layers of the basic network framework alleviates the gradient vanishing and gradient exploding problems that occur in deep neural networks, making the propagation of contextual information smoother in both forward and backward directions.

[0038] S3: Determine whether the distillation loss between the student network and the teacher network is less than a set threshold. If yes, the current student network is the target detection neural network model. If no, the distillation loss is used for backpropagation of the distillation network, and the rotation IoU loss is also used for backpropagation of the CSPNet v5 network to adjust the distillation network and the CSPNet v5 network until the distillation loss is less than the set threshold.

[0039] It should be noted that the loss function of this invention includes two constraints: a rotation IoU loss for performing regression tasks on YOLOX, used to constrain the initial model training process; and a distillation loss for information transfer between the teacher and student networks, used to constrain the training process of the distillation network. This invention defines the following loss function combining classification and regression tasks:

[0040]

[0041] Where M represents the network parameters of the model, This represents the rotational IoU loss constraint, where ω is the balance parameter, and L... tea-stu (X,Y) represents the distillation loss between the student network and the teacher network.

[0042] To address the rotation IoU loss in regression tasks, this invention primarily considers the diverse orientations of targets in optical remote sensing images acquired by satellite payloads, such as ships and vehicles arranged arbitrarily. Therefore, this invention employs a 180° five-parameter representation method to accurately locate the target position, achieving tilted bounding box position regression for rotating targets. The five parameters include the box width, box height, x-coordinate of the target center point, y-coordinate of the target center point, and the box rotation angle, i.e., the angle between the long side of the target box and the x-axis.

[0043] Furthermore, the rotational IoU loss constraint is defined as follows:

[0044]

[0045] in, N represents the corner coordinates of the target's true detection box, generated using the 180° five-parameter representation method. The corner coordinates are generated using the 180° five-parameter representation method to characterize the target prediction box after distillation. The corner coordinates include the box width, box height, x-coordinate of the target center point, y-coordinate of the target center point, and the rotation angle of the box.

[0046] The distillation loss between models is defined as follows:

[0047]

[0048] Where W, H, and S represent the length, width, and number of channels of the feature map generated during the training of the distillation network, respectively, and F feature This represents the distillation loss relationship between the teacher network and the student network, providing guidance for the knowledge distillation process. stu (x,y,z) represents the feature map obtained by extracting features from the student network, f tea (x,y,z) represents the feature map obtained by the teacher network after feature extraction, ω is the set balance parameter, and M is the feature map f. tea The sum of the number of non-zero pixels in (x,y,z).

[0049] The training process of the CSPNet v5 network and distillation network of this invention is described below.

[0050] The optical remote sensing images in the DIOR dataset are automatically batched and fed into the first convolutional layer of the CSPNet v5 network. The data is iterated under the constraint of rotation IoU loss to generate the first-level target feature map. The obtained first-level target feature map is fed into the second convolutional layer and iterated again to generate the second-level target feature map, until all convolutional layers of the CSPNet v5 network have completed the calculation.

[0051] The output of the teacher network is fed into the distillation network, and iteratively processed under the constraints of the distillation loss function. The main network nodes and feature information are retained. The larger the weight between nodes, the more important the node is. This results in a lightweight student network model structure that maintains a high degree of integrity of the target feature information.

[0052] The distillation loss of the student network is obtained. The final lightweight student network model is determined based on whether the distillation loss is less than a set threshold. If not, the total loss L(X,Y,N|M) is used for backpropagation of the network and fed back from the student network generation stage to the CSPNet v5 network stage. The convolution kernel parameters of each layer of the network are adjusted layer by layer until the distillation loss of the student network is less than the set threshold.

[0053] Furthermore, this invention can also modify the initial training parameters to optimize the network, resulting in an optimized target detection network. This optimization method can improve the model's detection accuracy to a certain extent, generating a lightweight optical remote sensing image target detection model that can be deployed on spacecraft.

[0054] Finally, the lightweight detection model trained by this invention can be deployed on a satellite hardware platform. The onboard GPU should be a low-power, low-latency, high-performance computing unit, with industrial-grade or aerospace-grade processing chips selected based on the application scenario. In other words, by deploying the lightweight neural network model of this invention on a satellite hardware platform, and in conjunction with optical payloads, the experimental verification work for the satellite's design mission can be completed.

[0055] It should be noted that this invention constructs an object detection model based on the PyTorch deep learning framework and performs iterative training of a lightweight network model.

[0056] The following example, using a target detection neural network model deployed on the NVIDIA Jetson TX2 spaceborne processing platform, further illustrates the lightweight optical remote sensing target detection method based on knowledge distillation networks of this invention.

[0057] (1) Training of lightweight models

[0058] This embodiment uses 1964 remote sensing images of aircraft targets extracted from Google Earth. The dataset is expanded to 15712 images by cropping, rotating and other methods. The dataset is then divided into training set, validation set and test set in a ratio of 7:2:1.

[0059] After setting the detailed parameters for network training, a lightweight neural network based on knowledge distillation is trained using the training set to obtain the object detection neural network model.

[0060] (2) Testing and Validation of the Lightweight Model

[0061] Considering that the "HiSilicon-1" satellite has already used NVIDIA Jetson TX1 to verify the feasibility of deploying a small GPU platform onboard, and given that the NVIDIA Jetson TX2 platform has a faster processing speed, it was selected as a component of the hardware-in-the-loop verification system to simulate and verify the constrained computing conditions under satellite on-orbit conditions.

[0062] The target detection neural network model was deployed on the NVIDIA Jetson TX2 spaceborne processing platform. Since the images simulated on the ground differ significantly from real-time satellite images, pre-stored 2880*2160 pixel wide-swath optical remote sensing images were used as input to the lightweight model to simulate its performance.

[0063] Based on actual detection results, the lightweight network model achieved a compression ratio of over 15 times, a detection rate of 1.16 FPS, and an average accuracy of 91.08%, meeting the frame rate requirements for on-board processing of wide-swath remote sensing images and achieving good detection performance. Figure 3 As shown.

[0064] The lightweight optical remote sensing image target detection model CKD-YOLOX based on YOLOX and knowledge distillation provided by this invention verifies the beneficial effect of the lightweight model on target detection of spaceborne optical remote sensing images, and also proves the practicality of this network on satellite embedded platforms.

[0065] Therefore, this invention can be applied to spaceborne GPU platforms with limited computing resources for target detection tasks in optical remote sensing images. The network structure of this method implements a lightweight design based on knowledge distillation, enabling the learning of more effective student networks. This allows the model to maintain high detection accuracy while improving network detection speed, achieving real-time on-orbit processing of target detection.

[0066] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A lightweight method for optical remote sensing target detection based on knowledge distillation networks, characterized in that, A target detection neural network model based on knowledge distillation is used to detect targets in real-time optical remote sensing images captured by satellites. The method for obtaining the target detection neural network model includes the following steps: S1: Obtain the target region and background region of each optical remote sensing image, and use the optical remote sensing image and target region as the input and output of the CSPNet v5 network respectively to train the CSPNet v5 network and obtain the teacher network. S2: Using the teacher network as input to the distillation network, the distillation network is trained under the guidance of the teacher network's network parameters to obtain the student network; The distillation process includes the following steps: 1) Expand from the top-level output features to the intermediate-level features, enabling more knowledge to be transferred to the student network and outputting refined feature maps; 2) Design a distillation loss function to constrain the training process of the student network and the teacher network, assigning high weight values ​​to key target areas in the remote sensing image and low weight values ​​to background information areas; The loss function includes two constraints: rotation IoU loss for performing regression tasks on the object detection neural network, which constrains the training process of the initial model; and distillation loss for information transfer between the teacher network and the student network, which constrains the training process of the distillation network. 3) Update the parameters of the student network through backpropagation and update the weights of the teacher network nodes in the downstream task through forward propagation. Specifically, this includes: determining whether the distillation loss between the student network and the teacher network is less than a set threshold. If yes, the current student network is the target detection neural network model. If no, the distillation loss is used for the backpropagation of the distillation network, and the rotation IoU loss is also used for the backpropagation of the CSPNet v5 network. This is to adjust the distillation network and the CSPNet v5 network until the distillation loss is less than the set threshold. The loss function is expressed as: Where M represents the network parameters of the model, This indicates a rotational IoU loss constraint. For balancing parameters, This represents the distillation loss between the student network and the teacher network; The rotational IoU loss The definition is as follows: in, These are the corner coordinates generated using the 180° five-parameter representation method to characterize the true detection box of the target. The corner coordinates are generated using the 180° five-parameter representation method to characterize the target prediction box after distillation. The corner coordinates include the box width, box height, x-coordinate of the target center point, y-coordinate of the target center point, and the rotation angle of the box. distillation loss The definition is as follows: Where W, H, and S represent the length, width, and number of channels of the feature map generated during the training of the distillation network, respectively. This indicates the distillation loss relationship between the teacher network and the student network. This represents the feature map obtained by extracting features from the student network. This represents the feature map obtained by the teacher network from feature extraction. The set balance parameters, For feature map The sum of the number of non-zero pixels.

2. The lightweight optical remote sensing target detection method based on knowledge distillation networks as described in claim 1, characterized in that, The CSPNet v5 network uses the YOLOX framework, with depthwise separable convolutional layers and skip connections between network nodes.

3. The lightweight optical remote sensing target detection method based on knowledge distillation networks as described in claim 1, characterized in that, The convolution calculation process of CSPNet v5 network and distillation network is divided into depthwise convolution and pointwise convolution operations. The number of output channels of the Depthwise layer is set to 128 and the kernel size is 3*3*1. The number of output channels of the Pointwise layer is set to be consistent with that of the Depthwise layer and the kernel size is 1*1*1.

4. The lightweight optical remote sensing target detection method based on knowledge distillation networks as described in claim 1, characterized in that, The optical remote sensing images described in step S1 are extracted from the DIOR dataset.

5. A lightweight method for optical remote sensing target detection based on knowledge distillation networks as described in claim 1, characterized in that, After filtering and denoising each optical remote sensing image and uniformly cropping it into pixel blocks of a set size, the pixel blocks are further divided into target pixel blocks and background pixel blocks. Finally, the target pixel blocks and background pixel blocks are used as input to the CSPNet v5 network.

Images