Rotating frame remote sensing target detection method based on lightweight deep neural network

Abstract

The present invention relates to a rotating frame remote sensing target detection method based on a lightweight deep neural network, comprising: obtaining a remote sensing target detection data set and preprocessing it, dividing the data set into a training set and a test set; building a single-stage target detection network model based on a rotating frame; using the training set to iteratively train the single-stage target detection network model based on a rotating frame, so that the single-stage target detection network model based on a rotating frame converges; inputting the test set into the single-stage target detection network model based on a rotating frame, and verifying the performance; and performing structured pruning and compression processing on the single-stage target detection network model based on a rotating frame to make it lightweight. The method of the present invention can improve the detection and recognition accuracy and speed of slender and closely adjacent targets such as ships in port areas of remote sensing images.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and remote sensing image processing, and in particular to a rotating frame remote sensing target detection method based on a lightweight deep neural network. Background Art

[0002] With the rapid development of remote sensing earth observation technology, a large number of high-resolution optical remote sensing satellites have emerged, promoting the application of space remote sensing technology in many fields such as military, national economy, and global environmental protection. The rise of artificial intelligence has brought a new paradigm for the processing of massive remote sensing data. The use of artificial intelligence algorithms to improve the on-orbit processing capabilities of remote sensing satellite data can realize real-time and intelligent space-based information services.

[0003] However, the lack of precision of artificial intelligence algorithms at the fine-grained level and the limited computing power of onboard computing platforms have restricted the formation and development of space intelligent perception capabilities. Although current deep learning technology has been greatly improved compared to traditional methods, and has a good effect on detection tasks with large inter-class differences, such as the identification of aircraft, ships, and bridges, at the fine-grained level, such as the detection and identification of ship type-level granularity, its MAP value is far from crossing the actual application threshold, and there is still a long way to go from the indicators expected by users. Therefore, first of all, it is necessary to develop high-performance, fine-grained intelligent detection and identification algorithms with on-orbit application capabilities. Secondly, the on-orbit resources (computing, storage, energy consumption, etc.) of spacecraft are limited, which are 1 to 2 orders of magnitude lower than the general processing power on the ground. The deep neural network model is large in scale and often has the characteristics of intensive computing and intensive memory access. Directly using existing deep learning methods for on-orbit information processing cannot meet the real-time requirements of the task. So, when deep learning is applied to the aerospace field, how to keep the existing neural network performance basically unchanged while significantly reducing the network calculation amount and the storage of the neural network model so that the neural network model can run efficiently on resource-constrained onboard computing devices has become an increasingly prominent issue. Summary of the invention

[0004] In order to improve the detection and recognition capabilities of slender and closely adjacent ship targets in the port area of ​​remote sensing images, the present invention provides a rotating frame remote sensing target detection method based on a lightweight deep neural network.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is:

[0006] The present invention provides a rotating frame remote sensing target detection method based on a lightweight deep neural network, comprising: obtaining a remote sensing target detection data set and performing preprocessing, dividing the data set into a training set and a test set; building a rotating frame-based single-stage target detection network model; using the training set to iteratively train the rotating frame-based single-stage target detection network model to make it converge; inputting the test set into the rotating frame-based single-stage target detection network model and verifying the performance of the network model; and performing structured pruning and compression processing on the network model to form a lightweight rotating frame-based single-stage target detection network model.

[0007] According to one aspect of the present invention, the pre-processing process comprises:

[0008] Annotate the images in the data set to obtain corresponding labels and annotation files;

[0009] Perform overlapping cutting on the image, and make the cut image correspond to the label;

[0010] Convert the processed label format into a six-tuple format label;

[0011] Convert the updated dataset into a database file in LMDB format;

[0012] Perform target feature analysis on the dataset used for training.

[0013] According to one aspect of the present invention, the data annotation process includes: using the four-point method to annotate the coordinates of the four corner points (p1, p2, p3, p4) of the minimum circumscribed rectangular box of the image target to determine the position of the target, and annotating the image category label, wherein p1, p2, p3, p4 respectively represent the upper left vertex, upper right vertex, lower right vertex and lower left vertex of the circumscribed rectangular box arranged in a clockwise direction.

[0014] According to one aspect of the present invention, the overlapping cutting process includes: cutting the image into sub-images from the upper left corner in a sliding window manner, setting overlapping areas between adjacent sub-images, and eliminating incomplete targets whose intersection over union (IoU) of the remaining targets and the original targets is less than a threshold.

[0015] According to one aspect of the present invention, the process of six-tuple format label conversion includes: using a six-tuple (cls, cx, cy, w, h, angle) to encode a rotated bounding box for the image target, and determining its category information and position information in the original image, wherein cls represents the target category, cx and cy represent the coordinates of the center point of the target true bounding box in the original image, w and h represent the width and height of the target true bounding box respectively, and angle represents the angle between the width w corresponding to the positive direction of the target true bounding box and the positive direction of the x-axis, and its range is in the interval [0,2π).

[0016] According to one aspect of the present invention, the target characteristic analysis process includes: statistically analyzing the category distribution of the target, and understanding the size distribution of the target through clustering and averaging, where the size is the maximum width, maximum height, average width, and aspect ratio.

[0017] According to one aspect of the present invention, the process of building a single-stage target detection network model based on a rotating frame includes:

[0018] The improved VGG16 neural network is used as the backbone network to extract the preprocessed image features, obtain feature maps of different sizes, and construct a feature pyramid network model;

[0019] Using the SSD detection algorithm based on the rotation box to set a rotation target prior selection box (prior rotation box) with the feature point on the feature map as the center point to predict the target category and regress the position information;

[0020] Use the convolutional neural network Caffe deep learning framework to build a single-stage target detection network model based on rotating boxes.

[0021] According to one aspect of the present invention, the process of iteratively training the network model using the training set includes: using the training set images and annotation files obtained by overlapping segmentation to train the rotating box-based single-stage target detection network model, and stopping the iterative training when the total loss function of the rotating box-based single-stage target detection network model converges.

[0022] According to one aspect of the present invention, the process of verifying the performance of the single-stage target detection network model based on the rotating frame includes:

[0023] The images of the test set are cut in overlapping manner according to the input size to obtain sub-images, and then the sub-images are input into the trained single-stage target detection network model based on the rotation box to obtain target detection results with rotation information, and all the results are merged;

[0024] The redundant detection frames are eliminated by non-maximum suppression of the rotating frame to obtain the target detection result information of the original test image.

[0025] According to one aspect of the present invention, the process of performing structured pruning and compression processing on the single-stage target detection network model based on the rotating frame to form a lightweight network model includes:

[0026] Select the best performing single-stage rotating frame-based object detection network model, and readjust the number of channels and layers of the network model for streamlining;

[0027] Performing sensitivity analysis on the parameters of the rotating frame-based single-stage target detection network model to measure the importance of neurons;

[0028] Pruning the rotating frame based single-stage target detection network model and cutting off unimportant neurons;

[0029] Fine-tuning the pruned single-stage rotation box-based object detection network model to regain the lost performance;

[0030] The fine-tuned single-stage rotating box-based object detection network model is post-processed to delete the parts whose parameters are 0 during compression.

[0031] Beneficial effects:

[0032] According to the solution of the present invention, by using a rotating box instead of a horizontal box as a preset box, the cross interference of adjacent targets and the cross interference between the target and the background are reduced. The scale distribution of the target in a given data set is obtained by analyzing the target characteristics, so as to adaptively design the parameters of the single-stage target detection network model based on the rotating box to improve the accuracy of target detection. By constructing a feature pyramid network model, classification and position regression tasks are performed simultaneously on multiple different feature maps to achieve recognition of targets of different scales. In order to deploy the network model on a hardware device, the Caffe framework is used to build a single-stage target detection network model based on the rotating box, and the trained network model is structured pruned and compressed, and the model is lightweight to accelerate the reasoning speed of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart schematically showing a rotating frame remote sensing target detection method based on a lightweight deep neural network according to an embodiment of the present invention;

[0034] Figure 2 A schematic diagram schematically showing a data set annotation format of a rotating frame remote sensing target detection method based on a lightweight deep neural network according to an embodiment of the present invention;

[0035] Figure 3A schematic diagram schematically shows a result of analyzing target characteristics of a data set of a rotating frame remote sensing target detection method based on a lightweight deep neural network according to an embodiment of the present invention;

[0036] Figure 4 A schematic diagram schematically showing a rotating default frame design of a rotating frame remote sensing target detection method based on a lightweight deep neural network according to an embodiment of the present invention;

[0037] Figure 5 A schematic diagram schematically showing the overall structure of a target detection network model of a rotating frame remote sensing target detection method based on a lightweight deep neural network according to an embodiment of the present invention;

[0038] Figure 6 A structured pruning flowchart of a rotating frame remote sensing target detection method based on a lightweight deep neural network according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.

[0041] Figure 1 The schematic diagram shows the various execution steps and processes of a rotating frame remote sensing target detection method based on a lightweight deep neural network in this embodiment. Figure 1 As shown, a rotating frame remote sensing target detection method based on a lightweight deep neural network in this embodiment specifically includes the following steps:

[0042] First, a remote sensing ship data set is obtained and preprocessed, and the data set is divided into a training set and a test set according to a certain ratio, such as 8:2. The data set of this embodiment contains 1780 remote sensing ship images. The above preprocessing process specifically includes: firstly, the images of the remote sensing ship data set are annotated using the four-point method to obtain the labels and annotation files corresponding to the images. Then, at this time, the data set contains 1780 remote sensing ship images and their corresponding labels (included in the annotation file). Then, the remote sensing ship images are overlapped and cut, and the cut images are matched with the labels, and then the label format represented by the four-point method is converted into a six-tuple format label. After that, the sorted data set is converted into a database file in the flash memory-mapped database (Lightning Memory-Mapped Database, LMDB) format required in the training process of the convolutional neural network (Convolutional Architecture for Fast Feature Embedding, Caffe). Finally, the target characteristics of the data set used for training are analyzed to refine the parameter design of the network model.

[0043] Figure 2 The following is a schematic diagram showing the dataset annotation format of the rotating frame remote sensing target detection method based on a lightweight deep neural network according to the present embodiment. Figure 2 As shown in (a), the process of four-point annotation is as follows: mark the position of the target using the coordinates of the four corner points of the target's minimum circumscribed rectangular box (p1, p2, p3, p4), and annotate the image category label according to the type of ship. It is stipulated that (p1, p2) represents the two vertices in the direction pointed by the bow of the ship, p1 represents the upper left point of the bow, p2 represents the upper right point of the bow, and they are arranged in a clockwise direction. P3 represents the lower right point of the bow, and p4 represents the lower left point of the bow.

[0044] The above overlapping cutting process includes: cutting the original super-large remote sensing ship image into sub-images of appropriate size from the upper left corner in a sliding window manner, setting an overlapping area of ​​a certain size between adjacent sub-images, and eliminating incomplete targets whose intersection over union (IoU) between the remaining target and the original target is less than a certain threshold, that is, directly deleting the annotation information of the incomplete target, directly deleting the sub-image that does not contain the target, and processing the annotation file corresponding to the cut sub-image at the same time.

[0045] The above converts the cut and sorted annotation files into Figure 2(b) shows a label in the format of a six-tuple (cls, cx, cy, w, h, angle). The six-tuple (cls, cx, cy, w, h, angle) is used to encode the target's rotation bounding box to determine its category information and position information in the original image. Among them, cls represents the target category, cx and cy represent the coordinates of the center point of the target's true bounding box in the original image, w and h represent the width and height of the target's true bounding box respectively, and angle represents the angle between the width w corresponding to the positive direction of the target's true bounding box and the positive direction of the x-axis, and the angle range is in the interval [0,2π). That is, Figure 2 As shown in (a), the width w is the distance between p1 and p2, the height h is the distance between p1 and p4, and angle is a vector Specifically, using the Pythagorean theorem and trigonometric function formulas, Figure 2 The label format shown in (a) is transformed into Figure 2 (b) shows the label format.

[0046] The above converts the dataset and labels into the LMDB format database file required for Caffe training. The LMDB format dataset is a very fast, lightning-fast memory-mapped database that greatly reduces the time overhead of disk IO when the system accesses a large number of small files, so the images and labels are converted into the LMDB database format.

[0047] Figure 3 The following is a schematic diagram showing the target characteristic analysis results of a data set of a rotating frame remote sensing target detection method based on a lightweight deep neural network according to the present embodiment. The target characteristic analysis process is to use statistical analysis to analyze the target category distribution in the data set, and to understand the target size distribution through clustering, averaging and other methods. The size includes target angle distribution, target aspect ratio distribution, maximum width, maximum height, average width and other information. Figure 3 The number of targets in the data set shown in histogram (a) is very unbalanced, which makes model training difficult. We use category information to design a classification loss function. Figure 3 The angle information of the target shown in histogram (b) is widely distributed in the entire angle range, such as Figure 3 The histogram (c) shows that most of the target aspect ratios are concentrated in a narrow range. These characteristic information is used to design the parameters of the SSD detection algorithm's prior rotation box so that the distribution of the prior rotation box can cover the actual target box. By analyzing the target characteristics, the scale distribution of the target in a given data set is obtained, and the parameters of the single-stage target detection network model based on the rotation box are adaptively designed to improve the accuracy of target detection.

[0048] Secondly, we use the Caffe deep learning framework to build a single-stage target detection network model based on the rotating box, and design a loss function for network model training. The specific process includes the following steps:

[0049] The improved SSD-VGG-16 is used for target detection. First, the improved VGG16 neural network is used as the backbone network to extract the preprocessed image features, obtain feature maps of different sizes, and form a feature pyramid network model. Specifically, the improved VGG16 neural network changes the fully connected operations of FC6 and FC7 of the original VGG16 into convolution operations, and the improved SSD changes the original horizontal box operator into a rotation box operator, including: rotation target box feature extraction, rotation target box IOU calculation, rotation target box non-maximum suppression (NMS), etc., and the feature maps obtained by multi-layer convolutional layers are used to form a feature pyramid network model, and Softmax classification and position regression are performed simultaneously on these feature layers of different scales. Then, the SSD detection algorithm based on the rotation box is used to set the rotation target pre-selection box prior rotation box with the feature point on the feature map as the center point to predict the target category and regress the position information. Specifically, SSD uses the prior rotation box to generate a set of rotation target pre-selection boxes with different aspect ratios, areas, and angles with each feature point as the center point in the different scale feature layers, such as Figure 4 (a) is shown. After determining whether the prior rotation box contains the target, the precise position of the target is obtained through bounding box regression. Since the position information of the ship target has an angle component and the angles of the targets in the entire data set are widely distributed, multiple angles are set when laying the prior rotation box at each feature point to cover targets of each angle as much as possible, such as Figure 4 As shown in (b), the rotated target pre-selected box (prior rotation box) can better cover the real box. Figure 4 The schematic diagram shows the design of the rotating default box of the rotating box remote sensing target detection method based on a lightweight deep neural network in this embodiment. The final target prediction outputs the target candidate box and category with angle information. The target candidate box includes six parameters (cls, cx, cy, w, h, angle). The rotating candidate area prediction network model adopts a multi-task loss function composed of a classification loss function and a position loss function. The loss function is as follows:

[0050]

[0051] Among them, the classification loss function is a typical softmax loss, as follows:

[0052]

[0053]

[0054]

[0055] Position loss function in traditional smooth L1 The angle component is added to the loss as follows:

[0056]

[0057]

[0058] On the one hand, this embodiment uses a rotating frame instead of a horizontal frame as a preset frame to reduce the cross interference between adjacent targets and the cross interference between the target and the background. On the other hand, by constructing a feature pyramid network model, classification and position regression tasks are performed simultaneously on multiple different feature maps to achieve recognition of targets of different scales.

[0059] Figure 5 The overall structure diagram of the target detection network model of the rotating frame remote sensing target detection method based on a lightweight deep neural network in this embodiment is schematically shown. Figure 5 As shown, the target detection network model of this embodiment is designed in an end-to-end form, which improves the speed of the target detection process.

[0060] Then, the generated training set is input into the single-stage object detection network model based on the rotating box, and the network model is iteratively trained until the network converges. The specific process includes: using the training set images and annotation files obtained by overlapping segmentation to train the built network model, and when the total loss function of the network converges, the iterative training is stopped.

[0061] After that, the test samples are input into the trained network model, and the model performance is verified based on the classification and target detection results of the ship images with rotation angle information. The specific process includes: overlapping cutting the test set images according to the network model input size to obtain sub-images of a certain size, inputting each sub-image into the trained network model to obtain the target detection results with rotation information, summarizing all the results, and using the rotation box NMS method to eliminate the detection results of redundant target boxes, and finally obtaining the target detection result information of the original test image.

[0062] Then, since different training hyperparameters will affect the training results of the model during the iterative training process, and the model detection accuracy will also change from low to high to low as the training time increases, it is necessary to evaluate the results of multiple trainings and select the model with the highest detection accuracy as the final model. Finally, the final model is compressed by structured pruning to form a lightweight model, which reduces the storage space of the original model and improves the model running speed while ensuring accuracy, so that the lightweight model can be deployed in embedded devices with limited hardware resources. Figure 6 The structured pruning flow chart of the rotating frame remote sensing target detection method based on a lightweight deep neural network of this embodiment is schematically shown. Figure 6 As shown in the figure, the operation steps of the above structured pruning are as follows:

[0063] Taking the trained over-parameterized model as a benchmark, the trained large model is pruned according to specific standards, that is, the number of channels and layers in the network model structure are readjusted to obtain a streamlined network structure;

[0064] Conduct sensitivity analysis on the parameters of the network model to measure the importance of neurons in the neural network;

[0065] The pruning ratio is set according to the network parameters, and the pruning threshold is calculated according to the pruning ratio, and neurons smaller than the threshold are pruned;

[0066] Fine-tune the pruned model to regain the lost performance;

[0067] Post-process the model obtained by fine-tuning and delete the parts whose parameters are 0 during compression. Fine-tuning here refers to retraining the pruned model.

[0068] This implementation method achieves lightweight target detection deep neural network model by compressing the trained and verified model through structured pruning, which greatly accelerates the reasoning speed of the model. According to the scheme of this implementation method, in order to solve the problem that the existing remote sensing image ship detection model cannot finely classify different ships and is not hardware-friendly, it can detect ship targets in remote sensing images and classify them in a fine-grained manner, and use the caffe framework to build a single-stage network model that is hardware-friendly.

[0069] The above is only one embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A rotating frame remote sensing target detection method based on a lightweight deep neural network, comprising: Acquire a remote sensing target detection data set and perform preprocessing, and divide the data set into a training set and a test set; Build a single-stage object detection network model based on rotating boxes; Iteratively training the rotating frame-based single-stage target detection network model using the training set to converge the rotating frame-based single-stage target detection network model; Inputting the test set into the rotating frame-based single-stage target detection network model, and verifying the performance of the rotating frame-based single-stage target detection network model; Performing structured pruning and compression processing on the rotating frame-based single-stage target detection network model to make the rotating frame-based single-stage target detection network model lightweight; The pre-treatment process includes: Annotate the images in the data set to obtain corresponding labels and annotation files; Perform overlapping cutting on the image, and make the cut image correspond to the label; Convert the processed label format into a six-tuple format label; Converting the converted labels and the annotated images into a database file in a flash memory mapping database format; performing target characteristic analysis on the training set; The data annotation process includes: using the four-point method to annotate the coordinates of the four corner points (p1, p2, p3, p4) of the minimum bounding rectangle of the target in the image to determine the position of the target, and annotating the image category label, wherein p1, p2, p3, p4 respectively represent the upper left vertex, upper right vertex, lower right vertex and lower left vertex of the bounding rectangle arranged in a clockwise direction; The process of six-tuple format label conversion includes: using a six-tuple (cls, cx, cy, w, h, angle) to encode a rotated bounding box for the target of the image, and determining its category information and position information in the original image, wherein cls represents the target category, cx and cy represent the coordinates of the center point of the target real bounding box in the original image, w and h represent the width and height of the target real bounding box respectively, and angle represents the angle between the width w corresponding to the positive direction of the target real bounding box and the positive direction of the x-axis, and its range is in the interval [0,2π).

2. The remote sensing target detection method according to claim 1, characterized in that: The overlapping cutting process includes: cutting the image into sub-images from the upper left corner in a sliding window manner, setting overlapping areas between adjacent sub-images, and eliminating incomplete targets whose intersection-over-union ratios of the remaining targets and the original targets are less than a threshold.

3. The remote sensing target detection method according to claim 1, characterized in that: The target characteristic analysis process includes: statistically analyzing the category distribution of the target, and obtaining the size distribution of the target by clustering and averaging, wherein the size is the maximum width, the maximum height, the average width, and the aspect ratio.

4. The remote sensing target detection method according to claim 1, characterized in that: The process of building a single-stage target detection network model based on a rotating frame includes: Use neural network as the backbone network to extract preprocessed image features, obtain feature maps of different sizes, and construct a feature pyramid network model; Using a rotating frame-based SSD detection algorithm, a rotating target pre-selection frame is set with the feature point on the feature map as the center point to predict the target category and regress the position information; Use the convolutional neural network deep learning framework to build a single-stage target detection network model based on rotating boxes.

5. The remote sensing target detection method according to claim 1, characterized in that: The process of iteratively training the network model using the training set includes: using the images and annotation files in the training set obtained by overlapping segmentation to train the single-stage target detection network model based on the rotating box, and stopping the iterative training when the total loss function of the single-stage target detection network model based on the rotating box converges.

6. The remote sensing target detection method according to claim 1, characterized in that: The process of verifying the performance of the single-stage target detection network model based on the rotating frame includes: The images of the test set are cut in overlapping manner according to the input size to obtain sub-images, and then the sub-images are input into the trained single-stage target detection network model based on the rotation box to obtain target detection results with rotation information, and all the results are merged; The rotating frame non-maximum suppression method is used to remove redundant detection frames and obtain the target detection result information of the original test image.

7. The remote sensing target detection method according to claim 1, characterized in that: The process of performing structured pruning and compression processing on the single-stage target detection network model based on the rotating frame to make it lightweight includes: Adjusting the number of channels and layers of the rotating frame-based single-stage object detection network model; Performing sensitivity analysis on the parameters of the rotating frame based single-stage object detection network model; Pruning the rotation box based single-stage object detection network model; Fine-tune the pruned single-stage rotation box-based object detection network model; Post-processing of the fine-tuned rotation box based single-stage object detection network model.

Images