Method and device for aircraft detection in optical remote sensing images based on cascade regression correction

Abstract

The present invention provides an optical remote sensing image aircraft detection method and device based on cascade regression correction, comprising: cutting the collected original optical remote sensing image to obtain multiple slice images; inputting the multiple slice images into an aircraft detection model in sequence, and outputting the detection frame position information of the target aircraft detected in each slice; the aircraft detection model is cascaded based on a preliminary detection frame correction model and a secondary correction classification model, the preliminary detection frame correction model is obtained by training based on a sample first image and a target object annotation frame, and the secondary correction classification model is obtained by training based on a sample second image, a corresponding target object annotation frame, and whether the target object is an aircraft classification label; the detection frame position information is mapped back to the coordinates of the original optical remote sensing image and merged to obtain the final aircraft detection result. The method provided by the present invention realizes efficient detection of target aircraft in remote sensing images, and the robustness and adaptability of the detection results are improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and in particular to an optical remote sensing image aircraft detection method and device based on cascade regression correction. Background Art

[0002] In recent years, with the development of remote sensing technology and the improvement of remote sensing image resolution, the acquisition method has become more convenient, and its research value has become increasingly higher. As an important means of transportation and conventional weapon, aircraft plays an important role in the civil and military fields and is the focus of remote sensing target detection. In the civil field, with the rapid development and progress of society and the continuous improvement of people's quality of life, aircraft has become one of the indispensable means of transportation for people to travel. By analyzing the remote sensing images of the airport area, the number and model of the current aircraft can be automatically detected, which can help airlines better implement scheduling management and aircraft statistics. It not only saves manpower costs, but also improves safety. In the military field, air strikes are the most important part of military strikes. There is no doubt that aircraft are the most important combat tools. Therefore, the positioning and identification of aircraft can provide important battlefield dynamics and intelligence.

[0003] Remote sensing image processing is mainly done through manual interpretation. Although the accuracy is relatively high, it consumes a lot of human resources and has a slow processing speed, which makes it difficult to meet the timeliness required in the modern information age. Therefore, people consider using computers to efficiently process objects to solve the problem of remote sensing image processing. At present, many automated remote sensing image target detection methods have been proposed, which can be divided into two categories: traditional machine learning methods and deep learning methods. For the former, the results are obtained mainly by manually extracting features and training classifiers, so the manually extracted target features are very important. However, these features usually lack robustness and versatility, and it is difficult to obtain accurate results. In recent years, convolutional neural networks (CNNs) have been proposed, which has attracted widespread attention from researchers and is a milestone development in the field of computer vision. Target recognition technology based on convolutional neural networks can automatically extract various features of images through network learning, and finally use the extracted features to locate and classify targets. This method avoids the manually designed feature extraction process, and can also automatically learn the deep information of the target, the surrounding and background information of the target. In addition, the locator constructed by the convolution layer also enhances the accuracy of positioning. Therefore, the robustness and accuracy of the target recognition method based on convolutional neural network are much higher than the traditional method.

[0004] However, large-field optical remote sensing images usually have complex backgrounds, many false alarm interference factors, and large scale differences between aircraft targets. General deep learning methods are difficult to achieve accurate detection of aircraft in remote sensing images.

[0005] Therefore, how to avoid the inability of existing aircraft target detection algorithms to utilize remote sensing image characteristics, resulting in weak robustness and adaptability of aircraft target detection results and low detection efficiency, remains a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] The present invention provides an optical remote sensing image aircraft detection method and device based on cascade regression correction, which is used to solve the problem that the existing aircraft target detection algorithm cannot utilize the remote sensing image characteristics, resulting in weak robustness and adaptability of the obtained aircraft target detection results and low detection efficiency.

[0007] The present invention provides an optical remote sensing image aircraft detection method based on cascade regression correction, comprising:

[0008] The collected original optical remote sensing image is trimmed to obtain a plurality of slice images;

[0009] Input the plurality of slice images into an aircraft detection model in sequence, and output detection frame position information of a target aircraft detected in each slice;

[0010] The aircraft detection model is formed by cascading a preliminary detection frame correction model and a secondary correction classification model, wherein the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, and the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame, and a classification label of whether the target object is an aircraft, wherein the sample first image comes from a first sample set constructed based on a cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame;

[0011] The detection frame position information is mapped back to the coordinates of the original optical remote sensing image and merged to obtain a final aircraft detection result.

[0012] According to an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, the network structure of the preliminary detection frame correction model training includes a multi-scale feature extraction network, a priori anchor frame generation network and a detection frame correction network connected in sequence;

[0013] The multi-scale feature extraction network is used to downsample any input image to obtain N size feature maps of the image, and fuse the N size feature maps according to a preset rule to obtain N fused feature maps of the image;

[0014] The prior anchor frame generation network is used to determine the number of prior anchor frames generated at each point of N fusion feature maps based on the number of preset prior anchor frame aspect ratios and the number of prior anchor frame scales;

[0015] The detection frame correction network is used to optimize and correct the prior frame parameters based on the offset between the prior anchor frame and the real target object annotation frame.

[0016] According to an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, the re-corrected classification model includes the multi-scale feature extraction network, the detection frame re-correction network and the classification network;

[0017] The detection frame correction network is used to optimize and correct the parameters of the preliminary detection frame based on the offset between the preliminary detection frame sample and the real object annotation frame;

[0018] The classification network is used to optimize the confidence parameters in the classification network based on the classification error between the probability that the predicted target object is an aircraft and the target aircraft annotation frame determined by the predicted detection frame output by the correction network again based on the detection frame.

[0019] According to an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, the construction of the first sample set specifically includes:

[0020] The collected sample optical remote sensing image is subjected to several rotation, horizontal and vertical flipping operations to obtain a preliminary expanded sample optical remote sensing image;

[0021] Performing mosaic data enhancement on the initially expanded sample optical remote sensing image to obtain a fully expanded sample optical remote sensing image;

[0022] Each image in the fully expanded sample optical remote sensing images is cropped to obtain a first sample set.

[0023] According to an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, the construction of the second sample set specifically includes:

[0024] Input the images in the first sample set into the trained preliminary detection frame correction model, and output a sample second image with a preliminary detection frame for each sample image;

[0025] A second sample set is constructed based on the second sample image.

[0026] According to an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, the collected original optical remote sensing image is clipped to obtain multiple slice images, specifically including:

[0027] Using overlapping sliding windows to crop the acquired original optical remote sensing images, a cropped image sequence is obtained in which there are overlapping pixels between the previous and next cropped images;

[0028] The images in the cropped image sequence are sequentially difference-upsampled to obtain a plurality of slice images.

[0029] According to an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain the final aircraft detection result specifically includes:

[0030] Determine a restoration mapping function based on the sampling rule of the difference upsampling;

[0031] Mapping the detection frame position information back to the coordinates of the original optical remote sensing image based on the restoration mapping function to obtain target result coordinates;

[0032] The overlapping data in the target result coordinates are merged using a non-maximum suppression rule to obtain the final aircraft coordinate detection result.

[0033] The present invention also provides an optical remote sensing image aircraft detection device based on cascade regression correction, comprising:

[0034] A cropping unit, used for cropping the collected original optical remote sensing image to obtain a plurality of slice images;

[0035] A detection unit, used to sequentially input the plurality of slice images into an aircraft detection model, and output detection frame position information of a target aircraft detected in each slice;

[0036] The aircraft detection model is formed by cascading a preliminary detection frame correction model and a secondary correction classification model, wherein the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, and the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame, and a classification label of whether the target object is an aircraft, wherein the sample first image comes from a first sample set constructed based on a cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame;

[0037] The stitching unit is used to map the detection frame position information back to the coordinates of the original optical remote sensing image and merge them to obtain a final aircraft detection result.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the optical remote sensing image aircraft detection method based on cascade regression correction as described in any one of the above are implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the optical remote sensing image aircraft detection method based on cascade regression correction as described in any one of the above.

[0040] The present invention provides an optical remote sensing image aircraft detection method and device based on cascade regression correction. Since the aircraft detection model provided by the present invention is based on the cascade of a preliminary detection frame correction model and a secondary correction classification model, a cascade regression correction module is designed. Compared with the single regression in the traditional method, the cascade regression aircraft detection model corrects the generated prior anchor frame, and then continues to perform a secondary correction through regression based on the previous preliminary correction result to obtain the final aircraft target detection result. The two regression correction methods can obtain more accurate aircraft detection results, and the network structure complexity of the aircraft detection model is not high. Therefore, the method and device provided by the present invention realize the efficient detection of target aircraft in remote sensing images, and the robustness and adaptability of the detection results are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A schematic diagram of the flow of an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention;

[0043] Figure 2 A schematic diagram of the flow of an optical remote sensing image aircraft detection method based on a cascaded regression correction module provided by the present invention;

[0044] Figure 3 A schematic diagram of the structure of an optical remote sensing image aircraft detection device based on cascade regression correction provided by the present invention;

[0045] Figure 4 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The existing aircraft target detection algorithms generally fail to utilize the characteristics of remote sensing images, resulting in weak robustness and adaptability of the aircraft target detection results and low detection efficiency. Figure 1-Figure 2 The present invention describes an optical remote sensing image aircraft detection method based on cascade regression correction. Figure 1 A schematic diagram of a flow chart of an optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, such as Figure 1 As shown, the method includes:

[0048] Step 110, cropping the collected original optical remote sensing image to obtain a plurality of slice images.

[0049] Specifically, since the collected original remote sensing images are usually high-resolution, large-field-of-view images, their pixel size usually reaches more than four digits. Due to the limitations of hardware conditions and the network structure itself, the aircraft detection model that can process remote sensing images introduced later cannot directly process high-resolution, large-field-of-view original remote sensing images. Therefore, it is necessary to crop them to crop an original remote sensing image into multiple slice images, and the combination of the multiple slice images includes all pixel information in the original remote sensing image.

[0050] Step 120, inputting the plurality of slice images into an aircraft detection model in sequence, and outputting the detection frame position information of the target aircraft detected in each slice;

[0051] Among them, the aircraft detection model is composed of a cascade of a preliminary detection frame correction model and a secondary correction classification model, the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame and whether the target object is an aircraft classification label, the sample first image comes from a first sample set constructed based on the cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame.

[0052] Specifically, a plurality of slice images are sequentially input into the aircraft detection model, and the aircraft model will correspondingly sequentially output the detection frame position information of the target aircraft detected in each slice image, and the position information is measured in the image coordinate system of each slice image.

[0053] Here, the aircraft detection model is further limited to be composed of a cascade of a preliminary detection frame correction model and a secondary correction classification model. By designing a cascade regression correction module, the generated prior anchor frame is corrected, and then a secondary correction is performed through regression based on the previous preliminary correction result to obtain the final aircraft target detection result. The two regression correction methods can obtain more accurate aircraft detection results. The preliminary detection frame correction model is only used to obtain a preliminary correction amount and output a preliminary corrected detection frame based on a preset prior frame. When the secondary correction classification model is corrected with the same correction principle, the correction basis is no longer the preset prior frame used in the preliminary detection frame correction model, but the preliminary correction detection frame that has been adjusted once output by the preliminary detection frame correction model. A secondary correction is performed on the basis of the preliminary correction detection frame to obtain a more accurate detection frame position. Secondly, the secondary correction classification model also includes a classification function, and the fully corrected anchor frame obtained by the secondary correction classification model is classified into two categories to determine whether the anchor frame contains an aircraft target. The classification loss function adopts focal loss.

[0054] Therefore, the preliminary detection frame correction model is obtained after training based on the sample first image and the corresponding target object annotation frame; the second correction classification model is obtained after training based on the sample second image, the corresponding target object annotation frame and whether the target object is an aircraft classification label, wherein the second correction classification model includes a secondary correction network and a classification network, the parameters to be optimized in the secondary correction network are adjusted based on the positioning loss function constructed based on the positioning error of the complete correction frame, and the parameters to be optimized in the classification network are adjusted based on the positioning loss function constructed based on the positioning error of the complete correction frame. The sample first image in the first sample set should be cropped in the same way as the input original remote sensing image to be detected when using the aircraft detection model, and the sample second image in the second sample set is the sample second image obtained by processing the sample first image in the first sample set using the trained preliminary detection frame correction model after the preliminary detection frame correction model is trained, that is, the cropped optical remote sensing image with the preliminary detection frame. Therefore, the second correction classification model in the aircraft detection model can accurately mark the detection frame of the target object detected in each remote sensing image slice, and can also classify and identify whether the target object is a target aircraft, and obtain the final target aircraft detection frame as the output of the entire aircraft detection model.

[0055] Step 130 : Mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain a final aircraft detection result.

[0056] Specifically, the detection result coordinates of each optical remote sensing image slice obtained in step 120 must be mapped back to the coordinate system of the original remote sensing image. The mapping rule is determined based on the method of cropping the original remote sensing image into remote sensing image slices, that is, the mapping rule is the inverse operation of the cropping method to obtain the real coordinates, and then the real coordinates are spliced ​​to obtain the final aircraft detection positioning frame position information.

[0057] The optical remote sensing image aircraft detection method based on cascade regression correction provided by the present invention, because the aircraft detection model provided by the present invention is based on the cascade of the preliminary detection frame correction model and the secondary correction classification model, by designing a cascade regression correction module, compared with the single regression in the traditional method, the cascade regression aircraft detection model corrects the generated prior anchor frame, and then continues to perform secondary correction through regression based on the previous preliminary correction result to obtain the final aircraft target detection result, the two regression correction method can obtain more accurate aircraft detection results, and the network structure complexity of the aircraft detection model is not high. Therefore, the method provided by the present invention realizes efficient detection of target aircraft in remote sensing images, and the robustness and adaptability of the detection results are improved.

[0058] Based on the above embodiment, in the method, the network structure of the preliminary detection box correction model training includes a multi-scale feature extraction network, a priori anchor box generation network and a detection box correction network connected in sequence;

[0059] The multi-scale feature extraction network is used to downsample any input image to obtain N size feature maps of the image, and fuse the N size feature maps according to a preset rule to obtain N fused feature maps of the image;

[0060] The prior anchor frame generation network is used to determine the number of prior anchor frames generated at each point of N fusion feature maps based on the number of preset prior anchor frame aspect ratios and the number of prior anchor frame scales;

[0061] The detection frame correction network is used to optimize and correct the prior frame parameters based on the offset between the prior anchor frame and the real target object annotation frame.

[0062] Specifically, the multi-scale feature extraction network uses a convolutional neural network to extract features from the input image data. The specific convolutional neural network adopts the resnet-50 architecture, and its output results are feature maps of different sizes. The specific steps are as follows:

[0063] 1. The input image is sent to the resnet-50 convolutional neural network, and a variety of feature maps of different sizes are obtained through convolution calculation. In this embodiment, four feature maps are preferably used as input for subsequent prediction of the position of the aircraft target to adapt to the aircraft targets of target size in the optical remote sensing image. The sizes of these four feature maps are obtained by downsampling the original image by 4 times, 8 times, 16 times and 32 times, respectively.

[0064] 2. The four feature maps of different sizes obtained above are fused. Specifically: First, by using 1x1 convolution to reduce the dimension, the number of channels in each feature map is reduced to reduce the network scale and computational complexity. Secondly, after the low-resolution feature map is interpolated and upsampled, the two sets of feature maps are preliminarily fused through the feature map splicing operation, that is, the feature map downsampled 32 times is upsampled to 16 times feature map and then spliced ​​with the original 16 times feature map to update the 16 times feature map, and then the updated 16 times feature map is upsampled to 8 times feature map to splice the original 8 times feature map to update the 8 times feature map, and finally the updated 8 times feature map is upsampled to 4 times feature map to splice the original 4 times feature map to update the 4 times feature map, that is, the original downsampled 32 times feature map is retained, and other original downsampled feature maps are updated. The updating method is splicing fusion, that is, the number of channels of each updated feature map will double; finally, the feature map is finally fused using a convolution operation with a convolution kernel of 3x3 size to obtain the final feature map for prediction of the results of subsequent steps. The final fusion method is to double the number of channels of the updated feature map caused by the previous splicing and fusion, and perform inverse processing to restore the number of channels, that is, the true numerical fusion of the corresponding channels. The result of this step is also 4 feature maps, and their sizes are the same as in the previous step. Through the fusion operation, the information in the feature map can be enriched, making the model prediction results more accurate.

[0065] The detection architecture of the preliminary detection frame correction model in the aircraft detection model of this embodiment is a deep learning detection method based on the anchor frame class, so a priori anchor frames are generated first. Specifically, 9 priori anchor frames are generated at each point of the four feature maps obtained above to ensure the accuracy and recall rate of the detection results; wherein, the number N of the priori frames generated is based on the following formula:

[0066] N = ratios × scales

[0067] Among them, ratios represents the number of preset aspect ratios of generated anchor boxes, and scales represents the number of preset scales of generated anchor boxes. In this embodiment, ratios is set to 0.5, 1, and 2, and scales is set to 1, 1.26, and 1.59. The above values ​​are empirical values ​​obtained through a wide range of experiments. Therefore, N = 3×3 = 9.

[0068] The four different scale feature maps output by the multi-scale feature extraction network are used to correct the prior anchor frame generated by the prior anchor frame generation network, and the data result is the corrected anchor frame. This operation offsets the center point of the generated prior anchor frame and scales the width and height so that the generated prior anchor frame can locate the aircraft in the input image. The correction operation is the regression of the prior anchor frame, and its specific operation is as follows:

[0069] The intersection-and-union ratio of the generated prior anchor frame and the manually labeled anchor frame is calculated. When the intersection-and-union ratio is greater than the manually set threshold, the generated anchor frame is considered to be capable of regressing into the final detection result and is retained. The threshold is preferably set to 0.4; and the retained prior anchor frame is matched with the manually labeled anchor frame. When the intersection-and-union ratio of the same prior anchor frame and multiple manually labeled anchor frames is greater than the threshold, it is matched with the manually labeled anchor frame with the largest intersection-and-union ratio.

[0070] The specific formula for calculating the offset value δ between the generated prior anchor box and the manually labeled anchor box after retention is as follows:

[0071] δ cx =(g x -b x ) / b w

[0072] δ cy =(g y -b y ) / b h

[0073] δ w =log(g w / b w )

[0074] δ h =log(g h / b h )

[0075] Among them, g represents the location information of the manually labeled box, b represents the generated prior anchor box, and x, y, w, and h represent the horizontal and vertical coordinates of the center point, the width and height of the anchor box, respectively.

[0076] Calculate the final loss model regression loss L loc , the specific formula is as follows:

[0077]

[0078] L loc =smooth L1 (Δt-Δp)

[0079] Among them, Δt represents the offset value δ, and Δp is the offset value predicted by the model, that is, the parameter to be optimized that the preliminary detection frame correction model needs to learn.

[0080] Based on the above embodiment, in this method, the re-corrected classification model includes the multi-scale feature extraction network, the detection frame re-correction network and the classification network;

[0081] The detection frame correction network is used to optimize and correct the parameters of the preliminary detection frame based on the offset between the preliminary detection frame sample and the real object annotation frame;

[0082] The classification network is used to optimize the confidence parameters in the classification network based on the classification error between the probability that the predicted target object is an aircraft and the target aircraft annotation frame determined by the predicted detection frame output by the correction network again based on the detection frame.

[0083] Specifically, the four different scale feature maps obtained by the feature extraction method in the multi-scale feature extraction network are still used to further classify and regress the preliminary detection frame samples to obtain the final detection results. Specifically, the regression operation is consistent with the regression operation in the preliminary detection frame correction model, except that the intersection and union ratio of the generated prior anchor frame and the manually labeled anchor frame is calculated. When the intersection and union ratio is greater than another artificially set threshold, it is considered that the generated anchor frame has the ability to regress into the final detection result and is retained. The other set threshold is preferably 0.6, and the set threshold of the correction is raised. The purpose is to further correct the corrected anchor frame to obtain a more accurate aircraft target detection result. The purpose of classification is to perform binary classification on the fully corrected anchor frame obtained by re-correcting the classification model to determine whether the anchor frame contains an aircraft target. The classification loss function uses focal loss, and the specific steps are as follows:

[0084] The intersection-and-union ratio of the generated prior anchor box and the manually annotated anchor box is calculated. When the intersection-and-union ratio is greater than the manually set threshold of 0.6, the generated anchor box is considered to contain the target. When the intersection-and-union ratio is less than the manually set threshold of 0.4, the generated anchor box is considered to contain the target, and the prior anchor boxes generated between 0.4 and 0.6 are ignored.

[0085] Calculate the classification loss of the final loss model. The specific formula is as follows:

[0086]

[0087] Among them, α represents the category weight, γ represents the difficulty weight, both of which are hyperparameters set artificially according to the distribution of the training data set. In this embodiment, α=0.25, γ=2, target=1 represents that the generated anchor frame contains the aircraft target, target=0 represents that the generated anchor frame does not contain the aircraft target, and p is the confidence of the model prediction that the generated anchor frame contains the aircraft target, that is, the parameters to be adjusted that need to be learned by the classification network in the classification model are corrected again.

[0088] Based on the above embodiment, in the method, the construction of the first sample set specifically includes:

[0089] The collected sample optical remote sensing image is subjected to several rotation, horizontal and vertical flipping operations to obtain a preliminary expanded sample optical remote sensing image;

[0090] Performing mosaic data enhancement on the initially expanded sample optical remote sensing image to obtain a fully expanded sample optical remote sensing image;

[0091] Each image in the fully expanded sample optical remote sensing images is cropped to obtain a first sample set.

[0092] Specifically, this embodiment uses commonly used and currently more advanced data augmentation methods to increase the number of aircraft targets in remote sensing images, indirectly increase the sample size of network training, and improve network detection performance. The data augmentation steps used in this embodiment are as follows:

[0093] 1. Image rotation and flipping

[0094] The rotation operation of the image is one of the important ways to augment optical remote sensing images. Compared with general optical images, since it is an aerial view from a bird's-eye view, the target itself is a rotation target. Therefore, it is considered to be an effective and efficient data augmentation method in the field of remote sensing image processing and is widely used. In this embodiment, the original image is first rotated 90°, 180° and 270° respectively. Different from the rotation of the image, it refers to the operation of mirroring the image, which specifically includes horizontal flipping and vertical flipping. In this embodiment, all the image data obtained after the rotation are subjected to two operations, horizontal and vertical flipping, to achieve the purpose of expanding the sample size.

[0095] 2. Mosaic data enhancement

[0096] In order to further increase the sample size and improve the detection performance of the subsequent model, this embodiment also uses the target advanced mosaic enhancement technology on the data obtained after the image rotation and flipping processing steps, which is embedded in the training process of the model, as follows:

[0097] Randomly select 4 images from the augmented dataset obtained by the image rotation and flipping steps, and crop these images at random positions.

[0098] The four randomly cropped images are stitched together to obtain an image of the same size as in the dataset. During the model training process, the stitched image of the four images is used as the input data in the model training process.

[0099] After the first step of data augmentation, the mosaic enhancement technology is used to obtain images as subsequent input for the training data set of the aircraft detection model, so as to obtain a more accurate network model with detection capabilities.

[0100] Based on the above embodiment, in the method, the construction of the second sample set specifically includes:

[0101] Input the images in the first sample set into the trained preliminary detection frame correction model, and output a sample second image with a preliminary detection frame for each sample image;

[0102] A second sample set is constructed based on the sample second image.

[0103] Specifically, the images in the first sample set are input into the trained preliminary detection frame correction model, and the trained preliminary detection frame correction model outputs a sample second image with a preliminary detection frame for each sample image, and the second sample set is constructed with the sample second images.

[0104] Based on the above embodiment, in the method, the step of clipping the acquired original optical remote sensing image to obtain a plurality of slice images specifically includes:

[0105] Using overlapping sliding windows to crop the acquired original optical remote sensing images, a cropped image sequence is obtained in which there are overlapping pixels between the previous and next cropped images;

[0106] The images in the cropped image sequence are sequentially difference-upsampled to obtain a plurality of slice images.

[0107] Preferably, the original remote sensing image is cropped into slices of 600x600 size. Since the target may be cut off during the cropping process, an overlapping sliding window cropping method is used, where the overlapping size of the sliding window is 200 pixels. In view of the fact that the targets in the remote sensing image are usually small targets, we perform difference upsampling on them before sending them to the network, and upsample the 600x600 slices to 800x800 size.

[0108] Based on the above embodiment, in the method, mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain the final aircraft detection result specifically includes:

[0109] Determine a restoration mapping function based on the sampling rule of the difference upsampling;

[0110] Mapping the detection frame position information back to the coordinates of the original optical remote sensing image based on the restoration mapping function to obtain target result coordinates;

[0111] The overlapping data in the target result coordinates are merged using a non-maximum suppression rule to obtain the final aircraft coordinate detection result.

[0112] Specifically, the detection result coordinates of the optical remote sensing image slice are mapped back to the original image coordinates. Since the present invention uses an overlapping sliding window method to detect the original image, repeated detection will occur in the overlapping image part. Non-maximum suppression is used to merge the overlapping results to obtain the final detection result. The specific implementation process is as follows.

[0113] Assume that there are 4 directional anchor boxes in the detection results of a remote sensing image, and they are sorted from large to small according to the confidence of the classification results. The specific steps are as follows:

[0114] (1) Starting from the maximum probability oriented anchor box A, determine whether the intersection-over-union ratios of A and B, A and C, and A and D are greater than a manually set threshold (set to 0.4 in the present invention).

[0115] (2) Assuming that the intersection-over-intersection ratio of A and B, and A and C is greater than the set threshold, the detection results of B and C are discarded, and the detection result of A is retained.

[0116] (3) If the remaining A and C do not meet the set threshold, it is considered that the detection result of C and the detection result of A are not located at the same target to be detected, so the detection result of C is retained.

[0117] (4) Repeat steps (1), (2) and (3) to find all retained oriented anchor boxes as the final detection results.

[0118] Based on the above embodiments, the present invention further provides an optical remote sensing image aircraft detection method based on a cascaded regression correction module. Figure 2 The flowchart of the optical remote sensing image aircraft detection method based on the cascade regression correction module provided by the present invention is as follows: Figure 2As shown, the collected original remote sensing image is input as training data into the aircraft data augmentation module of the optical remote sensing image, and the image is rotated and flipped, and then processed by mosaic data enhancement, and then input into the detection model based on the cascade regression correction module as direct training data that can be directly used for detection model training, and the training steps of feature extraction, prior frame generation, prior frame correction, final position confirmation (secondary correction confirmation based on the preliminary correction frame) and result screening are successively performed to train the model with oil detection capability. When using this model, the large-size original remote sensing image to be detected is first cropped based on the sliding window to obtain image slices, and each image slice is input into the trained detection model, and then the result prediction based on the detection model is mapped back to the coordinate system of the original remote sensing image, and the final detection result is output after the result merging based on non-maximum suppression. This method can achieve the following three beneficial effects:

[0119] (1) The sample size of remote sensing image data sets is very limited, resulting in poor training effect of network models. The present invention proposes an augmentation method for optical remote sensing image data samples, which fully increases the sample size of training data to improve the accuracy of the detection model after training.

[0120] (2) Aiming at the problems of complex background, many false alarm interference factors and large scale difference between aircraft targets in remote sensing images with large field of view, the present invention constructs a detection model structure based on a cascade regression correction module of deep learning. Specifically, the problem of large scale difference between aircraft targets is solved by fusing multiple feature maps of different sizes in the feature extraction stage and jointly participating in the prediction of the results. Aiming at the characteristics of complex background and many false alarm interference factors in remote sensing images with large field of view, a cascade regression correction module is designed. Compared with the one-time regression of the general method, this method corrects the generated prior anchor frame and then obtains the final detection result through regression. This two-time regression method obtains more accurate aircraft detection results.

[0121] (3) In the target detection stage, the present invention first slices the original large-size image using a sliding window, then performs an upsampling operation to enlarge the aircraft target in the image, and then sends it to the trained model for detection. This operation avoids the problem of reducing the size of the aircraft target and reducing the detection accuracy when reducing the large-size image during general detection. At the same time, an upsampling step is added to the slice to enlarge the aircraft target and improve the detection accuracy. Finally, after mapping the detection results of the slice back to the original image, non-maximum suppression is used to remove duplicate detection results.

[0122] The following is a description of the optical remote sensing image aircraft detection device based on cascade type regression correction provided by the present invention. The optical remote sensing image aircraft detection device based on cascade type regression correction described below and the optical remote sensing image aircraft detection method based on cascade type regression correction described above can be referenced to each other.

[0123] Figure 3 The schematic diagram of the structure of the optical remote sensing image aircraft detection device based on cascade regression correction provided by the present invention is as follows: Figure 3 As shown, the optical remote sensing image aircraft detection device based on cascade regression correction includes a clipping unit 310, a detection unit 320 and a splicing unit 330, wherein:

[0124] The clipping unit 310 is used to clip the collected original optical remote sensing image to obtain multiple slice images;

[0125] The detection unit 320 is used to sequentially input the plurality of slice images into an aircraft detection model, and output the detection frame position information of the target aircraft detected in each slice;

[0126] The aircraft detection model is formed by cascading a preliminary detection frame correction model and a secondary correction classification model, wherein the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, and the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame, and a classification label of whether the target object is an aircraft, wherein the sample first image comes from a first sample set constructed based on a cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame;

[0127] The stitching unit 330 is used to map the detection frame position information back to the coordinates of the original optical remote sensing image and merge them to obtain a final aircraft detection result.

[0128] The optical remote sensing image aircraft detection device based on cascade regression correction provided by the present invention, because the aircraft detection model provided by the present invention is based on the cascade of the preliminary detection frame correction model and the secondary correction classification model, by designing a cascade regression correction module, compared with the single regression in the traditional method, the cascade regression aircraft detection model corrects the generated prior anchor frame, and then continues to perform secondary correction through regression based on the previous preliminary correction result to obtain the final aircraft target detection result. The two regression correction methods can obtain more accurate aircraft detection results, and the network structure complexity of the aircraft detection model is not high. Therefore, the device provided by the present invention realizes efficient detection of target aircraft in remote sensing images, and the robustness and adaptability of the detection results are improved.

[0129] On the basis of the above-mentioned embodiment, in the optical remote sensing image aircraft detection device based on cascade regression correction, the network structure of the preliminary detection frame correction model training includes a multi-scale feature extraction network, a priori anchor frame generation network and a detection frame correction network connected in sequence;

[0130] The multi-scale feature extraction network is used to downsample any input image to obtain N size feature maps of the image, and fuse the N size feature maps according to a preset rule to obtain N fused feature maps of the image;

[0131] The prior anchor frame generation network is used to determine the number of prior anchor frames generated at each point of N fusion feature maps based on the number of preset prior anchor frame aspect ratios and the number of prior anchor frame scales;

[0132] The detection frame correction network is used to optimize and correct the prior frame parameters based on the offset between the prior anchor frame and the real target object annotation frame.

[0133] On the basis of the above-mentioned embodiment, in the optical remote sensing image aircraft detection device based on cascade regression correction, the re-corrected classification model includes the multi-scale feature extraction network, the detection frame re-correction network and the classification network;

[0134] The detection frame correction network is used to optimize and correct the parameters of the preliminary detection frame based on the offset between the preliminary detection frame sample and the real object annotation frame;

[0135] The classification network is used to optimize the confidence parameters in the classification network based on the classification error between the probability that the predicted target object is an aircraft and the target aircraft annotation frame determined by the predicted detection frame output by the correction network again based on the detection frame.

[0136] On the basis of the above embodiment, in the optical remote sensing image aircraft detection device based on cascade regression correction, the construction of the first sample set specifically includes:

[0137] The collected sample optical remote sensing image is subjected to several rotation, horizontal and vertical flipping operations to obtain a preliminary expanded sample optical remote sensing image;

[0138] Performing mosaic data enhancement on the initially expanded sample optical remote sensing image to obtain a fully expanded sample optical remote sensing image;

[0139] Each image in the fully expanded sample optical remote sensing images is cropped to obtain a first sample set.

[0140] On the basis of the above embodiment, in the optical remote sensing image aircraft detection device based on cascade regression correction, the construction of the second sample set specifically includes:

[0141] Input the images in the first sample set into the trained preliminary detection frame correction model, and output a sample second image with a preliminary detection frame for each sample image;

[0142] A second sample set is constructed based on the second sample image.

[0143] On the basis of the above-mentioned embodiment, in the optical remote sensing image aircraft detection device based on cascade regression correction, the collected original optical remote sensing image is clipped to obtain a plurality of slice images, specifically including:

[0144] Using overlapping sliding windows to crop the acquired original optical remote sensing images, a cropped image sequence is obtained in which there are overlapping pixels between the previous and next cropped images;

[0145] The images in the cropped image sequence are sequentially difference-upsampled to obtain a plurality of slice images.

[0146] On the basis of the above-mentioned embodiment, in the optical remote sensing image aircraft detection device based on cascade regression correction, mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain the final aircraft detection result specifically includes:

[0147] Determine a restoration mapping function based on the sampling rule of the difference upsampling;

[0148] Mapping the detection frame position information back to the coordinates of the original optical remote sensing image based on the restoration mapping function to obtain target result coordinates;

[0149] The overlapping data in the target result coordinates are merged using a non-maximum suppression rule to obtain the final aircraft coordinate detection result.

[0150] Figure 4 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute the optical remote sensing image aircraft detection method based on cascade regression correction, the method comprising: cropping the collected original optical remote sensing image to obtain multiple slice images; inputting the multiple slice images into the aircraft detection model in sequence, and outputting the detection frame position information of the target aircraft detected in each slice; wherein the aircraft detection model is cascaded based on a preliminary detection frame correction model and a secondary correction classification model, the preliminary detection frame correction model is obtained after training based on the sample first image and the corresponding target object annotation frame, the secondary correction classification model is obtained after training based on the sample second image, the corresponding target object annotation frame and whether the target object is an aircraft classification label, the sample first image comes from a first sample set constructed based on the cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with the preliminary detection frame; the detection frame position information is mapped back to the coordinates of the original optical remote sensing image and merged to obtain the final aircraft detection result.

[0151] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0152] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the optical remote sensing image aircraft detection method based on cascade regression correction provided by the above methods, the method comprising: cropping the collected original optical remote sensing image to obtain multiple slice images; inputting the multiple slice images into an aircraft detection model in sequence, and outputting the detection box position information of the target aircraft detected in each slice; wherein the aircraft detection model is cascaded based on a preliminary detection box correction model and a secondary correction classification model, the preliminary detection box correction model is obtained after training based on a sample first image and a corresponding target object annotation box, the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation box and whether the target object is an aircraft classification label, the sample first image comes from a first sample set constructed based on the cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection box; the detection box position information is mapped back to the coordinates of the original optical remote sensing image and merged to obtain the final aircraft detection result.

[0153] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the optical remote sensing image aircraft detection method based on cascade regression correction provided by the above methods, the method comprising: cropping the collected original optical remote sensing image to obtain multiple slice images; inputting the multiple slice images into an aircraft detection model in sequence, and outputting the detection frame position information of the target aircraft detected in each slice; wherein the aircraft detection model is cascaded based on a preliminary detection frame correction model and a secondary correction classification model, the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame and whether the target object is an aircraft classification label, the sample first image comes from a first sample set constructed based on the cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame; the detection frame position information is mapped back to the coordinates of the original optical remote sensing image and merged to obtain a final aircraft detection result.

[0154] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0155] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting aircraft in optical remote sensing images based on cascade regression correction, characterized in that: include: The collected original optical remote sensing image is trimmed to obtain a plurality of slice images; Input the plurality of slice images into an aircraft detection model in sequence, and output detection frame position information of a target aircraft detected in each slice; The aircraft detection model is formed by cascading a preliminary detection frame correction model and a secondary correction classification model, wherein the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, and the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame, and a classification label of whether the target object is an aircraft, wherein the sample first image comes from a first sample set constructed based on a cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame; Mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain a final aircraft detection result; The re-corrected classification model includes a multi-scale feature extraction network, a detection frame re-correction network and a classification network; The detection frame correction network is used to optimize and correct the parameters of the preliminary detection frame based on the offset between the preliminary detection frame sample and the real object annotation frame; The classification network is used to optimize the confidence parameter in the classification network based on the classification error between the probability that the predicted target object is an aircraft and the target aircraft labeling frame determined by the predicted detection frame output by the correction network again based on the detection frame; The method of trimming the collected original optical remote sensing image to obtain a plurality of slice images specifically includes: Using overlapping sliding windows to crop the acquired original optical remote sensing images, a cropped image sequence is obtained in which there are overlapping pixels between the previous and next cropped images; Performing difference upsampling on the images in the cropped image sequence in sequence to obtain a plurality of slice images; Mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain the final aircraft detection result specifically includes: Determine a restoration mapping function based on the sampling rule of the difference upsampling; Mapping the detection frame position information back to the coordinates of the original optical remote sensing image based on the restoration mapping function to obtain target result coordinates; The overlapping data in the target result coordinates are merged using a non-maximum suppression rule to obtain the final aircraft coordinate detection result.

2. The method for detecting aircraft in optical remote sensing images based on cascade regression correction according to claim 1, characterized in that: The network structure of the preliminary detection frame correction model training includes a multi-scale feature extraction network, a priori anchor frame generation network and a detection frame correction network connected in sequence; The multi-scale feature extraction network is used to downsample any input image to obtain N size feature maps of the image, and fuse the N size feature maps according to a preset rule to obtain N fused feature maps of the image; The prior anchor frame generation network is used to determine the number of prior anchor frames generated at each point of N fusion feature maps based on the number of preset prior anchor frame aspect ratios and the number of prior anchor frame scales; The detection frame correction network is used to optimize and correct the prior frame parameters based on the offset between the prior anchor frame and the real target object annotation frame.

3. The method for detecting aircraft in optical remote sensing images based on cascade regression correction according to claim 1, characterized in that: The construction of the first sample set specifically includes: The collected sample optical remote sensing image is subjected to several rotation, horizontal and vertical flipping operations to obtain a preliminary expanded sample optical remote sensing image; Performing mosaic data enhancement on the initially expanded sample optical remote sensing image to obtain a fully expanded sample optical remote sensing image; Each image in the fully expanded sample optical remote sensing images is cropped to obtain a first sample set.

4. The method for detecting aircraft in optical remote sensing images based on cascade regression correction according to claim 3, characterized in that: The construction of the second sample set specifically includes: Input the images in the first sample set into the trained preliminary detection frame correction model, and output a sample second image with a preliminary detection frame for each sample image; A second sample set is constructed based on the sample second image.

5. An optical remote sensing image aircraft detection device based on cascade regression correction, characterized in that: include: A cropping unit, used for cropping the collected original optical remote sensing image to obtain a plurality of slice images; A detection unit, used to sequentially input the plurality of slice images into an aircraft detection model, and output detection frame position information of a target aircraft detected in each slice; The aircraft detection model is formed by cascading a preliminary detection frame correction model and a secondary correction classification model, wherein the preliminary detection frame correction model is obtained after training based on a sample first image and a corresponding target object annotation frame, and the secondary correction classification model is obtained after training based on a sample second image, a corresponding target object annotation frame, and a classification label of whether the target object is an aircraft, wherein the sample first image comes from a first sample set constructed based on a cropped optical remote sensing image, and the sample second image comes from a second sample set constructed based on the sample first image with a preliminary detection frame; A stitching unit, used for mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain a final aircraft detection result; The re-corrected classification model includes a multi-scale feature extraction network, a detection frame re-correction network and a classification network; The detection frame correction network is used to optimize and correct the parameters of the preliminary detection frame based on the offset between the preliminary detection frame sample and the real object annotation frame; The classification network is used to optimize the confidence parameter in the classification network based on the classification error between the probability that the predicted target object is an aircraft and the target aircraft labeling frame determined by the predicted detection frame output by the correction network again based on the detection frame; The method of trimming the collected original optical remote sensing image to obtain a plurality of slice images specifically includes: Using overlapping sliding windows to crop the acquired original optical remote sensing images, a cropped image sequence is obtained in which there are overlapping pixels between the previous and next cropped images; Performing difference upsampling on the images in the cropped image sequence in sequence to obtain a plurality of slice images; Mapping the detection frame position information back to the coordinates of the original optical remote sensing image and merging them to obtain the final aircraft detection result specifically includes: Determine a restoration mapping function based on the sampling rule of the difference upsampling; Mapping the detection frame position information back to the coordinates of the original optical remote sensing image based on the restoration mapping function to obtain target result coordinates; The overlapping data in the target result coordinates are merged using a non-maximum suppression rule to obtain the final aircraft coordinate detection result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the optical remote sensing image aircraft detection method based on cascade regression correction as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the optical remote sensing image aircraft detection method based on cascade regression correction as described in any one of claims 1 to 4 are implemented.

Images