Multi-class object detection method for optical remote sensing image based on cross-scale feature fusion

Abstract

The application relates to a kind of optical remote sensing image multi-class target detection methods based on cross-scale feature fusion, to train data as the input of convolutional neural network to extract image features, obtain multi-scale feature map from the output of different convolutional layers, add extrusion-stimulating module at the topmost layer feature to remodel the channel information of top layer feature;Then cross-scale feature fusion operation is carried out to the obtained feature map, then the multi-scale feature map is trained on regional suggestion network, the suggestion frame for subsequent task is obtained from the trained regional suggestion network, and is sent to classification network and regression network for training;Finally, accurate detection of optical remote sensing image multi-class target on multi-scale feature map is realized through post-processing operations such as non-maximum suppression. Using the method of the application, multiple types of targets can be detected from optical remote sensing images in complex background. The application has high detection and recognition accuracy and fast speed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-class target detection method based on optical remote sensing image, and relates to a multi-class target detection method based on cross-scale feature fusion of optical remote sensing image, which realizes cross-scale fusion of features and can be applied to multi-type target detection tasks of complex background optical remote sensing image. BACKGROUND

[0002] With the rapid development of aerial remote sensing technology, it has become increasingly easy to obtain a large amount of remote sensing data from high altitudes. At the same time, various tasks based on remote sensing images are emerging, such as target detection, scene classification, data compression, etc. As an application of remote sensing image processing technology, target detection of complex background optical remote sensing image is a key technology in the field of image processing, and has always been a research hotspot and difficult problem in the field, and has attracted more and more attention. The core task of target detection in optical remote sensing image is to find all the targets of interest in the image and determine their specific categories, positions and sizes.

[0003] At present, the implementation of the target detection task mainly adopts the following two types of methods: one type is a single-stage target detection method, and a representative work is “You Only Look Once: Unified, Real-Time Object Detection” published by Ross Girshick et al. in IEEE Conference on Computer Vision and Pattern Recognition 2016. This method regards the target detection task as a regression task. The advantage of this method is fast detection speed, and the disadvantage is that the detection accuracy is lower than that of the two-stage target detection method; the other type is a two-stage target detection method. This method first generates a series of anchor frames in the image. The length, width, proportion and number of these anchor frames can be set according to the task requirements. The region proposal network (RPN) is used to solve the two-classification problem of whether the target in the anchor point is foreground or background, and a rough regression is performed on the original anchor frame coordinates, and then the classification and regression tasks are performed. The advantage of this method is high detection accuracy, and the disadvantage is slow speed and slightly longer time spent in task inference process. However, there are great differences between remote sensing images and natural scene images. Due to the differences in imaging platform and imaging method, the targets in optical remote sensing images have different degrees of deformation, occlusion, scale change and direction diversity. For those small-scale targets, their feature information is often overwhelmed by the complex surrounding scene. Some categories of targets are too densely arranged, and some categories of targets have high similarity in color and appearance shape. These problems increase the difficulty of optical remote sensing image target detection task. SUMMARY

[0004] Technical problems to be solved

[0005] In order to avoid the shortcomings of the prior art, the present application proposes an optical remote sensing image multi-class target detection method based on cross-scale feature fusion, which improves the existing target detection method to solve the problem of large difference between the inter-class similarity and the target scale of the optical remote sensing image target.

[0006] Technical solutions

[0007] An optical remote sensing image multi-class target detection method based on cross-scale feature fusion, characterized by the following steps:

[0008] Step 1, data preprocessing: calculating the mean R ave ,G ave ,B ave and standard deviation R var ,G var ,B var of the RGB components of each picture in the experimental database, and adjusting the picture length and width to MxM;

[0009] Step 2, feature extraction: taking the processed picture data as the input of the feature pyramid network model, obtaining five scale feature maps {F A1 ,F A2 ,F A3 ,F A4 ,F A5} from different convolution layers of the feature extraction network, and keeping the channel number of the five groups of feature maps consistent, and their length and width are where is the rounding up operation;

[0010] Step 3, adding SE block squeeze-excitation module: for the multi-scale feature maps {F A1 ,F A2 ,F A3 ,F A4 ,F A5}, adding SE block squeeze-excitation module in the form of residual connection on the topmost feature map, realizing the re-calibration of the feature channel weight, obtaining a new group of feature maps, then adding a group of 1*1 convolution on each scale of the new group of feature maps, realizing the reduction of the feature channel number, obtaining multi-scale feature maps {F B1 ,F B2 ,F B3 ,F B4 ,F B5}, and keeping their feature channel numbers consistent;

[0011] Step 4, cross-scale feature fusion: in the multi-scale feature maps {FB1 ,F B2 ,F B3 ,F B4 ,F B5} Select any feature map at one scale and perform cross-scale feature fusion with feature maps at the other four scales to obtain another set of multi-scale feature maps {F C1 ,F C2 ,F C3 ,F C4 ,F C5};

[0012] Step 5: Generate Regions of Interest (RoIs) using a Region Proposal Network (RPN): In the multi-scale feature map {F... C1 ,F C2 ,F C3 ,F C4 ,F C5 In this study, the RPN method is used to set anchor points. Based on the Intersection over Union (IoU) between the anchor point and the ground truth, the label of each anchor point is determined, including positive samples, negative samples, and ignored samples. Anchor points that meet the conditions are selected to train the Region Proposal Network.

[0013] Step 6: Perform end-to-end training of the network: After the region proposal network is trained, it will output the score of each anchor point belonging to the foreground or background based on the learned network parameters. Then, the K anchor points with the higher scores are selected as regions of interest and provided to the Faster R-CNN part. Finally, M candidate boxes are selected to train the network parameters of this part. The classification task uses the Cross Entropy Loss function, and the regression task uses the SmoothL1 loss function.

[0014] Step 7, Object Detection: Use the trained network model to predict the category and location of objects in the test set, and use the non-maximum suppression (NMS) method to filter out redundant detection boxes, and give the final detection result for each image.

[0015] The RPN method is described in the article S. Ren, R. Girshick, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 2017.

[0016] The Faster R-CNN method can be found in the article S.Ren, R.Girshick, R.Girshick, and J.Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region ProposalNetworks," IEEE Transactions on Pattern Analysis&Machine Intelligence, vol.39, no.6, pp.1137-1149, 2017.7.

[0017] Beneficial effects

[0018] This invention proposes a method for multi-target detection in optical remote sensing images based on cross-scale feature fusion. First, the training data is initialized and used as input to a convolutional neural network to extract image features. Multi-scale feature maps are obtained from the outputs of different convolutional layers. A squeezing-excitation module is added to the top-level feature map to remodel the channel information of the top-level feature map. Then, cross-scale feature fusion is performed on the obtained feature maps. Next, a region proposal network is trained on these multi-scale feature maps. Proposal boxes for subsequent tasks are obtained from the trained region proposal network. Based on this, the proposal boxes are matched with the multi-scale features. These matched features are then fed into classification and regression networks for training. Finally, post-processing operations such as Non-Maximum Suppression (NMS) are performed to achieve accurate detection of multiple targets in optical remote sensing images on the multi-scale feature maps.

[0019] This invention can achieve high accuracy and recall in remote sensing image databases. Practical experience has shown that this method has high detection precision and good robustness. Attached Figure Description

[0020] Figure 1 : Basic flowchart of the method of the present invention

[0021] Figure 2 Example of a portion of the database images used in the method of this invention.

[0022] Figure 3 : Schematic diagram of the implementation of the cross-scale feature fusion module used in the method of this invention

[0023] Figure 4 Schematic diagram of the extrusion-excitation module used in the method of this invention.

[0024] Figure 5 Example of detection results of the method of the present invention Detailed Implementation

[0025] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0026] The hardware environment used for implementation was an Intel(R) Core(TM) i3-8100 CPU computer with 8.0GB of memory. The software environment running was PyCharm 2016 and Ubuntu 16.04.5LTS. This experiment used the publicly available DIOR Dataset, which contains 23,463 images with 192,472 instances across 20 categories. Each image is 800×800 pixels. To verify the effectiveness of the proposed scheme, 11,725 ​​images were selected from the dataset for the training phase, and the remaining 11,738 images were used as the test set.

[0027] The specific implementation of this invention is as follows:

[0028] 1. Data Preprocessing: For the DIOR remote sensing image dataset used in the experiment, the mean and standard deviation of the RGB three channels of the 11725 images used for training were statistically analyzed. They are respectively R ave G ave B ave and R std G std B std The following uses R ave R std Taking the calculation method as an example, the specific calculation formula is explained as follows:

[0029]

[0030]

[0031] Where N = 11725, M = 800, R ij R represents the R component value at the current position (i,j) in an image. ave R var These represent the mean and standard deviation of the R component, respectively. The mean and variance of the G and B components are calculated using the same method. The image data initialization method is as follows:

[0032]

[0033] mean=(R ave G ave B ave )

[0034] std = (R std G std B std )

[0035] img ori Represents raw, unprocessed image data, img after This represents the image data result obtained by subtracting the mean from each pixel in the image and then dividing by the variance. The mean represents the mean of the RGB components of the dataset, and the std represents the variance of the RGB components of the dataset.

[0036] 2. Feature Extraction: The processed image data is used as input to the detection network. ResNet-101 is used as the feature extraction backbone network. Four sets of feature maps with decreasing width and height and increasing channel number are output from conv2_3, conv3_4, conv4_23, and conv5_3. These are then processed by a Feature Pyramid Network (FPN) to obtain a new set of feature maps {F}. A1 ,F A2 ,F A3 ,F A4}, for F A4 Performing max pooling with stride=2 yields the top-level feature map F. A5 This yields feature maps at five scales, with dimensions {200*200, 100*100, 50*50, 25*25, 13*13}, and each feature map has 256 feature channels.

[0037] The Resnet-101 network structure described above can be found in the article K. He, X. Zhang, S. Ren, and J. Sun, "Deepresidual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.

[0038] The FPN method described above can be found in the article T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117-2125.

[0039] 3. Add SE block: In step 2 above, the feature map {F} is obtained. A1 ,F A2 ,F A3 ,F A4 ,F A5}, in the top-level feature map {F A5 At point}, an SE block is added using residual connections to obtain a new feature map {F'}. A5 This allows for the recalibration of the channel weights in the feature map {F}. A1 ,F A2 ,F A3 ,F A4 ,F' A5 Adding a set of 1*1 convolutions to each of the layers reduces the number of feature channels, resulting in a new multi-scale feature map {F}. B1 ,F B2 ,F B3 ,F B4 ,F B5 Their feature channels are all 64.

[0040] The SE blocking method described above can be found in the article J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132-7141.

[0041] 4. Cross-scale feature fusion: using the final output F C3 Taking layer feature information as an example, in order to ensure that the length, width, and number of feature channels of the feature map match during the feature fusion process, the following operations are performed in sequence: For layers with length and width of... F with 64 feature channels B1 Perform a 4x downsampling; for dimensions of [missing information] F with 64 feature channels B2 Perform 2x downsampling; for dimensions of... F with 64 feature channels B4 Perform 2x upsampling; for dimensions of [missing information] F with 64 feature channels B5 After performing a 4x upsampling, four groups with length and width of [missing information] were obtained. The feature map is concatenated by calling the torch.cat() function to produce a feature map with dimensions of [length and width]. A new feature map with 256 feature channels This feature map integrates semantic feature information from all layers except the current one. F A3 The length and width are The original feature maps with 256 feature channels are then fused using an element-wise addition method. and F A3 The feature information is then processed, and a set of 3*3 convolution kernels are added to the feature map at each scale to eliminate channel aliasing, resulting in the feature map F used for the final detection task. C3 The output feature maps of the remaining layers can then undergo the corresponding feature fusion operations according to the rules described above.

[0042] 5. Generate regions of interest using a region proposal network: In the multi-scale feature map {F} obtained above C1 ,F C2 ,F C3 ,F C4 ,F C5 In this study, a Region Proposal Network (RPN) method is used to map each point onto the original image to generate corresponding anchor points. When the IoU between the anchor point and the ground truth is greater than 0.7, the anchor point is marked as a positive example; when the IoU is less than 0.3, the anchor point is marked as a negative example. Boxes outside the image boundaries and instances with IoU between 0.3 and 0.7 are marked as ignored examples and not included in the training process. Three anchor point ratios are set on the feature maps at each scale, with ratios of {1:2, 1:1, 2:1}. 256 anchor points meeting the criteria are selected for training the RPN, with a 1:1 sampling ratio for positive and negative examples. 128 anchor points are randomly selected from both positive and negative examples.

[0043] The RPN method described above can be found in the article S. Ren, R. Girshick, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 2017.

[0044] 6. End-to-end training of the network: The training is set for a total of 12 epochs, with an initial learning rate of 0.01. The learning rate begins to decrease at epochs 8-11 at a rate of 0.1, and reaches 0.0001 at the 12th epoch. The trained region suggestion network generates corresponding candidate boxes. The 2000 candidate boxes with the highest scores are selected as regions of interest and provided to the Faster R-CNN part. Finally, 512 candidate boxes are selected to train the network parameters for this part, with a positive to negative sample ratio of 1:3. The network parameters are optimized based on the classification loss function and the regression loss function, with the objective loss function defined as:

[0045]

[0046] L cls =-log[p i * p i +(1-p i * (1-p) i )]

[0047]

[0048] Where i is the index of the i-th anchor point in the mini-batch, and N is the normalized value of the classification item. cls p is the size of the mini-batch. i p represents the predicted target class probability. i * It is the ground-truth label information, and its value is an indicator function. Normalized value N of the regression term reg The number of anchor points determines the weighting of the two loss functions, which are calculated by a balancing parameter λ, here set to λ = 10. Only positive samples contribute to the regression loss calculation. i t represents the offset predicted by the detection network. i * The offset between the ground-truth coordinates and the anchor point coordinates, where x = t i -t i * .

[0049] The Faster R-CNN method described above can be found in the article S. Ren, R. Girshick, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 2017.7.

[0050] Object detection: The trained network generates a category score and bounding box for each input image. A score threshold of 0.3 is set, and NMS = 0.5 is used to filter out redundant detection boxes generated for the same object, yielding the final detection result. mAP is used to evaluate the effectiveness of this invention, where mAP is defined as:

[0051]

[0052] N represents the total number of target categories in the dataset, i represents one of the target categories, and AP i The average accuracy value represents the target type. The detection results obtained by this invention were compared with the target detection algorithm based on Faster R-CNN+FPN. The comparison results are shown in Table 1. The target detection accuracy demonstrates the effectiveness of the method of this invention.

[0053] Table 1 Evaluation of Test Results

[0054]

Claims

1. A multi-class target detection method for optical remote sensing images based on cross-scale feature fusion, characterized in that, The steps are as follows: Step 1, data pre-processing: calculate the mean of RGB components of each picture in the experimental database , , and standard deviation , , , and adjust the length and width of the picture to ; Step 2, feature extraction: the processed picture data is taken as the input of the feature pyramid network model, and five scale feature maps are obtained from different convolution layers of the feature extraction network The channel numbers of the five groups of feature maps are consistent, and the lengths and widths thereof are , wherein is a rounding up operation, and the feature channel numbers thereof are all 256; Step 3, add SE block squeeze-activation module: multi-scale feature map Add SE block squeeze-activation module in a residual connection manner on the topmost feature map, realize the re-calibration of feature channel weight, get a new group of feature maps, then add a group of Convolution on each scale of the new group of feature maps respectively, realize the reduction of feature channel number, get multi-scale feature map , their feature channel numbers remain the same; Step 4, cross-scale feature fusion: selecting a feature map of any one scale from the multi-scale feature maps to respectively perform cross-scale feature fusion with the feature maps of the other four scales to obtain another group of multi-scale feature maps ; Step 5, generating regions of interest RoIs with a region proposal network: in the multi-scale feature map In the above, the RPN method is used to set anchor points, and the label of each anchor point is determined according to the intersection over union IoU between the anchor point and the ground truth, including positive examples, negative examples and ignore examples, and the anchor points meeting the conditions are selected to train the region proposal network; Step 6, end-to-end training of the network: after the regional suggestion network is trained, the learned network parameters are output to determine the foreground and background scores of each anchor point, then the K anchor points with higher scores are selected as the regions of interest to provide to the Faster R-CNN part, and finally H candidate boxes are selected to train the network parameters; the classification task adopts a cross entropy loss Cross Entropy Loss function, and the regression task adopts a SmoothL1 loss function; Step 7, target detection: using the trained network model to predict the class and position of the target in the test set, using the non-maximum suppression NMS method to filter out redundant detection boxes, and giving the final detection result of each picture; In step 2, the feature pyramid network model, that is, the feature extraction network, takes Resnet-101 as the backbone of the feature extraction network, and outputs four groups of feature maps with decreasing length and width and increasing channel number from conv2_3, conv3_4, conv4_23 and conv5_3, and obtains a new group of feature maps through the feature pyramid network , and performing max-pooling operation with stride=2 to obtain the topmost feature map ; In step 3, the SE block is added in a residual connection manner to the topmost feature map , to obtain a new feature map , to realize the re-calibration of the channel weights of the layer, and a group of convolutions are respectively added on the feature map to realize the reduction of the number of feature channels, to obtain a new multi-scale feature map ; In step 4, was obtained by the following steps: For a long and wide , the characteristic channel number is 64 4 times down-sampling, for a long and wide , the characteristic channel number is 64 2 times down-sampling, for a long and wide , the characteristic channel number is 64 2 times up-sampling, for a long and wide , the characteristic channel number is 64 4 times up-sampling, thereby obtaining 4 groups of feature maps with a long and wide ​ The channel splicing of the feature map is implemented by calling the torch.cat function, and a new feature map with a length and width of and 256 feature channels , fuses semantic feature information of all layers except the current layer is long and wide , the original feature map with the characteristic channel number of 256, using element-wise addition method to fuse and feature information, and then add a group of convolution kernel to eliminate channel aliasing effect, get the feature map for the final detection task .

Images