A feature weaving method for remote sensing image target detection
By using deep residual network and feature pyramid network in remote sensing image object detection, the problem of low detection accuracy of multi-scale object in the prior art is solved, and more efficient recognition of multi-scale object is achieved, and the remote sensing image is improved Detection performance.
Patent Information
- Application Number
- CN202110329215.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-03-27
AI Technical Summary
The existing remote sensing image object detection algorithm has low multi-scale object detection accuracy and poor generalization ability in complex scenarios, and has not fully considered the overall enhancement of multi-scale features, resulting in limited improvement in effect.
The deep residual network ResNet 101 is used to extract image features, combine feature pyramid network (FPN) for feature weaving, fuse high- and low-level features, generate areas of interest, and classify them through feature pooling and full-connection layers to improve multi-scale Target detection efficiency.
Through the connection method of feature braiding, the recognition accuracy of small targets and large targets are enhanced, and the target detection performance in remote sensing images is generally improved.
Smart Images

Figure CN113111740B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to an image feature knitting method. Background Art
[0002] Remote sensing imaging techniques, such as optical or hyperspectral aerial image processing, have quickly become one of the most important techniques in image processing, especially in object detection. Object detection in remote sensing images is widely used in smart city management, environmental monitoring, search and rescue, and military fields. However, since remote sensing images are all taken from a high altitude "bird's eye view" perspective, the height and attitude of the sensor change when the image is collected. Compared with natural images, remote sensing images have the following main differences:
[0003] Objects of interest, such as ships, cars, etc., are usually densely packed and may appear as only a few pixels.
[0004] Remote sensing targets have arbitrary directions and different scales. For example, the size of the airplanes at an airport is not uniform and the directions are completely random.
[0005] Images often have changing lighting and complex backgrounds.
[0006] Due to the above problems, although some algorithms have been proposed for remote sensing image target detection, the results are not optimal and there is a lot of room for performance improvement. Therefore, remote sensing image target detection still has great research value.
[0007] Yao Qunli (Research on Remote Sensing Target Detection Based on Multi-scale Convolutional Neural Network, Acta Optica Sinica, 2019, 39(11): 346-353) proposed a multi-scale convolutional neural network remote sensing target detection framework to address the problems of low accuracy and poor generalization ability of existing remote sensing image target detection algorithms for multi-scale targets in complex scenes. This method enhances the network's ability to extract target features by constructing a deep feature pyramid; at the same time, it uses focused classification loss to enhance the network's learning ability for difficult samples. However, this method is relatively simple in constructing the pyramid and does not take into account the overall enhancement of multi-scale features, so the improvement effect is relatively limited. Summary of the invention
[0008] In order to overcome the shortcomings of the prior art, the present invention provides a feature weaving method for remote sensing image target detection. First, the deep residual network ResNet 101 is used to extract the features of the image, and the feature pyramid network (FPN) is used to further extract the features and generate the region of interest, wherein the high-level and low-level features are woven in the connection of the feature pyramid network, and the multi-scale features are integrated to improve the detection efficiency of multi-scale targets. After the candidate region is generated, the feature pooling is used to align the features, and the fully connected layer is used to classify the features to generate the target category and the border. The present invention uses the feature weaving connection method to replace the lateral connection of the feature pyramid, fully considers the multi-scale feature information, and increases the recognition accuracy of small targets by integrating features of different scales. The recognition rate of large targets is also optimized to a certain extent, thereby improving the overall target detection performance in remote sensing images.
[0009] The technical solution adopted by the present invention to solve the technical problem includes the following steps:
[0010] Step 1: Use a public image dataset; crop the labeled images in the public image dataset into multiple 1024*1024 size images; then randomly divide the cropped images into training set, validation set and test set;
[0011] Step 2: Construct a deep neural network model for target detection, including Res101 residual network, feature weaving structure, region generation network, feature pooling layer and fully connected layer, as follows;
[0012] Step 2-1: Input the training set images into the Res101 residual network, extract features through the Res101 residual network, and output 6 feature maps of different sizes from different layers of the Res101 residual network, denoted as C1 to C6, with feature map scales of 512*512, 256*256, 128*128, 64*64, 32*32, and 16*16 respectively;
[0013] Step 2-2: Select feature maps C2, C3, C4 and C5 to generate a feature pyramid. The layers in the feature pyramid are recorded as P2, P3, P4 and P5, as follows:
[0014] Step 2-2-1: Use feature map C5 directly as feature map P5 of feature pyramid;
[0015] Step 2-2-2: The characteristic braided structure comprises three branches;
[0016] Input feature maps C2, C3, C4 and C5 into the first branch; perform 1*1 convolution on feature map C4; downsample the features of feature maps C2 and C3 after 1*1 convolution respectively; perform bilinear interpolation upsampling on the features of feature map C5 after 1*1 convolution to keep it consistent with the dimension of feature map C4; finally, add the four processed features and add them element by element with the upsampled result of P5 to generate feature map P4;
[0017] Input feature maps C2, C3, C4 and C5 into the second branch; perform 1*1 convolution on feature map C3; downsample the features after 1*1 convolution on feature map C2; perform bilinear interpolation upsampling on the features after 1*1 convolution on feature maps C4 and C5 respectively, so that both are consistent with the dimension of feature map C3; finally, add the processed four features, and then add them element by element with the upsampled result of P4 to generate feature map P3;
[0018] Input feature maps C2, C3, C4 and C5 into the third branch; perform 1*1 convolution on feature map C2; perform bilinear interpolation upsampling on the features of feature maps C3, C4 and C5 after 1*1 convolution respectively, so that the three are consistent with the dimension of feature map C2; finally, add the processed four features, and then add them element by element with the upsampled result of P3 to generate feature map P2;
[0019] Step 2-3: The size of each layer of feature maps P2, P3, P4, P5 and P6 in the feature pyramid is 32 respectively. 2 , 64 2 , 128 2 , 256 2 and 512 2 ; Set the aspect ratios of the three anchor points to 1:2, 1:1 and 2:1 respectively; then there are 15 different anchor points in the entire feature pyramid;
[0020] Use the region generation network to generate anchor points and then obtain the candidate region. The calculation formula is:
[0021]
[0022] Among them, (x c ,y c ) is the anchor point coordinate, (w,h) are the width and height of the candidate region, (x 1 ,y 1 ) and (x 2 ,y 2 ) are the coordinates of the upper left corner and lower right corner of the candidate area respectively;
[0023] Calculate the intersection over union (IoU) of the label and the candidate region used when training the candidate region; if IoU>0.7, the candidate region is a positive sample; if IoU<0.3, the candidate region is a negative sample; if 0.3≤IoU≤0.7, no marking is done;
[0024] Step 2-4: Input the candidate regions into different feature pooling layers according to the scales of feature maps P2, P3, P4, and P5;
[0025] Calculate the number of layers k' in the feature pyramid corresponding to each candidate region after feature pooling:
[0026]
[0027]
[0028] Among them, k 0 is the base value, set to 4;
[0029] The candidate area in each feature map in the feature pyramid passes through the corresponding feature pooling layer, outputs 7*7 results respectively, and extracts 49 features;
[0030] Step 2-5: According to the k' value calculated in step 2-4, select the feature pooling layer corresponding to the number of layers of the feature pyramid with the same k' value, and input all 49 features output by the feature pooling layer into two fully connected layers respectively. The output results of the two fully connected layers are the target category and the target bounding box;
[0031] Step 3: Use gradient descent and back propagation algorithms to train the target detection deep neural network model. After training, use the test set to test and obtain the final target detection deep neural network model.
[0032] The beneficial effects of the present invention are as follows:
[0033] The present invention uses a feature weaving connection method to replace the lateral connection of the feature pyramid, fully considers multi-scale feature information, and increases the recognition accuracy of small targets by fusing features of different scales. It also optimizes the recognition rate of large targets to a certain extent, thereby improving the overall target detection performance in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a flow chart of the method of the present invention.
[0035] Figure 2 It is a structural diagram of the deep neural network model for target detection according to the method of the present invention.
[0036] Figure 3 It is a schematic diagram of the characteristic braided single-branch output P2 structure of the method of the present invention. DETAILED DESCRIPTION
[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0038] Most of the existing remote sensing image target detection algorithms use general target detection methods. To solve the problems of non-fixed target scales in single images, the presence of multiple types of targets, and complex backgrounds in remote sensing images, the present invention designs a feature weaving method. By weaving high-level and low-level features in the connection of a feature pyramid network, the multi-scale features in the network are fully utilized to improve the accuracy of target recognition in remote sensing images.
[0039] like Figure 1 As shown, a feature weaving method for remote sensing image target detection includes the following steps:
[0040] Step 1: Use a public image dataset; crop the labeled images in the public image dataset into multiple 1024*1024 size images; then randomly divide the cropped images into training set, validation set and test set;
[0041] Step 2: Construct a deep neural network model for target detection, including Res101 residual network, feature weaving structure, region generation network, feature pooling layer and fully connected layer, as follows;
[0042] Step 2-1: Input the training set images into the Res101 residual network, extract features through the Res101 residual network, and output 6 feature maps of different sizes from different layers of the Res101 residual network, denoted as C1 to C6, with feature map scales of 512*512, 256*256, 128*128, 64*64, 32*32, and 16*16 respectively;
[0043] Step 2-2: Select feature maps C2, C3, C4 and C5 to generate a feature pyramid. The layers in the feature pyramid are recorded as P2, P3, P4 and P5, as follows:
[0044] Step 2-2-1: Use feature map C5 directly as feature map P5 of feature pyramid;
[0045] Step 2-2-2: The characteristic braided structure comprises three branches;
[0046] Input feature maps C2, C3, C4 and C5 into the first branch; perform 1*1 convolution on feature map C4; downsample the features of feature maps C2 and C3 after 1*1 convolution respectively; perform bilinear interpolation upsampling on the features of feature map C5 after 1*1 convolution to keep it consistent with the dimension of feature map C4; finally, add the four processed features and add them element by element with the upsampled result of P5 to generate feature map P4;
[0047] Input feature maps C2, C3, C4 and C5 into the second branch; perform 1*1 convolution on feature map C3; downsample the features after 1*1 convolution on feature map C2; perform bilinear interpolation upsampling on the features after 1*1 convolution on feature maps C4 and C5 respectively, so that both are consistent with the dimension of feature map C3; finally, add the four processed features, and then add them element by element with the upsampled result of P4 to generate feature map P3.
[0048] Input feature maps C2, C3, C4 and C5 into the third branch; perform 1*1 convolution on feature map C2; perform bilinear interpolation upsampling on the features after 1*1 convolution of feature maps C3, C4 and C5 respectively, so that the dimensions of the three are consistent with feature map C2; finally, add the four processed features, and then add them element by element with the upsampled result of P3 to generate feature map P2.
[0049] Step 2-3: The size of each layer of feature maps P2, P3, P4, P5 and P6 in the feature pyramid is 32 respectively. 2 , 64 2 , 128 2 , 256 2 and 512 2 ; Set the aspect ratios of the three anchor points to 1:2, 1:1 and 2:1 respectively; then there are 15 different anchor points in the entire feature pyramid;
[0050] Use the region generation network to generate anchor points and then obtain the candidate region. The calculation formula is:
[0051]
[0052] Among them, (x c ,y c ) is the anchor point coordinate, (w,h) are the width and height of the candidate region, (x 1 ,y 1 ) and (x 2 ,y 2 ) are the coordinates of the upper left corner and lower right corner of the candidate area respectively;
[0053] Calculate the intersection over union (IoU) of the label and the candidate region used when training the candidate region; if IoU>0.7, the candidate region is a positive sample; if IoU<0.3, the candidate region is a negative sample; if 0.3≤IoU≤0.7, no marking is done;
[0054] Step 2-4: Input the candidate regions into different feature pooling layers according to the scales of feature maps P2, P3, P4, and P5;
[0055] Calculate the number of layers k' in the feature pyramid corresponding to each candidate region after feature pooling:
[0056]
[0057]
[0058] Among them, k 0 is the base value, set to 4;
[0059] The candidate area in each feature map in the feature pyramid passes through the corresponding feature pooling layer, outputs 7*7 results respectively, and extracts 49 features;
[0060] Step 2-5: According to the k' value calculated in step 2-4, select the feature pooling layer corresponding to the number of layers of the feature pyramid with the same k' value, and input all 49 features output by the feature pooling layer into two fully connected layers respectively. The output results of the two fully connected layers are the target category and the target bounding box;
[0061] Step 3: Use gradient descent and back propagation algorithms to train the target detection deep neural network model. After training, use the test set to test and obtain the final target detection deep neural network model. Specific embodiment:
[0063] 1. Process the DOTA dataset. Since the original data image size of the DOTA dataset is uncertain and the labeled data of the test set is not public, in order to facilitate neural network training, the 1869 labeled images are uniformly cropped into 1024*1024 images. When cropping, in order to prevent the target from being lost due to image cropping, the width and height retain a 10% pixel overlap rate. After processing, 19219 images and their annotation information are obtained, which are randomly divided into 11531 images for the training set, 3844 images for the validation set, and 3844 images for the test set, ensuring that the training set, validation set, and test set have no intersection in the image sample space.
[0064] 2. Build a deep neural network and use gradient descent and back propagation algorithms to train the deep neural network model. The specific process is as follows:
[0065] (1) Based on Figure 1 A deep neural network model is constructed. The model mainly includes Res101 feature extraction, FPN generation of candidate regions, local context perception of candidate regions, and finally feature pooling and fully connected layers to obtain target categories and bounding boxes.
[0066] (2) Since the neural network has many parameters and is difficult to train, the present invention uses a pre-trained model to initialize the Res101 model parameters before training the model to improve the training efficiency.
[0067] (3) The neural network is trained on the training data set. For each image with a size of 1024*1024 that is input to the neural network, after extracting features through the Res101 residual network, 6 feature maps of different sizes are generated, denoted as C1~C6, with scales of 512*512, 256*256, 128*128, 64*64, 32*32, and 16*16 respectively. C2, C3, C4, and C5 are selected to build the pyramid. C6 is not used because it is the highest-level feature and contains less information after feature extraction; using C1 will take up too much memory, so C1 is not used to build the pyramid.
[0068] (4) Generate a feature pyramid based on C2, C3, C4 and C5, and the results are recorded as P2, P3, P4 and P5. P2, P3 and P4 are the down-sampling of high-level features and are added to the features of the same layer C2, C3 and C4 through the feature weaving structure. Feature weaving includes three branches, each of which includes four 1*1 convolutions, and the input is the features of layers C2 to C5. These four 1*1 convolutions share parameters. Specifically, in the first branch, the features after the 1*1 convolution with the input C2 are not processed, and the features after the other three 1*1 convolutions are up-sampled by bilinear interpolation to keep them consistent with the dimension of C2. Finally, the four processed features are added and input into the top-down connection to generate P2. In the second branch, the features after the 1*1 convolution with the input C3 are not processed, the features after the 1*1 convolution with the input C2 are down-sampled, and the features after the other two 1*1 convolutions are up-sampled by bilinear interpolation to keep them consistent with the dimension of C3. Finally, the four processed features are added together and input into the top-down connection to generate P3. In the third branch, the features after the 1*1 convolution with input C4 are not processed, the features after the 1*1 convolution with input C2 and C3 are downsampled, and the features after the other 1*1 convolutions are upsampled by bilinear interpolation to keep them consistent with the C4 dimension. Finally, the four processed features are added together and input into the top-down connection to generate P4.
[0069] (5) Each feature map in the feature pyramid uses only one size when generating anchors using the region generation network, P2, P3, P4, P5, and P6 correspond to 32 respectively. 2 , 64 2 , 128 2 , 256 2 and 512 2 , the aspect ratios of each anchor are 1:2, 1:1 and 2:1. In this way, there are 15 different anchors in the entire feature pyramid. Note that P6 is not used in the subsequent stages.
[0070] (6) Because the feature maps of the four scales all have proposals mapped from the anchors on the original image, when using feature pooling, four different feature layers must also be used. The feature layer corresponding to the feature pooling is calculated using the following formula:
[0071]
[0072] Among them, w and h are the sizes of the corresponding proposals, 1024 is the original image size, and k 0 The base value is set to 4.
[0073] Since P2, P3, P4 and P5 are feature maps of the feature pyramid, the value rules of the corresponding feature layers are:
[0074]
[0075] The proposals in each feature map in the feature pyramid pass through the corresponding feature pooling layer and output 7*7 results respectively. That is, after the proposals on these four feature maps are pooled, 49 features are extracted.
[0076] (7) The 7*7 results of proposals are connected to two fully connected layers respectively, and the output results of the two fully connected layers are the target category and the target bounding box.
Claims
1. A feature knitting method for remote sensing image target detection, characterized in that: The following steps are involved: Step 1: Use a public image dataset; crop the labeled images in the public image dataset into multiple 1024*1024 size images; then randomly divide the cropped images into training set, validation set and test set; Step 2: Construct a deep neural network model for target detection, including Res101 residual network, feature weaving structure, region generation network, feature pooling layer and fully connected layer, as follows; Step 2-1: Input the training set images into the Res101 residual network, extract features through the Res101 residual network, and output 6 feature maps of different sizes from different layers of the Res101 residual network, denoted as C1 to C6, with feature map scales of 512*512, 256*256, 128*128, 64*64, 32*32, and 16*16 respectively; Step 2-2: Select feature maps C2, C3, C4 and C5 to generate a feature pyramid. The layers in the feature pyramid are recorded as P2, P3, P4 and P5, as follows: Step 2-2-1: Use feature map C5 directly as feature map P5 of feature pyramid; Step 2-2-2: The characteristic braided structure comprises three branches; Input feature maps C2, C3, C4 and C5 into the first branch; perform 1*1 convolution on feature map C4; downsample the features of feature maps C2 and C3 after 1*1 convolution respectively; perform bilinear interpolation upsampling on the features of feature map C5 after 1*1 convolution to keep it consistent with the dimension of feature map C4; finally, add the four processed features and then add them element by element with the upsampled result of P5 to generate feature map P4; Input feature maps C2, C3, C4 and C5 into the second branch; perform 1*1 convolution on feature map C3; downsample the features after 1*1 convolution on feature map C2; perform bilinear interpolation upsampling on the features after 1*1 convolution on feature maps C4 and C5 respectively, so that both are consistent with the dimension of feature map C3; finally, add the processed four features, and then add them element by element with the upsampled result of P4 to generate feature map P3; Input feature maps C2, C3, C4 and C5 into the third branch; perform 1*1 convolution on feature map C2; perform bilinear interpolation upsampling on the features of feature maps C3, C4 and C5 after 1*1 convolution respectively, so that the three are consistent with the dimension of feature map C2; finally, add the processed four features, and then add them element by element with the upsampled result of P3 to generate feature map P2; Step 2-3: The size of each layer of feature maps P2, P3, P4, P5 and P6 in the feature pyramid is 32 respectively. 2 , 64 2 , 128 2 , 256 2 and 512 2 ; Set the aspect ratios of the three anchor points to 1:2, 1:1 and 2:1 respectively; then there are 15 different anchor points in the entire feature pyramid; Use the region generation network to generate anchor points and then obtain the candidate region. The calculation formula is: Among them, (x c ,y c ) is the coordinate of the anchor point, (w,h) are the width and height of the candidate area, (x1,y1) and (x2,y2) are the coordinates of the upper left corner and lower right corner of the candidate area respectively; Calculate the intersection over union (IoU) of the label and the candidate region used when training the candidate region; if IoU>0.7, the candidate region is a positive sample; if IoU<0.3, the candidate region is a negative sample; if 0.3≤IoU≤0.7, no marking is done; Step 2-4: Input the candidate regions into different feature pooling layers according to the scales of feature maps P2, P3, P4, and P5; Calculate the number of layers k' in the feature pyramid corresponding to each candidate region after feature pooling: Among them, k0 is the reference value, which is set to 4; The candidate area in each feature map in the feature pyramid passes through the corresponding feature pooling layer, outputs 7*7 results respectively, and extracts 49 features; Step 2-5: According to the k' value calculated in step 2-4, select the feature pooling layer corresponding to the number of layers of the feature pyramid with the same k' value, and input all 49 features output by the feature pooling layer into two fully connected layers respectively. The output results of the two fully connected layers are the target category and the target bounding box; Step 3: Use gradient descent and back propagation algorithms to train the target detection deep neural network model. After training, use the test set to test and obtain the final target detection deep neural network model.
Citation Information
Patent Citations
Remote sensing image target detection method based on multi-scale semantic feature fusion
CN110909642A
Optical remote sensing image target detection method based on cavity feature pyramid network
CN111126202A
Complex scene dense target detection method based on local context awareness
CN111368775A
Image semantic feature constrained remote sensing target detection method
CN112101277A
Remote sensing imaging bridge detection method based on convolutional neural network
WO2018214195A1
