A method for ground object detection in infrared remote sensing images based on feature migration

Abstract

This paper proposes a method for detecting ground objects in infrared remote sensing imagery based on feature transfer. Based on the Faster-RCNN algorithm, this method adds a ResNet network to improve its recognition accuracy. It also introduces a feature transfer learning method, which uses learning from a large existing dataset of visible light remote sensing images and applies this knowledge to target detection in infrared remote sensing imagery datasets. This method not only improves the utilization of visible light remote sensing images but also enhances the detection capabilities of the target detection network at night.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image ground object detection, and is infrared remote sensing image ground object detection based on feature migration. Technical Background

[0002] Target recognition has long been a hot topic of research in the field of vision. Its successful application will promote the development and progress of both military and civilian fields and is of great significance to society. Infrared remote sensing systems use drones and other means to collect ground object information and draw infrared remote sensing images by monitoring the infrared energy radiated and reflected by ground objects. Because ground objects emit infrared radiation throughout the day, infrared remote sensing images have the ability to work all day long. However, visible light images only work well during the day. Because of this excellent characteristic, infrared images can be applied by researchers in areas such as road supervision and war simulation. Therefore, how to achieve good target recognition capabilities in infrared remote sensing images is of great significance. This invention mainly focuses on the Faster-RCNN algorithm. Some common target detection network models, such as VGGNet and ResNet network models, require training based on large data sets. Under real conditions, the size of the visible light band ground object target data set is much larger than that of the infrared band target data set, but the visible light band does not have the ability to work all day long. Typically, we have access to a large number of visible light datasets. To increase the efficiency of using these datasets, improve their adaptability in different scenarios, and achieve better target recognition capabilities in infrared remote sensing images, we need to improve the existing target detection network. Summary of the Invention

[0003] In order to overcome the shortcomings of the above-mentioned target detection network structure, the present invention proposes a method for infrared remote sensing image ground object target detection based on feature migration. This method, based on Faster-RCNN, replaces the backbone network of the original Faster-RCNN detection network, increases the ability of the neural network to extract corresponding features of complex scenes, and achieves higher target detection accuracy. The Faster-RCNN structure is as follows Figure 1 As shown in the figure. That is, adding the ResNet network improves its recognition accuracy. At the same time, this method performs target detection under the condition that there are more visible light object image datasets and fewer infrared object image datasets. The target detection network requires a large amount of data sets for training to ensure the accuracy of target detection. When there are not enough infrared object images, we introduce the transfer learning method and use the DANN network to distinguish the source domain from the target domain. The DANN structure diagram is shown in the figure. Figure 2As shown in the figure, the method uses a large number of existing visible light remote sensing image datasets for learning and applies the knowledge to target detection in infrared remote sensing image datasets, which not only improves the utilization rate of visible light remote sensing images, but also increases the detection capability of the target detection network at night. Figure 3 shown.

[0004] The technical solution adopted by the present invention is a method for detecting ground objects in infrared remote sensing images based on feature migration, which includes:

[0005] Step 1: Create a VOC type dataset to uniformly store images and annotation information.

[0006] Step 2: Input the source domain and target domain image datasets into the ResNet-50 network structure to extract the corresponding features.

[0007] Step 3: After the ResNet-50 network extracts the corresponding features, we import them into the image-level feature transfer network and transfer the overall features of the image to make P T (I) = P S (I), and finally the obtained features are returned to the feature extractor and propagated down the network.

[0008] Step 4: Input the feature map into RPN and output the candidate box (proposal) generated by the image.

[0009] Step 5: At the ROI layer, proposal feature maps are selected based on the original features proposed by ResNet-50 and the candidate boxes generated by the RPN network.

[0010] Step 6: After selecting the corresponding candidate box at the ROI layer, the location information and regional features of the candidate box are transferred to the instance-level feature migration module and migrated to make P T (B,I)=P S (B, I), and then return the corresponding features to the ROI layer and propagate downward

[0011] Step 7: Classify all candidate boxes into specific categories through the fully connected layer and softmax to achieve target classification and detection.

[0012] Furthermore, the specific method of step 2 is:

[0013] Step 2.1: Downsample the input image dataset by convolution with a 7×7 kernel and stride=2;

[0014] Step 2.2: Convolve the result of step 1 using 1×1, 3×3, and 1×1 convolution kernels respectively;

[0015] Step 2.3: Add a short-circuit mechanism between every three layers and add residual units through the short-circuit mechanism;

[0016] Step 2.4: Repeat steps 2.2 and 2.3.

[0017] Step 2.5: After 50 convolutions, perform average pooling

[0018] Furthermore, the specific method of step 3 is:

[0019] Step 3.1: Extract features from the input image and make the image feature probability distribution P(I) the same in the source and target domains, and pass it to the domain classifier;

[0020] Step 3.2: The domain classifier generates a classification loss function Ld, and the relevant gradients are back-propagated. The domain classifier improves the relevant parameters to reduce the classification error of the domain classification.

[0021] Furthermore, the specific method of step 4 is:

[0022] Step 4.1: Convolution is performed on the obtained feature map using a 3*3 sliding window

[0023] Step 4.2: After two 1x 1 convolutions, output multiple anchor boxes

[0024] Step 4.3: For the anchor box, use the position regression value to translate and scale the box to generate a large number of candidate boxes.

[0025] Furthermore, the specific method of step 6 is:

[0026] Step 6.1: Input the location information and regional features of the candidate boxes, and make the distribution of the size and position of each target feature and its corresponding bounding box consistent in the source and target domain images;

[0027] Step 6.2: The domain classifier generates a classification loss function Ld, and the relevant gradients are back-propagated. The domain classifier improves the relevant parameters to reduce the classification error of the domain classification.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The backbone network in Faster-RCNN is replaced with the ResNet-50 network structure with a residual structure. By using this structure, redundancy and even network degradation caused by too many layers in the convolutional network can be avoided. As a result, features of related vehicles can be extracted in some complex scenarios, such as the low-light scenario used this time.

[0030] (2) Target detection is performed under the condition that there are many visible light ground object image datasets but few infrared ground object image datasets. The target detection network requires a large number of datasets for training to ensure the accuracy of target detection. When there are not enough infrared ground object images, we introduce transfer learning methods to ensure accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the basic framework of Faster-RCNN;

[0032] Figure 2 This is the DANN network structure diagram;

[0033] Figure 3 This is a technical route of a method for detecting ground objects in infrared remote sensing images based on feature migration in the present invention;

[0034] Figure 4 This is a sample example of the VisDrone-DroneVehicle dataset;

[0035] Figure 5 The labeled image dataset and image frame display;

[0036] Figure 6 This is a demonstration of the effects of three detection algorithms;

[0037] Specific implementation steps

[0038] 1. Network structure design

[0039] The Faster-RCNN object detection network used in this paper adds modules such as the Region Proposal Network (RPN) compared to the original network. The RPN module can share the entire image feature map extracted by the backbone network with the detection network and can simultaneously predict bounding boxes and object scores. The addition of these modules optimizes the detection network's detection performance and significantly increases detection speed.

[0040] The Faster-RCNN object detection network was replaced with the ResNet-50 network. Compared to the original VGG-16 network, this network is deeper. In the neural network, Batch Normalization (BN) layers are used to address vanishing and exploding gradients, and a residual structure is used to mitigate degradation. ResNet-50 avoids redundancy and even network degradation caused by too many layers in convolutional networks, enabling feature extraction of relevant vehicles in complex scenarios, such as the low-light scenario used here.

[0041] The DANN network model used for transfer learning. In a DANN, the feature extractor first extracts features from the input image and passes them to the domain classifier. The domain classifier generates a classification loss function, Ld. The associated gradients are backpropagated, and the domain classifier improves the relevant parameters to reduce the classification error and minimize the loss function. When the gradients pass through the Gradient Reversal Layer (GRL), which negates them at a certain ratio and propagates them to the feature extractor, the original feature extractor changes its update direction, updating toward maximizing the loss function. This results in the feature extractor tending to extract similar features from samples in both domains. By using the GRL structure, the feature extractor and the domain classifier form a generative adversarial relationship, implementing a GAN-like generative adversarial network. The feature extractor acts as a generator, continuously extracting similar features from the source and target domains in an attempt to deceive the domain classifier, while the domain classifier acts as a discriminator, continuously improving its ability to discern the source of features. Ultimately, both continuously strengthen their capabilities during training, achieving the key point of feature learning and reducing inter-domain differences. In this paper, for global image features, after the ResNet-50 network extracts the corresponding features, we import them into the image-level feature transfer network and transfer the overall image features, so that PT(I) = PS(I). Finally, the obtained features are returned to the feature extractor and propagated downward along the network. For local image features, after the object detection network selects the corresponding candidate box at the RoI layer, the location information and regional features of the candidate box are passed to the instance-level feature transfer module and transferred, so that PT(B,I) = PS(B,I). The corresponding features are then returned to the RoI layer and propagated downward to achieve object classification and detection.

[0042] 2. Target Detection in Infrared Remote Sensing Images

[0043] This paper uses the VisDrone-DroneVehicle dataset as the main data. The training data consists of two parts: 1000 source training data (i.e., visible light remote sensing image dataset) that provides images and their annotations, and 500 target domain training data (i.e., infrared remote sensing image dataset). The dataset sample example is as follows: Figure 4 As shown in the figure, the features extracted from visible light images and infrared images of the same object are different. Without using transfer learning, using images of both scenes for training will cause the classification network to confuse the features of the objects. In order to avoid confusion, assuming that the infrared image dataset is small, the first two methods without transfer learning methods are trained using only 500 target domain training datasets. The detection effects of Faster-RCNN (VGG16), Faster-RCNN (ResNet50) and the network model algorithm proposed in this paper on the infrared remote sensing object image dataset are shown in the figure. Figure 5 shown

[0044] The final target detection result accuracy evaluation indicators are as follows Table 1:

[0045] Table 1 Detection accuracy of different methods on various infrared remote sensing target samples

[0046]

[0047] Table 1 shows that, compared with the other two methods, the original Faster RCNN object detection network performs poorly for infrared ground object detection. This may be due to the complex scene in the dataset we used, making it difficult for the traditional VGG16 to extract features. However, the detection network using the ResNet residual network slightly outperforms the original object detection network, demonstrating good performance for all four object categories: car, bus, van, and truck. It achieves the best performance for truck object detection.

[0048] Compared to the other two networks, the proposed network performs best in detecting various infrared remote sensing images. Analysis shows that this is due to the transfer learning module used, which can be trained on a larger number of visible light datasets and apply the knowledge learned from these visible light datasets to infrared images.

[0049] As can be seen from the table, the detection accuracy of the detection network varies for different targets. For the recognition of car targets, all three networks show good results. After analyzing the image dataset, we believe that this is because there are more cars driving on the road and parked on the campus when the camera is used to take pictures. The results of the three detection algorithms are shown as follows: Figure 6 shown.

Claims

1. A method for detecting ground objects in infrared remote sensing images based on feature migration, the method comprising: Step 1: Create a VOC type dataset to uniformly store images and annotation information; Step 2: Input the source domain and target domain image datasets into the ResNet-50 network structure to extract corresponding features; Step 2.1: Downsample the input image dataset by convolution with a 7×7 kernel and stride=2; Step 2.2: Convolve the result of step 1 using 1×1, 3×3, and 1×1 convolution kernels respectively; Step 2.3: Add a short-circuit mechanism between every three layers and add residual units through the short-circuit mechanism; Step 2.4: Repeat the steps 2.2 and 2.

3. Step 2.5: After 50 convolutions, perform average pooling Step 3: After the ResNet-50 network extracts the corresponding features, we import them into the image-level feature transfer network and transfer the overall image features so that PT(I) = PS(I). Finally, the obtained features are returned to the feature extractor and propagated down the network. Step 3.1: Extract features from the input image and make the image feature probability distribution P(I) the same in the source and target domains, and pass it to the domain classifier; Step 3.2: The domain classifier generates a classification loss function Ld, and the relevant gradients are back-propagated. The domain classifier improves the relevant parameters to reduce the classification error of the domain classification; Step 4: Input the feature map into RPN and output the candidate box generated by the image; Step 4.1: Perform convolution on the obtained feature map using a 3*3 sliding window; Step 4.2: After two 1x 1 convolutions, multiple anchor boxes are output; Step 4.3: For the anchor box, use the position regression value to translate and scale the box to generate a large number of candidate boxes; Step 5: At the ROI layer, based on the original features proposed by ResNet-50 and the candidate boxes generated by the RPN network, proposal feature maps are selected on the original features. Step 6: After selecting the corresponding candidate box at the ROI layer, the location information and regional features of the candidate box are passed to the instance-level feature transfer module, and they are transferred so that PT(B,I)=PS(B,I). Then, the corresponding features are returned to the ROI layer and propagated downwards. Step 6.1: Input the location information and regional features of the candidate boxes, and make the distribution of the size and position of each target feature and its corresponding bounding box consistent in the source and target domain images; Step 6.2: The domain classifier generates a classification loss function Ld, and the relevant gradients are back-propagated. The domain classifier improves the relevant parameters to reduce the classification error of the domain classification; Step 7: Classify all candidate boxes into specific categories through the fully connected layer and softmax to achieve target classification and detection.

2. The method according to claim 1, wherein: In step 2, the backbone network in Faster-RCNN is replaced with a ResNet-50 network structure with a residual structure. This structure avoids redundancy and even network degradation caused by too many layers in the convolutional network.

3. The method according to claim 1, wherein: Step 3 introduces the image-level feature transfer module.

4. The method according to claim 1, wherein: In step 6, the instance-level feature transfer module is introduced.

5. The method according to claim 1, wherein: The model used in the migration module introduced in steps 3 and 6 is the DANN network model.

Images