Remote sensing image target detection method based on attention and generative adversarial network

By embedding attention mechanism and feature pyramid FPN in the Mask R-CNN model and performing GAN adversarial training, the problem of the poor detection effect of small object in high-resolution remote sensing images is solved, and higher detection is achieved Accuracy and identification efficiency, suitable for target recognition in the military and agricultural fields.

CN113989612BInactive Publication Date: 2024-12-03CHINA UNIV OF GEOSCIENCES (WUHAN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110524297.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-13
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing convolutional neural networks are not effective when detecting small targets (such as aircraft, ships, and vehicles) in high-resolution remote sensing images, especially in images with large target changes, uneven sample distribution and complex backgrounds. Inadequate performance.

Method used

Embed attention mechanism and feature pyramid FPN in the Mask R-CNN model, and GAN adversarial training is performed on the Mask mask branch. Combined with the data augmentation method, use ResNet101 and FPN to generate feature maps, and target through the regional recommendation network and fully connected network. Detection and classification to improve the recognition accuracy and generalization ability of the model.

Benefits of technology

It improves the detection accuracy and recognition efficiency of small targets in remote sensing images, can extract target feature information more accurately, and is suitable for target recognition in military and civilian fields, enhancing the accuracy of command and decision-making and agricultural monitoring capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989612B_ABST
    Figure CN113989612B_ABST
Patent Text Reader

Abstract

The present invention provides a remote sensing image target detection method based on attention and generative adversarial network, including initial processing of data, then inputting the processed data set into an algorithm network, inputting an image stream through an image feature extraction network composed of a convolutional neural network; generating image target candidate regions by the network; mapping the candidate region position information to a corresponding feature map; inputting the mapped feature map into a fully connected network to predict the accurate bounding box position and complete the target classification, and a mask generation branch continues to generate a target mask based on the candidate region. The present invention is used to solve the problem of target detection of aircraft and ships in remote sensing images where some specific target data is scarce.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, deep learning, and target detection technology, and in particular to a remote sensing image target detection method based on attention and generative adversarial networks. Background Art

[0002] In the early days, due to the low spatial resolution of satellite remote sensing images, it was impossible to accurately detect some smaller objects in the image. The detection of remote sensing images mainly focused on extracting the spatial geographic attributes of a certain area, such as forests, lakes, vegetation distribution, etc. In recent years, thanks to the rapid development of remote sensing technology, the spatial resolution of aerial remote sensing images can also reach the sub-meter level, making it possible to detect individual individuals in the image.

[0003] With the support of massive high-resolution remote sensing image data, some target detection technologies applied to remote sensing images have begun to emerge one after another, especially target detection based on deep learning technology has developed rapidly. Although classification and detection models based on deep learning have been widely used in the field of remote sensing images, most of them are still focused on daily object detection. The difference between daily object detection and remote sensing image target detection is that it is easy to obtain, the scale change is relatively small, and generally there are fewer targets to be detected in a picture. These deep learning models cannot reflect their excellent learning performance in natural image recognition for remote sensing images with large target scale changes, uneven sample distribution, and complex background. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a remote sensing image target detection method based on attention and generative adversarial network to solve the problem that ordinary convolutional neural network has unsatisfactory detection effect in detecting small targets such as aircraft, ships, and vehicles in remote sensing images; the specific steps are as follows:

[0005] S1. Using the existing Mask R-CNN model, we embed the attention mechanism and feature pyramid FPN on the backbone network ResNet101, perform GAN adversarial training on the Mask branch, and obtain a trained deep network for object detection and recognition.

[0006] S2. Use data enhancement methods to expand the samples of the collected remote sensing image dataset;

[0007] S3, sending the expanded remote sensing image dataset in S2 to S1 to generate feature maps using the deep convolutional layer of the backbone network with ResNet101 and FPN;

[0008] S4, send the feature map generated in S3 to the region recommendation network in MaskR-CNN, use the convolutional neural network for classification and target detection box regression to obtain the candidate region;

[0009] S5, align the candidate regions generated in S4 to the regions of interest and reshape the size of the feature map;

[0010] S6. The resized feature map in S5 is sent to three fully connected networks to perform target classification, target detection box regression and mask generation on the image respectively.

[0011] The technical solution provided by the present invention has the following beneficial effects: the present invention applies high-performance target detection algorithms to the detection of small military targets such as aircraft and ships, improves the efficiency and accuracy of recognition, and can help commanders make correct decisions, which is of great significance to military operations and command. In the civilian field, the effective recognition of farmland and buildings can also be used to monitor agricultural production, identify illegal buildings, and other scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of a remote sensing image target detection method based on attention and generative adversarial network of the present invention;

[0013] Figure 2 It is a network architecture diagram of the remote sensing image target detection method based on attention and generative adversarial network of the present invention;

[0014] Figure 3 This is a detection effect diagram of the remote sensing image target detection method based on attention and generative adversarial network of the present invention. DETAILED DESCRIPTION

[0015] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0016] Please refer to Figure 1 The present invention provides a remote sensing image target detection method based on attention and generative adversarial network, and constructs a solution composed of MASK-RCNN+RPN+RA-ATTENTION+GAN, which can accurately extract the feature information of the target in the image and effectively improve the target detection accuracy in the remote sensing image.

[0017] In view of the complexity of targets in high-resolution remote sensing images, the small data scale and amount, and the need for military target detection, a complete aircraft target detection dataset is constructed in the data preprocessing stage for the input of convolutional neural networks. According to the accuracy requirements of military target detection, the data is initially processed, enhanced, and expanded. In the data enhancement stage, the "mosaic" enhancement technology is used to process the image to improve the recognition accuracy and generalization ability of the model.

[0018] The backbone network plans to use the Dense Net approach to connect blocks, and replace the blocks in the network with RA-Attention modules. Please refer to Figure 2 . Since there are some small targets with a proportion of less than 1% in the input data, in order to enable the model to better detect such targets, FPN (Feature Pyramid Network) is combined in the main network to reduce feature loss caused by scale changes, and RA-Attention (recurrent attention mechanism) is added to FPN to integrate contextual information that is not frequently used to ensure the integrity of data context information. In the MASK branch, the GAN network is used to pre-train the MASK discriminator to improve the accuracy of mask output.

[0019] The specific steps of the present invention are as follows:

[0020] S1. Using the existing Mask R-CNN model, we embed the attention mechanism and feature pyramid FPN on the backbone network ResNet101, perform GAN adversarial training on the Mask branch, and obtain a trained deep network for object detection and recognition.

[0021] S2. Use data enhancement methods to expand the samples of the collected remote sensing image dataset;

[0022] S3, sending the expanded remote sensing image dataset in S2 to S1 to generate feature maps using the deep convolutional layer of the backbone network with ResNet101 and FPN;

[0023] S4, send the feature map generated in S3 to the region recommendation network in MaskR-CNN, use the convolutional neural network for classification and target detection box regression to obtain the candidate region;

[0024] S5, align the candidate regions generated in S4 to the regions of interest and reshape the size of the feature map;

[0025] S6. The resized feature map in S5 is sent to three fully connected networks to perform target classification, target detection box regression and mask generation on the image respectively.

[0026] First, input the image to be processed, and then perform the corresponding preprocessing operation, or the preprocessed image; then, input it into a pre-trained neural network (ResNet50 / ResNet101) to obtain the corresponding feature map; then, according to the number of RoIs set for each point in this feature map, obtain multiple candidate RoIs; send these candidate RoIs to the RPN network for binary classification (foreground or background) and bbox regression, and filter out some candidate RoIs; then, perform RoIAlign operation on these remaining RoIs (that is, first correspond the pixels of the original image and the feature map, and then correspond the feature map to the fixed features);

[0027] Finally, these RoIs are classified (N-category classification), bbox regression and Mask generation (FCN operation is performed in each RoI). The Mask branch adds a generative adversarial network structure and adds a layer of discriminator to the Mask generator to improve the accuracy of Mask generation. Please refer to Figure 3 .

[0028] RoIAlign is used in the Mask R-CNN network, and a bilinear interpolation algorithm is introduced on the original pooling operation. The specific algorithm formula is as follows:

[0029]

[0030] x i Represents the pixel point on the feature map before pooling; y rj represents the jth point of the rth candidate region after pooling; i*(r, j) represents point y rj The source of the pixel value, the coordinates of the point with the maximum pixel value selected during the maximum pooling, d(.) represents the distance between the two points, Δh and Δw represent x i With x i *(r, j) The difference between the horizontal and vertical coordinates, which is used as the coefficient of bilinear interpolation and multiplied by the original gradient.

[0031] The loss function used in the Mask R-CNN basic network is the sum of classification loss, bounding box regression loss, and mask loss. The specific algorithm formula is as follows:

[0032] L=L cls +L box +L mask

[0033] Among them, L represents the total loss of the model, L cls represents the classification loss, L box represents the bounding box loss, L mask Indicates mask loss.

[0034] The attention module is used in the Mask R-CNN basic network. The specific algorithm formula is as follows:

[0035]

[0036] σ represents the sigmoid function, MLP represents a multi-layer perceptron, which is used to share parameters. and Represent the features of global average pooling and global maximum pooling output, respectively, 7*7 Indicates that a 7*7 convolution is performed on the concatenated feature map, M s (F) represents the final generated spatial attention feature.

[0037] The Mask R-CNN basic network uses the Mask branch network to construct a generative adversarial network. The specific optimization objective function is as follows:

[0038]

[0039] The corresponding loss function formula is as follows, where P data (x) is the distribution of real samples, D(x) is the discriminator, and G(z) is the generator;

[0040]

[0041]

[0042] Among them, L D is the loss function of the discriminator, L G is the loss function of the generator; n is the number of samples.

[0043] In the process of deep learning, after the input image passes through the neural network model, a feature matrix describing the input image will be obtained. Then, depending on the task to be solved, the feature matrix is ​​used to complete image classification, segmentation and other tasks. However, at the beginning of model training, the feature matrix cannot describe the input image very well. Therefore, it is necessary to adjust the parameters of each layer in the model according to the results of each training of the training set so that the model can better describe the input image through the feature matrix. The neural network model needs to use a loss function, through which the obtained neural network output result is compared with the actual value identified by the training set to obtain the loss value. When the loss function is 0, it means that the neural network model can perfectly fit all input data, so the model with a loss function of 0 is an ideal model. The purpose of training a neural network model is to minimize the loss value.

[0044] The neural network method trains the model through the back propagation algorithm, that is, the loss value calculated according to the loss function is propagated forward along the reverse direction of the neural network model. During this propagation process, the convolution parameter weight parameter w and the bias b of each layer are adjusted according to the direction of gradient descent. Through continuous iteration, a converged neural network model is obtained. In order to improve the convergence speed of the network model and the robustness of the feature results, this paper adopts the small batch stochastic gradient descent method for training.

[0045] In the training process of the improved Mask R-CNN neural network model, the input batch size is set to 2, and the number of network iterations is consistent with the original Mask R-CNN, which is 90000. The parameter update strategy is the momentum gradient descent optimization method and the learning rate decay method. The momentum gradient descent optimization method is shown in the formula:

[0046]

[0047] Where V dw represents the rate of decrease of weight w, V db represents the rate of decrease of bias b, β is the momentum value of gradient update, and α is the learning rate. The learning rate attenuation method is shown in the formula:

[0048]

[0049] Where α is the learning rate, d is the decay rate, n is the number of iterations, and α0 is the initial learning rate.

[0050] When training the improved Mask R-CNN, α0 = 0.0025, β = 0.9, d = 0.0001, and the piecewise constant learning rate decay method is used, setting n to 60000 and 80000. The method is shown in the formula:

[0051]

[0052] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A remote sensing image target detection method based on attention and generative adversarial network, characterized in that: It includes the following steps: S1. Using the existing MaskR-CNN model, we embed the attention mechanism and feature pyramid FPN on the backbone network ResNet101, perform GAN adversarial training on the Mask branch, and obtain a trained deep network for object detection and recognition. S2. Use data enhancement methods to expand the samples of the collected remote sensing image dataset; S3, sending the expanded remote sensing image dataset in S2 to S1 to generate feature maps using the deep convolutional layer of the backbone network with ResNet101 and FPN; S4, send the feature map generated in S3 to the region recommendation network in MaskR-CNN, use the convolutional neural network for classification and target detection box regression to obtain the candidate region; S5, align the candidate regions generated in S4 to the regions of interest and reshape the size of the feature map; S6, sending the resized feature map in S5 to three fully connected networks to perform object classification, object detection box regression and mask generation on the image respectively; RoIAlign is used in the MaskR-CNN network, and a bilinear interpolation algorithm is introduced on the original pooling operation. The specific algorithm formula is as follows: x i Represents the pixel point on the feature map before pooling; y rj Represents the jth point of the rth candidate region after pooling; i*(r,j) represents point y rj The source of the pixel value, the coordinates of the point with the maximum pixel value selected during the maximum pooling, d(.) represents the distance between the two points, Δh and Δw represent x i With x i *(r,j) The difference between the horizontal and vertical coordinates, which is used as the coefficient of bilinear interpolation and multiplied by the original gradient; The loss function used in the MaskR-CNN basic network is the sum of classification loss, bounding box regression loss, and mask loss. The specific algorithm formula is as follows: L=L cls +L box +L mask Among them, L represents the total loss of the model, L cls represents the classification loss, L box represents the bounding box loss, L mask represents mask loss; The attention module is used in the MaskR-CNN basic network. The specific algorithm formula is as follows: σ represents the sigmoid function, MLP represents a multi-layer perceptron, which is used to share parameters. and Represent the features of global average pooling and global maximum pooling output, respectively, 7 * 7 Indicates that a 7*7 convolution is performed on the concatenated feature map, M s (F) represents the final generated spatial attention feature; The Mask R-CNN basic network uses the Mask branch network to construct a generative adversarial network. The specific optimization objective function is as follows: The corresponding loss function formula is as follows, where P data (x) is the distribution of real samples, D(x) is the discriminator, and G(z) is the generator; Among them, L D is the loss function of the discriminator, L G is the loss function of the generator; n is the number of samples.

2. The remote sensing image target detection method based on attention and generative adversarial network according to claim 1 is characterized in that: The enhancement methods include scale transformation, rotation and mosaic.

3. The remote sensing image target detection method based on attention and generative adversarial network according to claim 1 is characterized in that: The acquisition of the target detection frame specifically includes: S11, normalizing the size of the initial image; S12. Use ResNet101 to extract image features, add an attention module to the backbone network, calculate the convolutional features of the image, input the last layer of convolutional features into the feature pyramid network, build a feature map containing semantics and position, and finally use the output feature map of the feature pyramid network to build a candidate region generation network recommendation map to generate candidate regions; S13, aligning the candidate region with the region of interest, inputting it into the fully connected layer for prediction, and obtaining multiple target detection candidate boxes; S14. Use the non-maximum suppression algorithm to remove redundant target detection frames and obtain the final target detection frame.