Aerial image detection method based on GAN and feature enhancement

Abstract

The present invention discloses a GAN and feature-enhanced aerial image detection method. For aerial images, on the basis of obtaining a concentrated detection area, the resolution of the concentrated detection area is improved by using an ESRGAN model, and the super-resolution image with improved resolution and the category features corresponding to the detection target class are used as inputs of the detection model. The detection model embedded with ROI Transformer is used to enhance the feature of the category features to realize target detection, which can greatly improve the accuracy of small target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and target detection, and specifically relates to an aerial image detection method based on GAN and feature enhancement. Background Art

[0002] Object detection is a fundamental task in computer vision. Object detection and recognition generally consists of two steps: first, locating the potential position of each target object, and then identifying the object and classifying it into different categories. Before the rise of deep learning methods, object detection methods relied on hand-crafted feature engineering and designed classifiers based on the way humans understand objects. In recent years, with the increasing maturity of deep learning technology, especially deep convolutional neural networks, object detection research has also made significant progress, and object detection technology has also been applied in real life. Now object detection has been widely used in many fields such as autonomous driving, visual search, virtual reality and augmented reality.

[0003] Although accurate detection of objects of different sizes in images has been achieved in many applications, accurate detection of small objects, such as a 20x20 pixel airplane in aerial images, remains very challenging. Small objects are often difficult to detect due to factors such as difficult to distinguish features, low resolution, complex backgrounds, and limited contextual information.

[0004] Small target detection is an increasingly popular research field. In recent years, small target detection technologies based on deep learning have been proposed and implemented one after another, and have achieved good results. Some research works have shown that combining different feature layers is conducive to enhancing features and improving the performance of small target detection, while other works have proved that combining contextual information is very useful. Although the research on small target detection has achieved certain results. However, due to the characteristics of small objects, there are still many challenges, such as the features generated by each layer in the basic CNN do not contain enough useful information for small object detection, lack of contextual information when detecting small targets, the imbalance of foreground and background training samples makes classification difficult, and there are insufficient training samples containing small targets.

[0005] Patent document CN 114445705A discloses an aerial image detection method based on GAN and feature enhancement. The detection method effectively distinguishes dense areas by using a method based on dense area perception, and then uses the global context information extracted during the rough detection of the entire image to assist the fine detection of dense areas, thereby improving the performance of small target detection in dense areas. Using a detector with a simpler structure in the processing of non-dense areas can greatly reduce the resources required for calculation, improve the detection speed of the target detection model, and reduce the video memory usage required during the detection process, avoiding unnecessary waste.

[0006] However, although the detection method provided by the patent document takes contextual information into account and performs fine detection on dense areas, aerial images still have problems of low resolution and complex background. When identifying small targets such as 20x20 pixel airplanes, it is still not able to identify accurately and cannot meet the detection and recognition requirements of extremely small targets in aerial images. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide an aerial image detection method based on GAN and feature enhancement, which improves the accuracy of target recognition in aerial images by improving the resolution and feature enhancement of concentrated detection areas.

[0008] To achieve the above-mentioned purpose of the invention, an aerial image detection method based on GAN and feature enhancement is provided in the embodiment, comprising the following steps:

[0009] Step 1: Obtain an aerial image and use YOLOv4 to perform ROI region detection on the aerial image, and merge the detected ROI regions into multiple centralized detection regions;

[0010] Step 2: Use the ESRGAN model to improve the resolution of the concentrated detection area to obtain a super-resolution image;

[0011] Step 3, using the CAM algorithm, threshold screening and spectral clustering method to determine the category features corresponding to the detection target class from the classification image data set;

[0012] Step 4: Based on FasterR-CNN including a feature extraction unit and a regression prediction unit, ROITransformer is embedded between the feature extraction unit and the regression prediction unit to form an object detection model;

[0013] Step 5: Concatenate the super-resolution image and the category features and input them into the target detection model. After feature extraction by the feature extraction unit, rotation invariant feature extraction by the ROI Transformer, and logistic regression calculation by the regression prediction unit, the detection result including the target prediction classification and target regression position is output.

[0014] Preferably, in step 1, YOLOv4 is used to perform target detection on the aerial image to determine the ROI area, and the spatial distances of all ROI areas are calculated to merge adjacent ROI areas to obtain multiple concentrated detection areas, and the concentrated detection areas are cropped out from the aerial image as input to the ESRGAN model.

[0015] Preferably, in step 2, the ESRGAN model is pre-trained before being applied, and the pre-training process includes: screening high-resolution images from an aerial photography dataset with a short aerial photography distance, and downsampling the high-resolution images to use the downsampling results as sample data to optimize the model parameters of the ESRGAN model.

[0016] Preferably, step 3 comprises:

[0017] Perform feature extraction on images in the classified image dataset, and divide the obtained feature vector into multiple sub-region features. For each detection target class, calculate the prediction score of each sub-region feature through the CAM algorithm, perform threshold screening on the prediction score, and convert each sub-region feature whose prediction score exceeds the threshold into an n-dimensional vector representation;

[0018] Spectral clustering is performed on the n-dimensional vector representations corresponding to all sub-region features to divide them into classes of interest and irrelevant classes. All n-dimensional vector representations corresponding to the classes of interest are averaged, and the average result is used as the category feature of the detection target class.

[0019] Preferably, when spectral clustering is used to cluster the n-dimensional vector representations corresponding to the features of all sub-regions, the Lanczos method is used to solve the eigenvalues ​​of the standardized Laplace matrix. When solving, the standardized Laplace matrix is ​​iteratively transformed into a sparse tridiagonal matrix through an orthogonal similarity transformation to achieve fast eigenvalue decomposition, that is, QR and SVD decomposition.

[0020] Preferably, a VGG model is used to extract features from images in a classification image dataset.

[0021] Preferably, in step 4, the target detection model is pre-trained before being applied, and the training process includes:

[0022] Extract the concentrated detection area from the aerial image sample in the manner of step 1. For each concentrated detection area, calculate the overlap rate between the concentrated detection area and the original labeled object in the aerial image sample. When the overlap rate is 1, the coordinates of the original labeled object minus the coordinates of the concentrated detection area are the label coordinates of the labeled object in the concentrated detection area, that is, the label of the concentrated detection area is obtained; when the overlap rate is greater than the preset threshold and less than 1, calculate the coordinate difference between the coordinates of the original labeled object and the coordinates of the concentrated detection area, and take the maximum value between the coordinate difference and zero as the label coordinates of the labeled object in the concentrated detection area, that is, the label of the concentrated detection area is obtained;

[0023] The high-resolution image obtained by improving the resolution of the concentrated detection area through the ESRGAN model is used as the training sample, and the label of the concentrated detection area is used as the true value label of the training sample;

[0024] The category features of the target class corresponding to the training samples and the true value labels are input into the target detection model, and the detection results are output after calculation. The loss function is constructed based on the difference between the target prediction classification contained in the detection results and the true value label to update the parameters of the target detection model.

[0025] Preferably, the aerial image detection method also includes: step 6, determining the prediction result of the input aerial image based on the detection result corresponding to the concentrated detection area, including: adding the coordinate information corresponding to the detection result to the coordinate of the concentrated detection area to obtain the prediction result of the input aerial image.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] Based on the centralized detection area obtained from the aerial photos, the resolution of the centralized detection area is improved by the ESRGAN model. The super-resolution image with improved resolution and the category features corresponding to the detection target class are used as the input of the detection model. The detection model embedded with ROI Transformer is used to enhance the feature of the category features to realize target detection, which can greatly improve the accuracy of small target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a flow chart of a GAN and feature-enhanced aerial image detection method provided in an embodiment;

[0030] Figure 2 It is a schematic diagram of the structure of the target detection model provided in the embodiment. DETAILED DESCRIPTION

[0031] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0032] The problem of inaccurate recognition of small objects in aerial images due to low resolution and complex background of aerial images, low resolution of small objects in aerial images, and few features of small objects in aerial images is addressed. The embodiment provides a GAN and feature-enhanced aerial image detection method, such as Figure 1 As shown, the following steps are included:

[0033] Step 1: Obtain an aerial image and use YOLOv4 to perform ROI region detection on the aerial image, and merge the detected ROI regions into multiple centralized detection regions.

[0034] Because the aerial images are very large and contain a lot of redundant background in addition to small targets, directly sending them to ESRGAN for enlargement will incur unnecessary resource consumption and produce poor results. Therefore, a rough screening of the concentrated detection area is performed before the resolution is increased.

[0035] In the embodiment, YOLOv4 is used to detect the target on the aerial image to determine the ROI area, and the spatial distance of all ROI areas is calculated to merge adjacent ROI areas to obtain multiple concentrated detection areas, and the concentrated detection areas are cropped from the aerial image as the input of the ESRGAN model. Since the concentrated detection area is extracted from the aerial image, it is still a low-resolution image.

[0036] Step 2: Use the ESRGAN model to improve the resolution of the concentrated detection area to obtain a super-resolution image.

[0037] In the embodiment, the SRGAN model is a generative adversarial network (GAN) for image super-resolution (SR), which is the first framework that can infer the effect of an image after being enlarged by 4 times. The ESRGAN model is an improvement based on SRGAN, a new model improved in three aspects: basic network structure, adversarial loss and perceptual loss, which can generate more realistic and natural textures than SRGAN.

[0038] Specifically, the ESRGAN model uses residual nested dense blocks RRDB instead of RB as the basic module of the network, with more layers and connections to improve network performance. The BN layer is removed to reduce computational costs (reduce learning parameters) and remove artifacts (when the statistical data of the test set is very different from that of the training set, the normalization operation will cause artifacts in the generated image).

[0039] The ESRGAN model also designs a relative discriminator to determine whether a real image is more realistic than a fake one, which helps the generator restore more realistic texture details. The generator loss function includes the gradient of the generated image and the real image in adversarial training. This adjustment will enable the network to learn more detailed textures.

[0040] The ESRGAN model also proposes a more effective perceptual loss. The original SRGAN used the activated VGG features, while ESRGAN chose to use the VGG features before activation, thus overcoming two shortcomings of the original design. First, the activated features are very sparse, resulting in performance degradation. Second, the use of activated features will also cause the reconstructed brightness to be inconsistent with the real image. The perceptual loss is proposed to enhance the detail information after super-resolution. Specifically, the fake high-resolution image generated by the generator and the real image are sent to the VGG19 network for feature extraction, and then the mean square error is calculated on the extracted feature map, so that a clearer small target image can be obtained.

[0041] In an embodiment, the ESRGAN model is pre-trained before being applied, and the pre-training process includes: when the experimental aerial photography dataset lacks high-resolution images, high-resolution images are screened from the aerial photography dataset of short aerial photography distances, and the high-resolution images are down-sampled to use the down-sampling results as sample data to optimize the model parameters of the ESRGAN model.

[0042] The concentrated detection area is input into the ESRGAN model with optimized model parameters. After calculation, the low-resolution concentrated detection area can be enlarged by four times to obtain a super-resolution image with improved resolution. This can solve the problem of inaccurate recognition of small targets due to low resolution of aerial images.

[0043] Step 3: Determine the category features corresponding to the detection target class from the classification image data set using the CAM algorithm, threshold screening and spectral clustering method.

[0044] Since the resolution of small targets in aerial images is low, the useful information (image features) contained is relatively small, making accurate detection more difficult. Therefore, the embodiment adopts a feature enhancement method, by pre-extracting useful category features, and inputting the category features as additional features into the target detection model for target detection.

[0045] In an embodiment, first, feature extraction is performed on images in a classified image data set. Specifically, a region with relatively active activation values ​​during classification using a network such as VGG can be used to achieve feature extraction, and the extracted feature vector is divided into multiple sub-region features. For example, the feature vector can be divided into multiple 5*5 sub-region features.

[0046] Among them, the classification image data set contains images of various types of targets. These images have high resolution and clear targets. Feature extraction is performed on these images, and the obtained feature vectors contain more target class features.

[0047] Then, for each detected target class, the prediction score Score of each sub-region feature is calculated by the CAM algorithm, where the prediction score Score is the category activation value of each sub-region feature, representing the importance of this region.

[0048] The CAM algorithm extracts the input sub-region features and then linearly weights the features using global average pooling (GAP, where the window is the average pooling of the entire feature map) and the parameters of the last fully connected layer, thereby obtaining areas of focus for different categories. For a position of a sub-region feature, its score is the weight of the different parameters of the last fully connected layer as weights and the feature value of this position.

[0049] Next, the prediction score score is thresholded and each sub-region feature whose prediction score exceeds the threshold is converted into an n-dimensional vector representation.

[0050] Finally, spectral clustering is performed on the n-dimensional vector representations corresponding to all sub-region features to divide them into classes of interest and irrelevant classes. All n-dimensional vector representations corresponding to the classes of interest are averaged, and the average result is used as the category feature of the detected target class.

[0051] The traditional spectral clustering algorithm process is as follows:

[0052]

[0053] The summary steps are:

[0054] 1. Construct the dot product of the sample similarity matrix W;

[0055] 2. Construct the degree matrix D;

[0056] 3. Calculate L = DW and solve the eigenvalues ​​and corresponding eigenvectors of the normalized Laplace matrix X = D-1 / 2L D1 / 2;

[0057] 4. Set the number of clusters k and select the eigenvectors with the smallest k eigenvalues ​​to form Un*k;

[0058] 5. Use k-means clustering to divide each row of U into different clusters.

[0059] In the embodiment, spectral clustering is performed on the n-dimensional vector representations corresponding to the features of all sub-regions, and an improved spectral clustering algorithm is used, that is, the Lanczos method is used to solve the eigenvalues ​​of the standardized Laplace matrix X. When solving, the standardized Laplace matrix X is continuously iterated through orthogonal similarity transformation to be converted into a sparse tridiagonal matrix, so as to achieve fast eigenvalue decomposition, that is, QR and SVD decomposition.

[0060] Step 4: Based on FasterR-CNN including a feature extraction unit and a regression prediction unit, ROITransformer is embedded between the feature extraction unit and the regression prediction unit to form an object detection model.

[0061] Figure 2 Schematic diagram of the structure of the target detection model provided in the embodiment. Figure 2 As shown in the figure, the target detection model includes a feature extraction unit, a regression prediction unit and a ROI Transformer, wherein the feature extraction unit and the regression prediction unit constitute FasterR-CNN. The feature extraction unit, as the feature extraction part of FasterR-CNN, is used to extract image features of input data, and the regression prediction unit is used to perform regression prediction of the target according to the image features.

[0062] Based on the fact that small objects in aerial images are arbitrarily oriented, objects in aerial images are usually arbitrarily oriented from a bird's-eye view. If represented by horizontal bounding boxes, crowded object instances are difficult to distinguish. Aircraft are all oriented at 45 degrees, and when detected by horizontal bounding boxes, the detection boxes will overlap and be indistinguishable. Therefore, accurate aerial image detection can be modeled as a directional object detection task.

[0063] The embodiment embeds the RoI Transformer into FasterR-CNN, specifically, between the feature extraction unit and the regression prediction unit, to solve the oriented detection problem in aerial images, learns the transformation parameters under the supervised learning of the rotated bounding box to obtain the rotated RoI, and realizes efficient detection of small aerial targets.

[0064] In the embodiment, RoI Transformer is composed of an RROI Learner and RROI warping, and RROILearner is used to learn the rotation ROI parameters, including x, y, w, h, θ, where x, y are the rotation center positions, w, h are the width and height of the bounding box, and θ is the rotation angle. RROI warping is used to obtain the corresponding rotation invariant feature vector according to the learned rotation RRoI parameters and the input feature vector, that is, to sample the input directional vector according to the rotation ROI parameters, perform a rotation transformation to obtain the point of the rotation position, so as to extract the features of the rotation position, that is, to obtain the rotation invariant feature vector, which is used for regression and classification of the subsequent regression prediction unit.

[0065] In the embodiment, the target detection model is pre-trained before being applied, and the training process includes:

[0066] Extract the concentrated detection area from the aerial image sample in the manner of step 1. For each concentrated detection area, calculate the overlap rate between the concentrated detection area and the original labeled object in the aerial image sample. When the overlap rate is 1, the coordinates of the original labeled object minus the coordinates of the concentrated detection area are the label coordinates of the labeled object in the concentrated detection area, that is, the label of the concentrated detection area is obtained; when the overlap rate is greater than the preset threshold and less than 1, calculate the coordinate difference between the coordinates of the original labeled object and the coordinates of the concentrated detection area, and take the maximum value between the coordinate difference and zero as the label coordinates of the labeled object in the concentrated detection area, that is, the label of the concentrated detection area is obtained;

[0067] The high-resolution image obtained by improving the resolution of the concentrated detection area through the ESRGAN model is used as the training sample, and the label of the concentrated detection area is used as the true value label of the training sample;

[0068] The category features of the target class corresponding to the training samples and the true value labels are input into the target detection model, and the detection results are output after calculation. The loss function is constructed based on the difference between the target prediction classification contained in the detection results and the true value label to update the parameters of the target detection model.

[0069] Step 5: Concatenate the super-resolution image and the category features and input them into the target detection model. After feature extraction by the feature extraction unit, rotation invariant feature extraction by the ROI Transformer, and logistic regression calculation by the regression prediction unit, the detection result including the target prediction classification and target regression position is output.

[0070] In an embodiment, when a target detection model is used to calculate super-resolution images and category features, feature enhancement is performed based on the super-resolution image and the category features to specifically identify the target class, predict the category of the target object in the image, and simultaneously regress and locate the position of the target object, thereby effectively improving the detection performance of small targets of the target class in the aerial image.

[0071] In an embodiment, the detection method further includes: determining a prediction result of the input aerial image based on a detection result corresponding to the concentrated detection area, including: adding coordinate information corresponding to the detection result to the coordinates of the concentrated detection area to obtain a prediction result of the input aerial image.

[0072] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for aerial image detection based on GAN and feature enhancement, characterized in that: The following steps are involved: Step 1: Obtain an aerial image and use YOLOv4 to perform ROI region detection on the aerial image, and merge the detected ROI regions into multiple centralized detection regions; Step 2: Use the ESRGAN model to improve the resolution of the concentrated detection area to obtain a super-resolution image; Step 3, using the CAM algorithm, threshold screening and spectral clustering method to determine the category features corresponding to the detection target class from the classification image data set; Step 4: Based on FasterR-CNN including a feature extraction unit and a regression prediction unit, ROITransformer is embedded between the feature extraction unit and the regression prediction unit to form an object detection model; Step 5: Concatenate the super-resolution image and the category features and input them into the target detection model. After feature extraction by the feature extraction unit, rotation invariant feature extraction by the ROI Transformer, and logistic regression calculation by the regression prediction unit, the detection result including the target prediction classification and target regression position is output.

2. The aerial image detection method based on GAN and feature enhancement according to claim 1, characterized in that: In step 1, YOLOv4 is used to detect targets in the aerial image to determine the ROI area. The spatial distances of all ROI areas are calculated to merge adjacent ROI areas to obtain multiple concentrated detection areas, and the concentrated detection areas are cropped out from the aerial image as the input of the ESRGAN model.

3. The aerial image detection method based on GAN and feature enhancement according to claim 1, characterized in that: In step 2, the ESRGAN model is pre-trained before being applied, and the pre-training process includes: screening high-resolution images from the aerial photography dataset of short aerial photography distance, and downsampling the high-resolution images to use the downsampling results as sample data to optimize the model parameters of the ESRGAN model.

4. The aerial image detection method based on GAN and feature enhancement according to claim 1, characterized in that: Step 3 includes: Perform feature extraction on images in the classified image dataset, and divide the obtained feature vector into multiple sub-region features. For each detection target class, calculate the prediction score of each sub-region feature through the CAM algorithm, perform threshold screening on the prediction score, and convert each sub-region feature whose prediction score exceeds the threshold into an n-dimensional vector representation; Spectral clustering is performed on the n-dimensional vector representations corresponding to all sub-region features to divide them into classes of interest and irrelevant classes. All n-dimensional vector representations corresponding to the classes of interest are averaged, and the average result is used as the category feature of the detection target class.

5. The aerial image detection method based on GAN and feature enhancement according to claim 4, characterized in that: When spectral clustering is used to cluster the n-dimensional vector representations corresponding to the features of all sub-regions, the Lanczos method is used to solve the eigenvalues ​​of the standardized Laplace matrix. During the solution, the standardized Laplace matrix is ​​iteratively transformed into a sparse tridiagonal matrix through orthogonal similarity transformation to achieve fast eigenvalue decomposition, that is, QR and SVD decomposition.

6. The aerial image detection method based on GAN and feature enhancement according to claim 4, characterized in that: The VGG model is used to extract features from images in the classification image dataset.

7. The aerial image detection method based on GAN and feature enhancement according to claim 1, characterized in that: Step 4: The target detection model is pre-trained before being used. The training process includes: Extract the concentrated detection area from the aerial image sample in the manner of step 1. For each concentrated detection area, calculate the overlap rate between the concentrated detection area and the original labeled object in the aerial image sample. When the overlap rate is 1, the coordinates of the original labeled object minus the coordinates of the concentrated detection area are the label coordinates of the labeled object in the concentrated detection area, that is, the label of the concentrated detection area is obtained; when the overlap rate is greater than the preset threshold and less than 1, calculate the coordinate difference between the coordinates of the original labeled object and the coordinates of the concentrated detection area, and take the maximum value between the coordinate difference and zero as the label coordinates of the labeled object in the concentrated detection area, that is, the label of the concentrated detection area is obtained; The high-resolution image obtained by improving the resolution of the concentrated detection area through the ESRGAN model is used as the training sample, and the label of the concentrated detection area is used as the true value label of the training sample; The category features of the target class corresponding to the training samples and the true value labels are input into the target detection model, and the detection results are output after calculation. The loss function is constructed based on the difference between the target prediction classification contained in the detection results and the true value label to update the parameters of the target detection model.

8. The aerial image detection method based on GAN and feature enhancement according to claim 1, characterized in that: It also includes: step 6, determining the prediction result of the input aerial photo according to the detection result corresponding to the concentrated detection area, including: adding the coordinate information corresponding to the detection result to the coordinate of the concentrated detection area to obtain the prediction result of the input aerial photo.

Images