Unmanned aerial vehicle small target detection method based on unsupervised adversarial learning

Abstract

The invention discloses an unmanned aerial vehicle small target detection method based on unsupervised adversarial learning, and relates to the technical field of small target detection. According to the method, image multi-scale degradation and enhancement are introduced, so that a detection model can better learn multi-element structures and modes in data, and a small target can be better identified. In this way, the model can assist a target detection framework in learning semantic features with higher identification degree and universality for small target recognition. A generative adversarial network is introduced into a target detection task, a feature extractor is used to replace a generator, corresponding feature maps are generated from an image background and a synthetic image through the feature extractor, the two different feature maps are transmitted into a discriminator at the same time, and the discriminator can discriminate and distinguish the differences of the two feature maps. The discriminator can focus on learning the difference of the two feature maps in continuous adversarial learning, and is used for carrying out precise positioning detection on a small target object in the composite image feature map.

Description

Technical Field

[0001] The present invention relates to the technical field of small target detection, and in particular to a method for detecting small targets of unmanned aerial vehicles based on unsupervised adversarial learning. Background Art

[0002] Drone target detection is an important branch of computer vision target detection. With the continuous advancement of drone technology, the scope of drone use has continued to expand, involving military, civil, commercial and other fields. The purpose of drone target detection is to locate, detect and classify drones from the images or videos to be inspected. However, the potential threats of drones have gradually become apparent, such as illegal intrusions by drones and malicious attacks by drones. Nowadays, drone target detection based on machine vision has been widely used in daily life, such as airports, confidential places, video surveillance and other fields.

[0003] Early object detection mainly relied on manually designed features and machine learning-based classifiers. Typical methods include Haar features and cascade classifiers (such as the Viola-Jones algorithm), as well as methods based on HOG (Histogram of Oriented Gradients) features. However, these methods perform poorly in complex backgrounds, multiple targets, and scale changes. With the continuous development of computer vision, the emergence of various detection algorithms has changed the above situation. R-CNN (Region with CNN feature) is one of the pioneering works using deep learning methods. It proposes to locate objects in images through region proposals, then use convolutional neural networks (CNN) to extract features for each proposal, and finally classify them. Fast R-CNN then improved R-CNN by integrating the entire detection process into a single network to improve speed and efficiency, and using RoI (Region of Interest) pooling layers to extract features from shared feature maps. Today, single-stage object detectors such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector) directly predict the category and location of the target through a single forward propagation process, greatly improving the detection speed. As an efficient one-stage algorithm, the YOLO-v5 network has the characteristics of strong generalization performance and fast inference speed.

[0004] At present, a large number of drone target detection algorithms, such as YOLO-v5, EfficientDet and other improved algorithms, have achieved good results in detecting drone targets. The essence of these algorithms is supervised learning. There are still some shortcomings in supervised learning target detection algorithms. For example, supervised learning algorithms usually require a large amount of labeled data for training, especially in the field of deep learning. The adaptability of supervised target detection algorithms to target occlusion and different postures is limited. During training, supervised target detection algorithms can usually only learn patterns in labeled data, and their generalization ability may be poor for targets in new fields or different environments. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for detecting small targets of unmanned aerial vehicles based on unsupervised adversarial learning to improve the accuracy of target detection of unmanned aerial vehicles.

[0006] The objective of the present invention is achieved through the following technical solutions: A method for detecting small targets of drones based on unsupervised adversarial learning, comprising: Get the original image containing the small drone target in the training set, extract the rectangular area containing the detection target in the original image, and obtain the detection target image and the pure background image without the detection target; Performing image degradation and enhancement operations on the detection target image, including: performing nearest neighbor interpolation, bilinear interpolation and bicubic interpolation on the detection target image respectively, and then randomly selecting an interpolation method for the obtained three groups of images with different degradation degrees to perform secondary degradation, so as to obtain another three groups of images with different degradation degrees, a total of six groups of different degraded images; performing multiple modes of enhancement processing on the obtained degraded images, so as to obtain multiple groups of degraded and enhanced images of the detection targets; Construct a small target detection model, including a feature extraction network and an unsupervised target discrimination network; the feature extraction network as a generator includes a feature extractor composed of a YOLO-v5 backbone network, a feature adapter composed of multiple fully connected layers and / or a multi-layer perceptron MLP; the unsupervised target discrimination network as a discriminator includes a binary classification discriminator; Training the small target detection model, i.e., conducting adversarial learning, includes: fusing the degraded enhanced image of the detection target with the pure background image to obtain a synthetic image, inputting the synthetic image and the pure background image into a feature extraction network respectively, generating an abnormal feature map and a normal feature map respectively; inputting the obtained abnormal feature map and normal feature map into a discriminator, the discriminator distinguishes the difference between the two feature maps, and judges the abnormal feature map as false and the normal feature map as true; Select the target image to be detected in the test set and input it into the trained small target detection model to output the UAV small target detection result.

[0007] Furthermore, the enhancement processing includes random scaling and translation transformation operations, image saturation and brightness adjustment operations, and fusion of two images with a certain degree of transparency.

[0008] Furthermore, the feature extractor includes: two Focus structures and two CSP structures, the Focus structure is used to perform a slice operation on the image before semantic feature extraction, and for the four similar complementary images obtained, the width W and height H information are concentrated in the channel space, and the input channel is expanded by 4 times, that is, the spliced ​​image becomes 12 channels relative to the original RGB three channels, and finally the obtained new image is subjected to a convolution operation to finally obtain a double down-sampled feature map without information loss; The CSP structure is used to split the feature map into two parts, one part is convolved, and the other part is concatenated with the result of the convolution operation of the previous part, and then multiple convolution operations are performed to extract the multi-layer semantic features of the processed feature map.

[0009] Furthermore, the binary classification discriminator is specifically a two-layer MLP structure, which is used as a normality scorer to directly estimate each position Normality of , where h is the height parameter of the image block location, and w is the width parameter of the image block location; The normality estimation process is expressed as: ,in, is a normality scorer that directly estimates each position The normality of is the adaptive feature after transformation.

[0010] Furthermore, the loss function of the discriminator adopts a binary cross entropy loss function, which is used to quantify the accuracy of the discriminator in identifying the mapped image features and the synthetic image features. The lower the value, the better the performance. The calculation formula of the discriminator loss function is as follows: Among them, G is the generator, D is the discriminator, x represents real data, y represents synthetic data, G(x) represents the normal feature map, G(y) represents the abnormal feature map, D[G(x)] represents the discriminator's judgment result on the normal feature map, and D[G(y)] represents the discriminator's judgment result on the abnormal feature map.

[0011] The beneficial effects of the present invention are: (1) The present invention introduces multi-scale image degradation and enhancement into the target detection algorithm, so that the model can better learn the multivariate structure and pattern in the data, and the detection algorithm can better identify small targets. In this way, the model can assist the target detection framework to learn more recognizable and universal semantic features for small target recognition. In addition, enhancing the degraded targets in multiple modes can not only make the training data as close as possible to the real distribution data, but also avoid problems such as sample imbalance, forcing the model to learn more robust features, thereby effectively improving the generalization ability of the model.

[0012] (2) The present invention introduces generative adversarial networks into the target detection task, uses a feature extractor instead of a generator, generates corresponding feature maps for the image background and the synthetic image through the feature extractor, and simultaneously transmits the two different feature maps to the discriminator. The discriminator will identify and distinguish the differences between the two feature maps, and the feature extractor will also continuously learn to strengthen the extraction of image features. The discriminator can focus on learning the differences between the two feature maps in the continuous adversarial learning, which is used to locate and detect small target objects in the synthetic image feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall network architecture diagram of the present invention; Figure 2 Schematic diagram of multi-scale image degradation and enhancement; Figure 3 Schematic diagram of adversarial learning between mapping features and image features. DETAILED DESCRIPTION

[0014] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0015] See also Figure 1-Figure 3 , the present invention provides a technical solution: A method for detecting small targets of drones based on unsupervised adversarial learning, comprising: S1. Obtain the original image containing the small drone target in the training set, extract the rectangular area containing the detection target in the original image, and obtain the detection target image and the pure background image without the detection target; Small target objects are usually small in size, which makes them have relatively few pixels in the image. The target object may be submerged by the adjacent background pixels, and the detection algorithm can easily identify the non-target area as a small target. The detection algorithm needs more and more discriminative features to distinguish small target objects from the background. The purpose of image multi-scale degradation and enhancement is to enable the model to fully learn the multivariate and multi-scale data features of small target objects of different scales and modalities.

[0016] S2. Perform image degradation and enhancement operations on the detection target image, including: performing nearest neighbor interpolation, bilinear interpolation and bicubic interpolation on the detection target image respectively, and then randomly selecting an interpolation method for the obtained three groups of images with different degradation degrees to perform secondary degradation, so as to obtain another three groups of images with different degradation degrees, a total of six groups of different degraded images; performing multiple modes of enhancement processing on the obtained degraded images to obtain multiple groups of degraded and enhanced images of the detection targets; the schematic diagram of the process of multi-scale image degradation and enhancement is shown in FIG. Figure 2 shown.

[0017] The enhancement processing includes random scaling and translation transformation operations, adjusting the saturation and brightness of the image, and merging two images with a certain degree of transparency.

[0018] In this embodiment, three different degradation scales and three degradation methods are used to generate a total of nine groups of different degraded images, generate diverse training samples, and complete the expansion and enrichment of the training data set, so that the detection model can learn more abundant and comprehensive target features. In actual scenes, small target detection is often affected by multiple factors, such as low resolution, occlusion, and illumination changes, which lead to unclear representation of the target in the image. The use of multi-scale image degradation can simulate this situation, so that the model can be exposed to more diverse data during the training process, so as to better adapt to various complex detection scenes and make the model better adapt to different scene changes.

[0019] By using the training sample data of the existing model and utilizing data augmentation technology to generate more training data, the expanded training data can be closer to the real distribution data. The data augmentation method uses a random combination of color channel transformation, image stitching, image scaling, and image blurring. This can force the model to learn more robust features, thereby effectively improving the generalization ability of the model.

[0020] S3, build a small target detection model, such as Figure 1 , including a feature extraction network and an unsupervised target discrimination network; the feature extraction network as a generator includes a feature extractor composed of a YOLO-v5 backbone network, a feature adapter composed of multiple fully connected layers and / or a multi-layer perceptron MLP; the unsupervised target discrimination network as a discriminator includes a binary classification discriminator; Specifically, the feature extractor includes: two Focus structures and two CSP structures. The Focus structure is used to slice the image before semantic feature extraction. For four similar complementary images, the width W and height H information are concentrated in the channel space. The input channel is expanded by 4 times, that is, the spliced ​​image becomes 12 channels relative to the original RGB three channels. Finally, the obtained new image is subjected to a convolution operation to finally obtain a double down-sampled feature map without information loss. The CSP structure is used to split the feature map into two parts, one part is convolved, and the other part is concatenated with the result of the convolution operation of the previous part, and then multiple convolution operations are performed to extract the multi-layer semantic features of the processed feature map.

[0021] In some specific embodiments, the feature extractor is composed of a number of convolutional layers, pooling layers, activation functions, batch normalization layers, activation functions, global average pooling layers and fully connected layers. The shallow semantic information of the image can be refined layer by layer through the feature extractor to obtain the deep and high-level semantic information of the image to serve the subsequent image target detection steps.

[0022] S4, training the small target detection model, i.e., performing adversarial learning, including: fusing the degraded enhanced image of the detection target with the pure background image to obtain a synthetic image, inputting the synthetic image and the pure background image into a feature extraction network respectively, generating an abnormal feature map and a normal feature map respectively; inputting the obtained abnormal feature map and normal feature map into a discriminator, the discriminator distinguishes the difference between the two feature maps, and judges the abnormal feature map as false, and the normal feature map as true; The feature extractor extracts features from the synthetic image to generate synthetic image features; the feature adapter is used to project the local features of the pure background image obtained by the feature extractor to the adaptive features during the training phase, so that the training features are transferred to the target domain to generate mapped image features.

[0023] The binary classification discriminator is specifically a two-layer MLP structure, which is used as a normality scorer to directly estimate each position Normality, where h is the height parameter of the image block location, and w is the width parameter of the image block location; the normality estimation process is expressed as: ,in, is a normality scorer that directly estimates each position The normality of is the transformed adaptive feature. Since negative samples are generated together with normal features, they are all input to the discriminator during training. The discriminator expects positive outputs for normal features and negative outputs for abnormal features.

[0024] The unsupervised target discriminant network sends abnormal feature maps (synthetic image features) and normal feature maps (mapped image features) to the discriminator, which continuously learns how to better distinguish between the two types of images. At the same time, the generator is also learning how to generate more realistic images to deceive the discriminator. In this process, the discriminator loss function calculates the performance of the discriminator in distinguishing between the two images.

[0025] In this embodiment, the loss function of the discriminator adopts a binary cross entropy loss function, which is used to quantify the accuracy of the discriminator in identifying the mapped image features and the synthetic image features. The lower the value, the better the performance. The calculation formula of the discriminator loss function is as follows: Among them, G is the generator, D is the discriminator, x represents real data, y represents synthetic data, G(x) represents the normal feature map, G(y) represents the abnormal feature map, D[G(x)] represents the discriminator's judgment result on the normal feature map, and D[G(y)] represents the discriminator's judgment result on the abnormal feature map.

[0026] During the model training process, the generator and the discriminator learn in an adversarial manner. The generator attempts to generate samples similar to real data, while the discriminator attempts to distinguish the generated samples from the real samples. Through this adversarial learning process, the generator continuously improves the quality of the generated samples, and the discriminator also continuously improves the ability to identify the generated samples. The present invention introduces the generative adversarial network into the target detection task, uses a feature extractor instead of the generator, and generates corresponding feature maps for the image background and the synthetic image through the feature extractor, respectively, and passes the two different feature maps to the discriminator at the same time. The discriminator will identify and distinguish the differences between the above two feature maps, and the feature extractor will also continuously learn to strengthen the extraction of image features. Through the above adversarial learning, the feature extractor will have a stronger feature extraction ability, and the discriminator will have a stronger ability to distinguish and distinguish. The discriminator can focus on learning the differences between the two feature maps in the continuous adversarial learning, and can realize the positioning detection of small target objects in the feature map of the synthetic image.

[0027] S5. Select the target image to be detected in the test set and input it into the trained small target detection model, and output the UAV small target detection result.

[0028] The discriminator performs adversarial learning based on the mapped image features and the synthetic image features and is used for target detection. Figure 3 ,The trained model can focus on the target area according to the features of the target image to be detected, and realize the positioning and detection of small targets.

[0029] Since multi-scale image degradation and enhancement and feature extraction processes are added to the network training, the trained model is more capable of identifying the difference between targets and backgrounds, has strong adaptability to real-world images, high real-world generalization, and high robustness, which effectively improves the detection accuracy.

[0030] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A method for detecting small targets of unmanned aerial vehicles based on unsupervised adversarial learning, characterized by: include: Get the original image containing the small drone target in the training set, extract the rectangular area containing the detection target in the original image, and obtain the detection target image and the pure background image without the detection target; Performing image degradation and enhancement operations on the detection target image, including: performing nearest neighbor interpolation, bilinear interpolation and bicubic interpolation on the detection target image respectively, and then randomly selecting an interpolation method for the obtained three groups of images with different degradation degrees to perform secondary degradation, so as to obtain another three groups of images with different degradation degrees, a total of six groups of different degraded images; performing multiple modes of enhancement processing on the obtained degraded images, so as to obtain multiple groups of degraded and enhanced images of the detection targets; Construct a small target detection model, including a feature extraction network and an unsupervised target discrimination network; the feature extraction network as a generator includes a feature extractor composed of a YOLO-v5 backbone network, a feature adapter composed of multiple fully connected layers and / or a multi-layer perceptron MLP; the unsupervised target discrimination network as a discriminator includes a binary classification discriminator; Training the small target detection model, i.e., conducting adversarial learning, includes: fusing the degraded enhanced image of the detection target with the pure background image to obtain a synthetic image, inputting the synthetic image and the pure background image into a feature extraction network respectively, generating an abnormal feature map and a normal feature map respectively; inputting the obtained abnormal feature map and normal feature map into a discriminator, the discriminator distinguishes the difference between the two feature maps, and judges the abnormal feature map as false and the normal feature map as true; Select the target image to be detected in the test set and input it into the trained small target detection model to output the UAV small target detection result.

2. The method for detecting small targets of unmanned aerial vehicles based on unsupervised adversarial learning according to claim 1, characterized in that: The enhancement process includes random scaling and translation transformation operations, image saturation and brightness adjustment operations, and an operation of fusing two images together with a certain degree of transparency.

3. The method for detecting small drone targets based on unsupervised adversarial learning according to claim 1, characterized in that: The feature extractor includes: two Focus structures and two CSP structures. The Focus structure is used to slice the image before semantic feature extraction. For four similar complementary images, the width W and height H information are concentrated in the channel space. The input channel is expanded by 4 times, that is, the spliced ​​image becomes 12 channels relative to the original RGB three channels. Finally, the obtained new image is subjected to a convolution operation to finally obtain a double down-sampled feature map without information loss. The CSP structure is used to split the feature map into two parts, one part is convolved, and the other part is concatenated with the result of the convolution operation of the previous part, and then multiple convolution operations are performed to extract the multi-layer semantic features of the processed feature map.

4. The method for detecting small drone targets based on unsupervised adversarial learning according to claim 1, characterized in that: The binary classification discriminator is specifically a two-layer MLP structure, which is used as a normality scorer to directly estimate each position Normality, where h is the height parameter of the image block location, and w is the width parameter of the image block location; The normality estimation process is expressed as: ,in, is a normality scorer that directly estimates each position The normality of is the adaptive feature after transformation.

5. The method for detecting small drone targets based on unsupervised adversarial learning according to claim 1, characterized in that: The loss function of the discriminator adopts a binary cross entropy loss function, which is used to quantify the accuracy of the discriminator in identifying the features of the mapped image and the synthetic image. The lower the value, the better the performance. The calculation formula of the discriminator loss function is as follows: Among them, G is the generator, D is the discriminator, x represents real data, y represents synthetic data, G(x) represents the normal feature map, G(y) represents the abnormal feature map, D[G(x)] represents the discriminator's judgment result on the normal feature map, and D[G(y)] represents the discriminator's judgment result on the abnormal feature map.

Images