A target detection method in high-resolution aerial images

Abstract

The application discloses a target detection method in high-resolution aerial images, constructs a first training picture set, and the initial training pictures in the first training picture set all contain personnel class targets; a target detector is used to perform target identification on the initial training pictures; a target frame that is not recognized by the target detector in the initial training pictures is selected, and a target instance library is generated according to the target frame; pseudo samples in the target instance library are implanted into the initial training pictures to generate a second training picture set; the target detector is trained based on the second training picture set; the trained target detector is used to perform target detection on high-resolution aerial images; by implanting pseudo samples into the initial training pictures that are not recognized by the target detector and training the target detector by using the final training images with the pseudo samples, the detection precision of the target detector can be greatly improved, the false detection rate and the missed detection rate can be reduced, and meanwhile, the demand for the final training pictures in the training process can be reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target recognition, and particularly relates to a target detection method in high-resolution aerial images. BACKGROUND

[0002] Aerial image target detection is one of the research hotspots and key technologies in the field of remote sensing. Unmanned aerial vehicles are widely used in military reconnaissance, civilian search and rescue, and monitoring and inspection due to their convenience, low cost, and high efficiency. With the significant improvement of unmanned aerial vehicle platforms and computer performance, a large number of unmanned aerial vehicles can obtain massive aerial images, which contain very rich geographical and image information and have important research significance and application value. In recent years, aerial image target detection has benefited from the great success of deep learning methods on natural images, and the unmanned aerial vehicle target detection algorithm has been improved and optimized through transfer learning.

[0003] Small targets are widely present in aerial images, and small target detection and recognition have been a difficulty and research hotspot in computer vision for a long time. On the target detection public dataset MS-COCO, the detection performance of small targets is usually only half of that of large and medium targets.

[0004] Existing target detection methods include YOLT, sliding window execution detection, and NMS method for merging and reasoning, ClusterNet, DMNet, SAHI, etc. However, these detection methods have low detection accuracy, high false detection rate, and high missing detection rate. SUMMARY

[0005] The purpose of the present application is to provide a target detection method in high-resolution aerial images to improve the detection accuracy of personnel class small targets in unmanned aerial vehicle aerial images and reduce the false detection rate and missing detection rate.

[0006] The present application adopts the following technical solution: a target detection method in high-resolution aerial images, comprising the following steps:

[0007] A first training picture set is constructed, and the initial training pictures in the first training picture set all contain personnel class targets;

[0008] A target detector is used to identify the targets in the initial training pictures;

[0009] The target boxes in the initial training pictures that are not recognized by the target detector are selected, and a target instance library is generated according to the target boxes;

[0010] Pseudo samples in the target instance library are implanted into the initial training pictures to generate a second training picture set;

[0011] The target detector is trained based on the second training picture set;

[0012] The trained target detector is used to detect targets in high-resolution aerial images.

[0013] Further, generating the target instance library according to the target frame comprises:

[0014] calculating a mask map of the target frame;

[0015] selecting a pseudo sample in the target frame according to the mask map.

[0016] Further, implanting the pseudo sample in the target instance library into the initial training picture comprises:

[0017] implanting the pseudo sample into a target position of the initial training picture according to the quantity threshold and the pseudo sample size threshold; wherein the target position is a position outside the existing target frame in the initial training picture.

[0018] Further, training the target detector based on the second training picture set comprises:

[0019] respectively constructing a first loss function of the pseudo sample prediction frame and a second loss function of the original sample of the picture, and performing weighted summation on the first loss function and the second loss function to obtain a total loss function.

[0020] Further, training the target detector based on the second training picture set further comprises:

[0021] segmenting the final training picture in the second training picture set using a density estimation network to obtain a plurality of cropped sub-pictures;

[0022] using the target detector to identify targets in the cropped sub-pictures and the final training picture;

[0023] updating the target detector and the density estimation network using the total loss function according to the target identification result.

[0024] Further, using the target detector to identify targets in the cropped sub-pictures and the final training picture comprises:

[0025] using the target detector to identify targets in the cropped sub-pictures to obtain a first anchor frame;

[0026] using the target detector to identify targets in the final training picture to obtain a second anchor frame corresponding to the position of the first anchor frame;

[0027] selecting the first anchor frame or the second anchor frame as a detection anchor frame of the corresponding target based on a non-maximum suppression method.

[0028] Further,

[0029] when using the target detector to identify targets in the cropped sub-pictures, determining the first anchor frame based on a maximum mean difference distance method;

[0030] When the target detector is used to recognize the target in the final training picture, the second anchor frame is determined based on a maximum mean difference distance method.

[0031] Further, the target detector comprises a plurality of target detection models with different scales.

[0032] Further, the construction of the first training picture set comprises:

[0033] Obtain a public picture dataset;

[0034] Label cleaning is performed on the public picture dataset to remove label information other than personnel class targets.

[0035] Another technical solution of the present application: a target detection device in a high-resolution aerial image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the above-mentioned target detection method in a high-resolution aerial image.

[0036] The present application has the beneficial effects that: the present application implants pseudo samples into the initial training pictures that are not recognized by the target detector, and trains the target detector through the final training images with pseudo samples, which can greatly improve the detection accuracy of the target detector, reduce the false detection rate and the missed detection rate, and also reduce the demand for the final training pictures in the training process. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The figure is a schematic diagram of the target detection method of the embodiment of the present application;

[0038] Figure 2 The figure is a schematic diagram of the target detection method of the embodiment of the present application;

[0039] Figure 3 The figure is a schematic diagram of the personnel class detection dataset annotation frame in the embodiment of the present application;

[0040] Figure 4 The figure is a schematic diagram of the target instance segmentation and target implantation method framework used in the embodiment of the present application;

[0041] Figure 5 The figure is a schematic diagram of the non-uniform image cropping strategy framework used in the embodiment of the present application;

[0042] Figure 6 The figure is a schematic diagram of the model training process based on the MMD evaluation method in the embodiment of the present application;

[0043] Figure 7 The figure is a schematic diagram of the visual detection result under the Heridal test set in the embodiment of the present application. DETAILED DESCRIPTION

[0044] The application will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0045] In large optical remote sensing images, the difficulty of small target detection is further increased due to the characteristics of large texture difference, low contrast, few pixels, severe scale transformation, arbitrary dense distribution of direction and ultra-fine occlusion of small targets. In particular, it is challenging to automatically, accurately and efficiently detect non-rigid personnel small targets containing only a few pixels in aerial images.

[0046] To meet the application requirements of personnel detection and search and rescue in aerial images, it is necessary to consider data source acquisition, efficient detection reasoning and model evaluation at the same time. With the continuous deepening of the application field of neural networks, the network that simply pursues high precision is no longer suitable for the scene that comprehensively considers optimization of resource limitation, low energy consumption and real-time performance. Therefore, when designing and deploying the method model, how to reduce the parameter quantity and the calculation quantity, meet the real-time requirement of reasoning, and still maintain high performance is a problem to be solved in neural network research.

[0047] In the prior art, YOLT cuts the original image into blocks, then inputs the model for detection respectively, and fuses the detection models of different scales, but it ignores the sparsity of personnel targets, resulting in poor model generalization ability and easy overfitting phenomenon; in the sliding window type execution detection and NMS way merging reasoning process, a large amount of computing resources are inefficiently applied to the regions with sparse or no objects; ClusterNet and DMNet introduce clustering operation and density estimation into the image cropping module, which greatly reduces the number of picture partitions for final target detection, thereby obtaining higher running time efficiency, but the real-time performance, portability and method deployment are poor; SAHI efficiently detects these smaller objects without retraining the model and without the need for larger GPU memory allocation, it detects after cutting the image into blocks, and then merges all the reasoning sub-images into a large image through non-maximum suppression, which is time-consuming and has boundary detection errors, which needs to be further optimized.

[0048] The application discloses a target detection method in high-resolution aerial images, as shown in Figure 1 and Figure 2 The method comprises the following steps: constructing a first training picture set, the initial training pictures in the first training picture set all contain personnel targets; using a target detector to identify the targets in the initial training pictures; selecting a target box in the initial training pictures that is not recognized by the target detector, and generating a target instance library according to the target box; implanting pseudo samples in the target instance library into the initial training pictures to generate a second training picture set; training the target detector based on the second training picture set; and using the trained target detector to detect targets in high-resolution aerial images.

[0049] The present application can greatly improve the detection accuracy of the target detector, reduce the false detection rate and the missed detection rate, and reduce the demand for final training pictures in the training process by implanting pseudo samples into the initial training pictures that are not recognized by the target detector, and training the target detector through the final training images with pseudo samples.

[0050] The method effectively balances the number of positive and negative samples by adopting a non-uniform image cropping strategy and alleviates the slight deviation of small target boxes by adopting a detection evaluation method based on maximum mean difference, and includes three parts: the first part is data set preprocessing and hyperparameter optimization, the second part is personnel class semantic segmentation model training and target implantation, and the third part is a non-uniform image cropping method, a target detector evaluation method, and network training and inference. Finally, an optimal model for personnel class small target detection under the perspective of a UAV is obtained.

[0051] In an embodiment of the present application, constructing the first training picture set comprises: obtaining a public picture data set; and performing label cleaning on the public picture data set to remove label information other than personnel class targets.

[0052] Specifically, the method amplifies personnel class samples in the Heridal data set, overcomes the low detection efficiency of traditional image cropping, optimizes the metric evaluation method of the small target detector, and obtains a detection model by training a deep learning network to detect small targets. The detection performance of personnel class small targets in high-resolution aerial images can be improved by the method.

[0053] The first part includes three steps:

[0054] Step 1: Obtain a personnel class aerial data set, select images with complex and changeable natural scenes, real and diverse targets, detailed instance labeling, and obvious illumination and occlusion influencing factors as the data set, wherein a part is used as a training set and another part is used as a test set, and the test set is not subjected to data enhancement and image transformation operations.

[0055] Step 2: Since the target detector in the present application includes multiple target detection models of different scales, the picture samples in the training set obtained in step 1 are normalized and batch processed, and the size is set to [640*640], [1280*1280] and [1536*1536] respectively. Data enhancement operations are performed on each picture in the training set to form a more complete training sample set.

[0056] Preferably, the specific implementation steps of step 2 are as follows:

[0057] Step 2.1: First, the data label cleaning in different public data sets in step 1 is performed, mainly including: merging [People, Pedestrian] labels into Person labels in VisDrone data set; removing the interference of other 8 categories [car, van, bus, truck, motor, bicycle, awning-tricycle, and tricycle]; and removing blurred data in the sample by using DeblurGAN method.

[0058] Step 2.2: Second, data augmentation operation is performed, and the basic data augmentation method in step 2 mainly includes image disturbance, changing brightness, contrast, saturation, hue, adding noise, random scaling, random cropping, flipping, rotation and random erasing, etc., and using Mosaic data augmentation method to splice 9 small pictures into one large picture after random cropping and scaling, to realize rich data set and increase small sample target.

[0059] Step 3: Perform the genetic evolution method of hyperparameters, extract the hyperparameters in the existing target detector to determine a basic parameter for mutation, determine the weight of each hyperparameter through the result obtained each time, obtain each hyperparameter and the weight of each hyperparameter, then use the evolved hyperparameters and their weights to continue to evolve other hyperparameters and corresponding hyperparameters, and finally obtain more accurate hyperparameters of the target detector.

[0060] Preferably, the specific implementation steps of step 3 are as follows:

[0061] Step 3.1: Randomly select a previous hyp as base hyp according to the weight of each hyp (hyperparameter), and then fuse all previous hyps to obtain the final evolution value according to the weight of each hyp.

[0062] Step 3.2: Record the result after each evolution, and when each evolution, the hyps are sorted from large to small according to the previous result, and then the weight of each hyp obtained by the previous evolution is calculated according to the fitness function to determine the evolution method and complete the evolution calculation.

[0063] The second part includes two steps:

[0064] Step 4: Using a top-down instance segmentation method, first detect each picture in the training sample set by a target detector (such as YOLOv5) to find the undetected target box and the region (Bounding Box) in the picture, and then perform semantic segmentation on the content in the target box corresponding to the region, thereby obtaining each small target of the personnel class in the picture, and taking these small targets of the personnel class as pseudo samples. For example, in a 4*4 target box, the small target of the personnel class is composed of 9 pixels, and the part composed of the 9 pixels is taken as a pseudo sample.

[0065] That is, in this process, the mask of the target box is first calculated; and the pseudo sample in the target box is selected according to the mask.

[0066] Preferably, the specific implementation steps of step 4 are as follows:

[0067] Step 4.1: Perform target detection on the initial training pictures in the training sample set to obtain target instances with high confidence and pseudo label information, wherein the target instance is the information in the picture; the pseudo label information refers to the size and position of the label box, and the confidence of the label box. Perform this operation on all pictures in the training sample set to construct a target instance library after detection.

[0068] Step 4.2: Use LabelMe or other labeling tools to construct a segmentation dataset, i.e., an instance set, from the samples in the target instance library, and correspond one-to-one with the pseudo label information obtained by detection.

[0069] Step 4.3: Add a segmentation head based on the YOLOv5 algorithm framework for semantic segmentation, and place the segmented images and labels in the segimages and seglabels folders. Each of the two folders has two subfolders, trian and val, and the subfolders store images (jpg, png) or labels (must be png, pixel value is class, ignore class value 255). Similar to the organization method of detection labels, modify the data source input configuration file and the hyperparameter settings to obtain an instance segmentation model.

[0070] Step 5: The target implantation module can implant the above-mentioned undetected pseudo samples into different positions in the pictures in the training sample set (which can be the original pictures or other pictures), and combine image smoothing and sharpening and other enhancement methods to make the implanted content fully integrated with the background picture.

[0071] It should be noted that the pseudo samples are implanted into the target positions of the initial training pictures according to the number threshold and the pseudo sample size threshold; wherein the target position is a position outside the existing target box in the initial training picture.

[0072] Specifically, the target implantation method in the embodiment of the present application implants pseudo samples in a non-target background region, ensures that the size of the target is less than 32*32 pixels in the background image, and the number can be controlled at about 3-5 according to the empirical value.

[0073] The third part includes two steps:

[0074] Step 6: The training sample picture is sent into the image cropping module, the density estimation network (i.e. the connected domain method) of the image cropping module is used to connect the regions with high pixel intensity, and a plurality of cropped sub-pictures are generated, and the position information of each cropped sub-picture in the sample picture is recorded. Then, the plurality of cropped sub-pictures and the corresponding sample picture are sent into the target detector (the target detector includes the cropping module and the preprocessing module) to obtain the trained target detector.

[0075] That is, the training of the target detector based on the second training picture set further includes: segmenting the final training picture in the second training picture set by using the density estimation network to obtain a plurality of cropped sub-pictures; performing target recognition on the cropped sub-pictures and the final training picture by using the target detector; and updating the target detector and the density estimation network according to the target recognition result by using the total loss function.

[0076] Preferably, the specific implementation steps of step 6 are as follows:

[0077] Step 6.1: The sample picture to be cropped in step 5 is input into the density estimation network. The network uses the MCNN network, uses branches of three different convolution kernels to correspond to different receptive fields, and captures multi-scale features. Two pooling layers are designed to shrink the output feature map to 1 / 4 of the image. In order to maintain the original resolution, the generated density map is recovered by cubic interpolation. For the image width and height which are not multiples of 4, directly scale to the original size.

[0078] Step 6.2: The result obtained in step 6.1 is used for the generation of the Ground Truth (GT) density map, and a Gaussian kernel is used to blur each labeled point to generate the GT density map.

[0079] Step 6.3: The appropriate image is cropped from the context information provided in the GT density map generated in step 6.2.

[0080] Step 6.4: First, detection is performed on the original sample picture and the cropped sub-picture. Then, the bounding boxes predicted from the cropped sub-picture are collected, and the detection results of the sample picture are fused. Finally, all the detection bounding boxes are used to calculate the final result by NMS. The threshold of NMS is set to 0.5.

[0081] It is worth noting in this step that the first anchor box is determined based on the Maximum Mean Discrepancy Distance (MMD) method when the target detector is used to identify the target in the cropped subgraph, and the second anchor box is determined based on the Maximum Mean Discrepancy Distance method when the target detector is used to identify the target in the final training picture.

[0082] That is, the first anchor box is obtained by using the target detector to identify the target in the cropped subgraph, and the second anchor box corresponding to the position of the first anchor box is obtained by using the target detector to identify the target in the final training picture.

[0083] In the fusion setting, the bounding boxes of the original detection results are not removed. As can be seen from the visualization results, the detection results of the sample pictures contain correctly detected large targets. Removing these detection results will reduce the detection performance of large targets and not fully display the performance of the target detector. Therefore, these detection boxes are retained during evaluation.

[0084] Step 7: The target detector trained in step 6 is used to combine the cropped subgraph and the detection results of the sample picture by the non-maximum suppression method to obtain the final detection results, and the final detection results are marked on the original picture; that is, the first anchor box or the second anchor box is selected as the detection anchor box of the corresponding target based on the non-maximum suppression method.

[0085] Preferably, the specific implementation steps of step 7 are as follows:

[0086] Step 7.1: For small targets, the bounding box usually contains some background pixels because most of the real targets are not strictly rectangular. In these bounding boxes, the foreground pixels and the background pixels are concentrated in the center and the boundary of the bounding box, respectively. In order to better describe the weight of different pixels in the bounding box, the bounding box can be modeled as two two-dimensional Gaussian distributions, where the center pixel has the highest weight, and the importance of the pixels from the center to the boundary decreases. Therefore, the bounding box is modeled as a two-dimensional Gaussian distribution.

[0087] Step 7.2: The similarity of the Gaussian distribution is measured by the normalized MMD distance, and the MMD distance has the advantage of measuring the similarity of the distribution even if the two bounding boxes do not overlap or contain each other. In addition, the MMD distance is not sensitive to targets of different scales, so it is more suitable for measuring the similarity between small targets.

[0088] Step 7.3: The anchor box mechanism algorithm YOLOv5 is used to prove the effectiveness of MMD, and the similarity of two mutually exclusive or mutually containing bounding boxes is measured.

[0089] In addition, a multi-scale fusion target detector is trained in the embodiment of the present application, and the target detector includes the target monitoring models corresponding to the three scales in step 2.

[0090] The basic parameter settings in the training process are as follows:

[0091] (1) In order to reduce the excessive redundant information generated by padding to the required square size in the preprocessing process, a non-square Rect training mode is adopted.

[0092] (2) Multi-scale training samples, randomly select a scale for training every certain iterations, can accept any size of picture as input, using small scale picture test speed will be fast, but the accuracy is low, using large scale picture test speed is slow, but the accuracy is high, enhance the robustness of the model.

[0093] (3) Set breakpoint resume training, warm up learning rate in the first few cycles of training, use the pre-set learning rate for training after the model slowly tends to be stable at a small learning rate, the initial learning rate is 0.01, and the final learning rate is 0.002, and the learning rate is decayed by using cosine annealing algorithm.

[0094] (4) Set different Batch_Size size according to the model depth and input picture size (expand the batch size to alleviate the problem of frequent gradient update due to limited facility configuration), during training, after completing a nominal batch, enter the back propagation process, automatically save the new model, and after completing an Epoch, enter the verification stage, calculate the precision, recall, mAP@0.5 and mAP@0.5:0.95 evaluation indexes, and evaluate the model performance in real time.

[0095] (5) Use CIoU Loss as the loss of the bounding box regression, binary cross entropy to calculate the class probability loss and target confidence loss, and set a class loss coefficient to balance the class loss, the coefficient is inversely proportional to the class frequency of the data set label, that is, the more the number of labels, the smaller the weight coefficient.

[0096] (6) Use automatic mixed precision to reduce the model size, and use stochastic gradient descent algorithm SGD as the optimizer.

[0097] Regarding the loss function of the target detector, a first loss function of the pseudo sample prediction frame and a second loss function of the original sample picture are constructed, the first loss function and the second loss function are weighted and summed to obtain a total loss function.

[0098] Specifically, the loss function is as follows:

[0099] (1)

[0100] As shown in formula (1), by the weight and adjusting the number of unreliable pseudo labels, the second loss function represents the offset of the original sample anchor frame, the first loss function represents the offset of the pseudo sample anchor frame, is the predicted anchor frame information, and is the real anchor frame information.

[0101] Finally, the multi-scale fusion detection model trained is used to infer and output all personnel class detection results in the to-be-detected image.

[0102] To sum up, since the above technical solutions are adopted, the beneficial effects of the present application are as follows:

[0103] 1. The present application adopts the idea based on image instance segmentation, finds the standard boundary in the Patch library, and then implants it into the existing data set sample, avoiding the influence of directly implanting the target into the background and the boundary problem, improving the data sample quantity and also improving the robustness of part of the personnel class target detection, such as different posture personnel class, different angle personnel class, personnel class at different heights, etc., and the applicable range is wider.

[0104] 2. The present application uses a non-uniform image cropping strategy to construct a dense personnel class sample library under different scales, effectively alleviating the problem of unbalanced positive and negative samples, and then trains a density map guided cropping model to extract the target dense region from the large image, and then performs fine detection on the target dense region, which can enhance the detection ability of small targets and effectively suppress repeated detection of non-target regions.

[0105] 3. Since MMD is used to evaluate the detection algorithm index, the problem of obvious mAP value oscillation caused by slight deviation of IoU boundary is avoided, and at the same time, the detection speed of the overall algorithm is faster and the running time is reduced.

[0106] As shown in formula (1), by the weight and adjusting the number of unreliable pseudo labels, the second loss function represents the offset of the original sample anchor frame, the first loss function represents the offset of the pseudo sample anchor frame, is the predicted anchor frame information, and is the real anchor frame information. Figure 1 As shown in formula (1), by the weight and adjusting the number of unreliable pseudo labels, the second loss function represents the offset of the original sample anchor frame, the first loss function represents the offset of the pseudo sample anchor frame, is the predicted anchor frame information, and is the real anchor frame information.

[0107] 1) The transfer learning method is used to realize the evolution of training parameters on the Heridal data set sample, and part of the instance segmentation mask sample is implanted in the limited personnel class instance background to form a pseudo label data set, and MMD distance matching is used instead of IoU measurement method.

[0108] The specific process diagram is shown in formula (1), which includes: Figure 2

[0109] ​(1.1) YOLOv5 adopts breakpoint hyperparameter evolution on Heridal training set, step-by-step freezing training each basic parameter, the depth of the pre-trained model is: YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x;

[0110] (1.2) Complete a model training task on Heridal training set, part of the parameter settings are as follows: Epoches = 300, Input_Size = 1536*1536, Batch_Size = 16, PretrainedModel = YOLOv5s, infer all samples on Heridal training set and test set and crop target instance region;

[0111] (1.3) Calculate IoU (set threshold value output result, optional IoU = 0.5, IoU = 0.75) after inference and save target instance label and real label, collect all unmatched labels to form a new small sample set.

[0112] (1.4) Find the best boundary of the target instance using LabelMe and other polygon labeling tools to form the target semantic segmentation data label. Put this data label and picture into the YOLOv5DS algorithm framework (add a semantic segmentation head to the original YOLOv5 algorithm), and the detection data label is the aspect ratio of the picture in (1.3) sample set, realize target instance segmentation in YOLOv5DS to obtain target data and mask information after instance segmentation.

[0113] (1.5) Randomly implant the samples in (1.4) in Heridal training set, at this time, the inserted position should not coincide with the original data label, and the size of the inserted target instance in the background picture should be less than 32*32 pixels, which belongs to the category of small targets defined in MS-COCO dataset.

[0114] (1.6) Crop all training set samples in (1.5) through a non-uniform cropping network, the sliding window size in the network can be set to: [128*128], [320*320], [640*640], crop subgraphs containing more target instances, and the cropped subgraphs in the same large graph can be spliced into new training samples by MOSAIC method.

[0115] (1.7) In the training stage, YOLOv5 framework is adopted, MMD distance matching is selected to measure the evaluation parameters, the BoundingBox can be connected as an elliptical equation, and mapped as a two-dimensional Gaussian distribution, and the similarity of two BoundingBoxes is calculated in Hilbert transform space.

[0116] (1.8) The loss function of YOLOv5 needs to add a pseudo-label loss. The method is similar to CIoU_Loss, except that the weight of the pseudo-samples is increased during training. The empirical values ​​of the weight ratio can be [0.1, 0.2, 0.3].

[0117] 2) Optimize existing public aerial photography personnel datasets [VisDrone, Heridal, TinyPerson, AFO] and combine them into a new dataset, VHTA, such as... Figure 3 As shown, this dataset contains the following features:

[0118] (2.1) As Figure 3 As shown in the first column, the samples include a wider variety of scenes (jungles, mountains, sea, seaside, streets, suburbs, campuses, etc.), more varied perspectives (oblique angle, overhead angle, horizontal angle, etc.), and diverse aerial photography altitudes (50 meters, 80 meters, 120 meters, etc.), providing a more massive amount of data for practical applications such as search and rescue.

[0119] (2.2) The data labels are more reasonable compared to, for example, Figure 3 The other datasets shown in the second column have larger label sizes in the ViSDone dataset and smaller labels in the Heridal dataset. To match the automatic calculation of anchor box size using the Kmeans method in the YOLOv5 algorithm, these data labels can be integrated to more accurately optimize the anchor box size.

[0120] (2.3) such as Figure 3 As shown in the third and fourth columns, the targets are more concentrated. The targets in the existing Heridal dataset are too scattered, which is not conducive to non-uniform image cropping. The target center points and the vertical and horizontal comparisons in the merged dataset are more concentrated.

[0121] 3) All sparse person instances in the Heridal dataset are already in the Patches library. A subset of these samples will be labeled and used as the training set, with all pixels being 80*80 pixels.

[0122] (3.1) Data labeling: When there are multiple objects of the same category in an image, all of them are labeled using the same category name; when a particle in an image is occluded, the group option in the labelme is used to depict the occluded particles separately and name them the same group. After labeling, the JSON label format is converted to the YOLOv5 recognition txt dataset format;

[0123] (3.2) Based on YOLOv5 multi-task model, at the cost of increasing a small amount of calculation and memory, target detection and semantic segmentation are completed at the same time (1024x512 input increases about 350MB, one BaseNet of the same size needs about 1.3GB, two single-task models independently input have additional delay). The model is trained on Heridal_Patch semantic segmentation dataset and target detection dataset converted from Heridal_Patch aspect ratio as label, and the results are as follows Figure 4 Instance segmentation library can accurately find the boundary of the target.

[0124] 4) The key of the target implantation algorithm is to generate a sufficient number of pseudo labels in unlabeled images to improve the prediction performance of the detection model.

[0125] (4.1) The patch library in Heridal (contains all personnel class instances in Heridal dataset, pixel size is 80*80) does not contain label data, and the aspect ratio of the image in the patch can be used as label content, but the pseudo label (Unrelible Ground Truth, U-GT) of part of the synthetic samples is obviously different from the real sample label (Ground Truth, GT), which needs to be improved.

[0126] (4.2) A highly reliable detection model is used to infer unlabeled images to obtain pseudo labels (U-GT), as shown in Figure 4 First, the detector infers the pre-trained model to obtain prediction samples with high confidence, and the prediction label and the real label are matched and calculated by IoU to obtain undetected target instances.

[0127] (4.3) Obtain the mask of the pseudo target through (3) to construct the target instance library.

[0128] (4.4) Randomly combine the targets in the instance library with the background to obtain pseudo sample instances, and then put the original samples and pseudo samples into the trainer to optimize the model parameters. However, this leads to a problem that most pixels may not be used due to their unreliability. This embodiment processes all predicted probability targets in a unified loss function, sets different weights on the unlabeled and labeled data accordingly, and uses a double-input layer to optimize the prediction entropy of the loss function between GT and unreliable labels.

[0129] 5) The high-resolution aerial image in (4) is cropped into non-overlapping blocks by a density estimation network, and the training label is mapped back to the cropped block, as shown in Figure 5 An image cropping strategy based on YOLOv5 is adopted:

[0130] (5.1) Generating density map of specific object by MCNN algorithm and class-wise-kernel density mask method. First, select all pixels with value 1 on the density mask, then connect the pixels into large candidate regions by eight-connected region. Finally, crop the original image using the outer rectangle of the candidate region, and filter out the image blocks with resolution lower than the density value to generate the cropped region.

[0131] (5.2) The original image first needs to pass through the density cropping network of the DMNet algorithm to generate a cropping frame with as many positive sample rectangles as possible. The sliding window used for cropping can be flexibly selected according to the density of the samples in the data set.

[0132] 6) MMD can be easily inserted into any anchor-based detector instead of IoU. Without loss of generality, this embodiment adopts YOLOv5 based on anchor box as shown in Figure 6 to prove the effectiveness of MMD. Specifically, all modifications are made in the three parts originally using IoU. Specifically as follows:

[0133] (6.1) Label assignment based on MMD, generate anchors of different scales and aspect ratios, and then assign binary labels to these anchors for training subsequent classification and regression. Positive labels will be assigned to two anchors: (i) when using GT box, the anchor has the highest MMD value, which should be greater than the set threshold; (ii) any GT box has a MMD value higher than the positive threshold. Correspondingly, if the MMD value is less than the negative threshold, it will be classified as a negative sample. In addition, anchors that are neither assigned to positive samples nor negative samples will not participate in training.

[0134] (6.2) NMS based on MMD, which is an integral part of target detection, is used to compress redundant prediction boxes, which uses IoU. First, sort the prediction boxes according to their values. Select the prediction box with the highest value, and delete the prediction boxes with high overlap. This process is applied to the remaining prediction boxes. However, the sensitivity of IoU to small targets makes the IoU value of a large number of prediction boxes below the threshold, resulting in many false positive predictions. Using MMD can overcome the scale sensitivity problem, which can be embedded into any small target detection method.

[0135] (6.3) MMD-based loss function, the Gaussian model of the prediction box and the Gaussian model of the real box are calculated by MMD distance to calculate their similarity, which can alleviate the position bias sensitivity problem.

[0136] 7) Model training and testing.

[0137] Due to the complex search and rescue scene, as shown in Figure 7 As shown in the figure, several representative real environments (sea surface, seaside, suburb, street, mountain, campus) are selected, among which the sea surface test case is derived from the AFO dataset, the seaside test case is derived from the TinyPerson dataset, and the other test cases are derived from the VisDrone2019 dataset, the basic model is obtained by migration training of YOLOv5x6, and the experimental platform is NVIDIA tesla v100. It can be found from the detection result that most small people in the middle and long distance can be detected, and when the target is blocked or crowded, as shown in Figure 7 (d) Scene, the detection effect is poor.

[0138] In summary, in view of the small size of personnel targets in aerial images, the small sample size and the long training time, a personnel search method and a detection model with high accuracy and strong robustness in high-resolution aerial images are explored. With the Heridal dataset as the main test case, a target implantation data enhancement method is proposed, the target distribution is represented by means of a density map for non-uniform cropping, multi-scale feature fusion is realized, and the MMD distance measurement evaluation method is used to improve the target detection performance. The experimental comparison and analysis on the YOLOv5 detection framework show that the method has higher detection accuracy of small personnel targets than other mainstream methods.

[0139] The present application firstly proposes a target implantation method based on image semantic segmentation for the small amount of personnel data samples in aerial images, optimizes and expands the training samples; secondly, a density map guided non-uniform image cropping method is used, a density map generation network is used to train a model with high robustness and predict the density map of each aerial image, a sliding window is used to collect the sum of pixel intensities on the density map, and the windows higher than the density threshold are connected to generate a cropped subgraph. Finally, in order to alleviate the problem that the slight deviation of the small target position leads to the sharp deterioration of the detection performance, a new anchor box detector evaluation metric is proposed, the MMD distance is used to measure the similarity of BBox instead of the standard IoU, the bounding box is modeled as a two-dimensional Gaussian distribution, and the similarity of the Gaussian distribution is measured in different domain spaces. The present application optimizes the detector in three aspects of data source, training strategy and evaluation metric method, improves the detection performance of small personnel targets, and has wide applicability.

[0140] The present application also discloses a target detection device in high-resolution aerial images, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned target detection method in high-resolution aerial images when executing the computer program.

[0141] It should be noted that the information interaction between the above devices, execution process and the like, since based on the same concept as the method embodiments of the present application, the specific functions and the brought technical effects can be specifically referred to the method embodiments part, and will not be repeated here.

[0142] The device can be a desktop computer, notebook, palm computer, radar and cloud server and the like computing devices. The device can include but not limited to a processor, a memory. Those skilled in the art can understand that more or less components can be included, or some components can be combined, or different components can be included, for example, input and output devices, network access devices and the like can also be included.

[0143] The processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0144] The memory can be an internal storage unit of the extraction device in some embodiments, such as a hard disk or a memory of the extraction device. The memory can also be an external storage device of the extraction device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card and the like equipped on the extraction device. Further, the memory can include both the internal storage unit and the external storage device of the extraction device. The memory is used to store an operating system, application programs, a boot loader, data and other programs, such as program codes of the computer program and the like. The memory can also be used to temporarily store data that has been output or will be output.

Claims

1. A method for detecting a target in a high-resolution aerial image, characterized in that, The method comprises the following steps: constructing a first training picture set, each initial training picture in the first training picture set containing a person class target; using a target detector to identify the target in the initial training picture; selecting a target frame in the initial training picture that is not identified by the target detector, and generating a target instance library according to the target frame; implanting a pseudo sample in the target instance library into the initial training picture to generate a second training picture set; the specific implementation mode of the pseudo sample implantation is: the implantation position: implanting in the area outside the existing target frame in the initial training picture; the implantation rule: the size of the pseudo sample is less than 32*32 pixels; the number of pseudo samples implanted in a single picture is controlled to be 3-5; training the target detector based on the second training picture set; using the trained target detector to detect the target in a high-resolution aerial image; the pseudo sample generation method: performing semantic segmentation on the content in the target frame that is not detected in the initial training picture to obtain each person class small target in the picture, and taking the person class small target as a pseudo sample.

2. The method of claim 1, wherein the step of detecting the target in the high resolution aerial image is characterized by, Generating a target instance library according to the target frame comprises: calculating a mask map of the target frame; selecting a pseudo sample in the target frame according to the mask map.

3. The method of claim 2, wherein the step of detecting the target in the high resolution aerial image is characterized by, Implanting a pseudo sample in the target instance library into the initial training picture comprises: implanting the pseudo sample into a target position of the initial training picture according to a quantity threshold and a pseudo sample size threshold; wherein the target position is a position outside the existing target frame in the initial training picture.

4. The method of claim 2 or 3, wherein the step of detecting the target in the high resolution aerial image is performed by using a neural network. Training the target detector based on the second training picture set comprises: respectively constructing a first loss function of a pseudo sample prediction frame and a second loss function of an original sample of a picture, performing weighted summation on the first loss function and the second loss function to obtain a total loss function.

5. The method of claim 4, wherein the step of detecting the target in the high resolution aerial image is characterized by, Training the target detector based on the second training picture set further comprises: using a density estimation network to segment a final training picture in the second training picture set to obtain a plurality of cropped sub-pictures; using the target detector to identify the target in the cropped sub-pictures and the final training picture; updating the target detector and the density estimation network according to the target identification result using the total loss function.

6. The method of claim 5, wherein the step of detecting the target in the high resolution aerial image is characterized by, Using the target detector to identify the target in the cropped sub-pictures and the final training picture comprises: using the target detector to identify the target in the cropped sub-pictures to obtain a first anchor frame; using the target detector to identify the target in the final training picture to obtain a second anchor frame corresponding to the position of the first anchor frame; selecting the first anchor frame or the second anchor frame as a detection anchor frame of the corresponding target based on a non-maximum suppression method.

7. The target detection method in a high-resolution aerial image according to claim 6, wherein when the target detector is used to identify the target in the cropped sub-pictures, the first anchor frame is determined based on a maximum mean difference distance method; when the target detector is used to identify the target in the final training picture, the second anchor frame is determined based on a maximum mean difference distance method.

8. A method of object detection in a high resolution aerial image as claimed in any of claims 5-7, characterized in that, The target detector contains a plurality of target detection models of different scales.

9. The method of claim 1, wherein the step of detecting the target in the high resolution aerial image is characterized by, The first training picture set is constructed by: obtaining a public picture dataset; performing label cleaning on the public picture dataset to remove label information other than personnel class targets.

10. An apparatus for detecting a target in a high resolution aerial image, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the target detection method in the high-resolution aerial image according to any one of claims 1-9 when executing the computer program.

Images