A method for detecting dense and similar objects in aerial images

Abstract

The present invention discloses a method for detecting dense and similar targets in aerial images, which belongs to the field of target detection technology. First, the easily confused classes are divided into one category, and a coarse detector is used to detect the targets. Then, a clustering method is used to group the results obtained in the previous step to create a foreground area image, which is then handed over to the coarse detector. Finally, the confused classes are classified, and the detection results and the detection results of the fine detector are subjected to non-maximum suppression to obtain the final detection results. The present invention can effectively detect dense small targets, identify easily confused classes, and further improve the detection results of dense targets and similar targets in aerial images.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method for detecting dense and similar targets in aerial images. Background Art

[0002] In recent years, with the advancement of deep learning, object detection has been applied in many fields, such as autonomous driving, lesion detection, disaster assessment, and disaster search and rescue. Currently, deep learning-based object detection algorithms are mainly divided into one-stage and two-stage algorithms, or anchor-based and anchor-free object detection algorithms. Two-stage object detection algorithms involve two steps: candidate region extraction and object detection. First, a candidate region extraction algorithm is used to extract candidate regions containing objects. Then, an object detection algorithm is used to detect and classify the candidate regions to obtain detection results. Common two-stage detection methods include R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN. One-stage object detection algorithms eliminate the candidate region extraction stage and perform detection directly on feature maps extracted by the backbone network. Key one-stage object detection algorithms include the Yolo series, SSD, and RetinaNet. Two-stage algorithms generally achieve higher detection accuracy than one-stage algorithms, but one-stage algorithms are generally faster and more efficient. Anchor-based methods use anchor boxes, which are paired with ground truth objects to facilitate rapid network convergence during training and more accurate object detection. Anchor-free methods primarily detect objects based on key points and centers. Generally speaking, anchor-based methods achieve higher detection accuracy than anchor-free methods.

[0003] Aerial images feature a large field of view and high resolution. However, targets generally occupy a relatively small proportion of the image, and their distribution is uneven, with some being densely packed and others sparsely packed. While object detection algorithms have been successfully applied to general-purpose object detection, and detection performance continues to improve, directly applying these general-purpose object detection algorithms to the task of detecting objects in aerial images has been less than satisfactory and remains a significant challenge. This is primarily due to two factors: first, targets occupy a relatively small and concentrated proportion of the aerial image's field of view; some areas are heavily clustered with targets, while others are almost entirely background. Second, different types of targets are very similar. Due to the significant variations in altitude and viewing angle during drone photography, as well as complex weather conditions, these different types of targets can appear similar, making it difficult for object detectors to confuse similar objects and effectively identify them.

[0004] In order to solve the above-mentioned problems, the present invention discloses a method for detecting dense and similar targets in aerial images, which can improve the detection accuracy and robustness of dense small targets and different types of similar targets. Summary of the Invention

[0005] The main purpose of this invention is to disclose a method for detecting dense and similar objects in aerial images, addressing the technical problem of effectively identifying dense small objects and similar objects of different classes in images. This method can effectively detect dense small objects and identify easily confused classes, further improving the accuracy of detection results.

[0006] In order to achieve the above objectives and solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] This paper designs a novel detection network framework, namely the aerial image detector, which is used to detect dense and similar objects in aerial images. The aerial image detector consists of five stages: a coarse detector detects the aerial image, the detection results are clustered and adaptively form a foreground image, the coarse detector detects the foreground image and classifies it using a class similarity classifier, a fine detector detects the aerial image, and non-maximum suppression fuses all detection results. The specific network structure is shown in the figure below. Figure 1 shown.

[0008] First, the aerial image is input into the coarse detector for detection

[0009] Currently, Transformer Prediction Heads (TPHs) are effective for detecting small, densely packed objects in aerial imagery. Therefore, the coarse detector of this invention uses TPHs to detect small objects in aerial images, achieving better detection results and thus facilitating subsequent clustering and foreground region generation. TPHs are based on YOLOv5 and are improved in three key aspects. First, TPHs retain the original version's use of the CSPDarknet53 and PANet networks as the backbone and neck, respectively. In the head, TPHs add a detection head for small object detection. Therefore, TPHs contain a total of four detection heads, one for detecting micro, small, medium, and large objects, respectively. Second, the original prediction head is replaced by a Transformer. Finally, to find regions of interest in images with large coverage, TPHs introduce a Convolutional Block Attention Module (CBAM) to sequentially generate attention maps along the channel and spatial directions.

[0010] To improve the overall TPH detection accuracy and the accuracy of easily confused classes, we group these easily confused classes together during TPH training. Taking the Visdrone2019 dataset as an example, there are ten classes. Because tricycle and awning-tricycle have similar appearances and are easily confused during detection, they are grouped together during training, leaving the other classes unchanged. This not only improves the detection accuracy of the confused classes, but also the overall class detection accuracy, facilitating subsequent clustering and foreground region generation.

[0011] Second, the detection results are clustered using an adaptive clustering algorithm to adaptively form a foreground area image.

[0012] Because aerial images have a relatively large field of view and relatively small and densely packed targets, most algorithms for preprocessing aerial images use a uniform image partitioning method, dividing the entire image into several patches. This partitioning method has two disadvantages. First, it is impossible to distinguish key areas. Each patch contains a large amount of background information and must be fed into the detector for detection, which increases the target detection computing power and seriously reduces detection efficiency. Second, through uniform partitioning, many targets are segmented, resulting in the detector being unable to fully detect the target. Therefore, the present invention uses the detection results obtained by the coarse detector to form new chips through clustering, and then adaptively combines different chips into a foreground area image. Figure 2 The process of generating the foreground area image is shown.

[0013] The foreground area image generation process is mainly divided into three steps:

[0014] (1) Using the output of the coarse detector, the foreground sub-region is clustered into several clustered regions;

[0015] (2) Adaptively enlarge small-sized aggregation areas;

[0016] (3) Assemble these clustered areas into a unified mosaic image.

[0017] Furthermore, the specific implementation process of step (1) of the foreground area image generation process is as follows:

[0018] First, the width and height of the detected bounding box are increased by β times from the center point, which can roughly contain the GT box.

[0019] The greedy foreground region generation algorithm (FRG) is used to fuse these enlarged regions. Specifically, FRG will enlarge the rough detection result B cAs input, select the smallest size bounding box A as the starting point. For any remaining detection box B, FRG searches for the smallest convex envelope C that can contain A and B. If the sum of the areas of A and B is greater than C, then use C to update A and extract the area from B. c Remove B from the box. Repeat this process until there is no bounding box B that satisfies the condition |A|+|B|≥C. At this point, take A as B r Repeat the above steps until B c is an empty set, and the final fused region set B is obtained r .

[0020] Furthermore, the specific implementation process of step (2) of the foreground area image generation process is as follows:

[0021] Through FRG, each image can be represented as several clustered regions of different sizes. In order to balance their sizes, especially for small objects, we first estimate B r Then the average size is changed to a fixed size, and the areas smaller than a certain threshold are enlarged (the present invention adopts a size of 96×96).

[0022] Furthermore, the specific implementation process of step (2) of the foreground area image generation process is as follows:

[0023] Clustered region assembly: Utilizing the PHSPPOG (Priority Heuristic Algorithm about StripPacking Problem with a Fixed Orientation and Guillotine Constraints) method, a priority heuristic approach for the strip packing problem with fixed orientation and one-size-fits-all constraints, this method places each chip within an image without overlap and minimizes the required image height. The aforementioned regions are stitched together into a mosaic image to form a foreground region image. The foreground region image contains most of the objects, while the background region accounts for a smaller proportion. Small objects are also enlarged due to adaptive scaling, making them more easily detected by subsequent detectors.

[0024] Third, the coarse detector detects the foreground image and uses the class similarity classifier to separate the easily confused classes.

[0025] After obtaining the generated foreground image, we continue detecting the foreground image, again using the TPH detector. The coarse TPH detector here uses a different input than the initial detector, which uses the original aerial image as input. This coarse detector uses the foreground image generated through clustering, so retraining the coarse TPH detector is necessary. The foreground image training set is generated using the foreground region generation algorithm. Because some classes in the dataset are similar and difficult to detect and identify, classes with similar appearance are grouped together.

[0026] For easily confused classes, a class similarity classifier is used for classification. Taking the Visdrone2019 dataset as an example, the easily confused classes in this dataset are tricycle and awning-tricycle. These are grouped into one category, tentatively referred to as "tricycle." Through visual observation of the coarse detector, it was found that within the detected "tricycle" class, there are a large number of non-"tricycle" classes. Inspired by this observation, the present invention first designed a "tricycle" and "non-tricycle" classifier, using the Resnet18 backbone network as a two-class classifier. In the "tricycle" detection results, the detection results belonging to tricycles are retained, and the detection results that do not belong to tricycles are discarded. Subsequently, all the remaining tricycle classes are classified. Here, a Resnet18 classifier is also designed, which is used to classify the tricycle class and the awning-tricycle class. Through this design, classes with similar appearances can be distinguished.

[0027] Figure 3 This is the flow chart of the similarity classifier. Its specific working process is as follows:

[0028] First, all objects belonging to "tricycle" are fed into the "tricycle" and "non-tricycle" classifier, which consists of Resnet18 and a fully connected layer.

[0029] Then all objects belonging to "tricycle" are fed into the "tricycle" and "awning-tricycle" classifiers, which are also composed of Resnet18 and fully connected layers;

[0030] Finally, the easily confused classes "tricycle" and "awning-tricycle" were classified.

[0031] Fourth, the aerial image is input into the fine detector for detection, and the detection results of all categories can be obtained.

[0032] Fifth, use non-maximum suppression to combine the detection results obtained in the third and fourth steps to obtain the final detection result

[0033] Because clustering algorithms cluster densely packed objects and form a foreground image, sparsely packed objects are not included in the foreground image. Using only the coarse detector to detect the foreground image will miss many sparse objects. Therefore, to avoid missing objects, the fine detection results are fused with those of the coarse detector. Both the fine and coarse detectors use the TPH detector, but their functions differ slightly. The coarse detector groups easily confused classes together for detection, while the fine detector detects all classes. The fine and coarse detectors' detection results are then fused using the class similarity classifier and subjected to non-maximum suppression to ultimately obtain the detection results for aerial image targets.

[0034] The effective benefits of this invention compared with the prior art are:

[0035] (1) The aerial photography detector designed in the present invention detects dense small targets and adaptively clusters the detection results to form a foreground area image, thereby reducing the background area and improving the detection efficiency.

[0036] (2) The present invention uses a strategy of merging similar classes to reduce the number of detection types and improve the detection accuracy, and uses a similar classifier to classify the merged similar classes, making the detection results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The network structure framework diagram of the aerial image detector of the present invention;

[0038] Figure 2 Schematic diagram of the foreground area image process of the present invention;

[0039] Figure 3 Schematic diagram of the similarity classifier flow of the present invention. DETAILED DESCRIPTION

[0040] Since the objects in drone images are small in size, dense, and have small inter-class differences, current detection methods are challenging. To this end, this paper proposes a method for detecting dense and similar objects in aerial images.

[0041] The design concept of the present invention is as follows: First, easily confused classes are grouped into one category, and a coarse detector is used to detect the target. Then, clustering is used to group the results from the previous step to create a foreground image, which is then passed to the coarse detector. Finally, the confused classes are classified, and the detection results are compared with those of the fine detector through non-maximum suppression to obtain the final detection result. Through this invention, the proposed algorithm can effectively detect densely packed small targets, identify easily confused classes, and further improve detection results.

[0042] The present invention will be explained and described in detail below with reference to the accompanying drawings. Figure 1 As shown in the figure, the flow chart of using adaptive clustering algorithm to generate foreground area image is as follows Figure 2 As shown, the similarity classifier flow chart is as follows Figure 3 As shown, the specific steps include:

[0043] Step 1: Coarse detector detects aerial images

[0044] Use a coarse detector to perform rough detection on the aerial image, and use an adaptive clustering algorithm to cluster the obtained detection results to adaptively form a foreground area image;

[0045] Step 2: The coarse detector detects the foreground area image

[0046] Use a coarse detector to detect foreground images. For example, the visdrone2019 dataset has 10 categories. Tricycles and awning-tricycles are very similar in appearance. Therefore, during the training of the coarse detector, tricycles and awning-tricycles are grouped into one category. This allows the coarse detector to detect nine categories of foreground images.

[0047] Step 3: Similarity classifier detection

[0048] Feed the detection results obtained in the previous step into the class similarity classifier for classification. This can help separate easily confused classes. Taking the visdrone2019 dataset as an example, feed the detection results of the tricycle and awning-tricycle classes into the trained similarity classifier, leaving the other classes unchanged. This will result in ten detection results.

[0049] Step 4: Fine detector detects aerial images

[0050] Inputting the aerial image into the fine detector for detection, we can get ten types of detection results;

[0051] Step 5: Fusion detection results

[0052] The detection results obtained in the third and fourth steps are combined using non-maximum suppression to obtain the final detection result.

[0053] In summary, the present invention designs a network architecture that can accurately detect dense small targets and similar class targets.

[0054] Example 1

[0055] Dataset Preparation: This paper primarily conducts validation experiments on the Visdrone2019 dataset. The Visdrone2019 dataset, created using drone aerial photography, contains 263 videos, 179,264 frames, and 10,209 still images. The large variation in object size in aerial images facilitates validation of the proposed algorithm. The object detection task uses 10,209 still images, including 6,471 images as a training set, 548 images as a validation set, and 1,610 images as a test set.

[0056] Implementation process: The network structure of the entire algorithm of the present invention is completed on the basis of the Pytorch framework, and the NVidia3090 image card is used for training and testing. Since the coarse detector and the fine detector used in the algorithm are both based on the TPH algorithm, the algorithm model is initialized using the Yolov5-x pre-training model before training. Since the original structure of Yolov5 has been changed, it is only necessary to select the part to be used to initialize the proposed model. The Adam optimizer is used to train the network structure for more than 300 epochs, the initial learning rate is set to 0.001, and the maximum side length of the input image is set to 1536. Due to the limitations of the graphics card, the batch size is set to 2. For two small two-category classifiers, the ResNet18 network structure is used as the backbone network, and the pre-training model trained with ImageNet is used for initialization. The learning rate is set to 0.01, and the SGD optimizer is used to train the network. The image input to the network is uniformly adjusted to 224*224 and normalized to a mean of 0.5 and a variance of 0.5.

[0057] The above is a further detailed description of the present invention in conjunction with specific embodiments, and the specific implementation of the present invention cannot be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for detecting dense and similar objects in aerial images, characterized in that: Design an aerial image detector to detect dense and similar objects in aerial images; The aerial image detector includes the following five stages: First, the coarse detector detects the aerial image; The coarse detector uses the TPH detector to detect small targets in aerial images, which facilitates subsequent clustering and generation of foreground areas; Second, the detection results are clustered and adaptively formed into a foreground image; that is, the detection results are clustered using an adaptive clustering algorithm to adaptively form a foreground area image; Using the detection results obtained by the coarse detector, new chips are formed by clustering, and then different chips are adaptively combined into a foreground area image; The foreground area image generation process is mainly divided into three steps: (1) Using the detection results output by the coarse detector, the foreground sub-region is clustered into several clustered regions; (2) Adaptively enlarging small-sized aggregation areas; (3) assembling these clustered areas into a unified mosaic image; The foreground sub-region clustering mainly includes: Third, the coarse detector detects the foreground image and uses the class similarity classifier to separate the easily confused classes. After obtaining the image generated by the foreground area, continue to detect the foreground image. Here, the TPH detector is still used. The coarse detector TPH here is different from the initial detector input. The initial coarse detector input is the original aerial image, while the coarse detector input here is the foreground area image generated by clustering. Therefore, the coarse detector TPH needs to be trained again. The foreground image training set here is formed by the foreground area generation algorithm. Since some classes in the data set are similar and difficult to detect and identify, classes with similar appearance are grouped into one class. For easily confused classes, we use a class-similar classifier for classification as follows: we design a "tricycle" and "non-tricycle" classifier, and use the Resnet18 backbone network as a second-class classifier; among the "tricycle" detection results, we keep the detection results that belong to tricycles and discard the detection results that do not belong to tricycles; then, we classify all the remaining tricycle classes and also design a Resnet18 classifier to classify the tricycle class and the awning-tricycle class; through this design, we can distinguish between classes with similar appearances; Fourth, fine detector detects aerial images Input the aerial image into the fine detector for detection and obtain detection results for all categories; Fifth, non-maximum suppression fuses all detection results; The detection results obtained in the third and fourth steps are combined using non-maximum suppression to obtain the final detection result.

2. The method for detecting dense and similar objects in aerial images according to claim 1, characterized in that: The specific implementation process of step (1) of the foreground area image generation process is as follows: First, increase the width and height of the detected bounding box from the center point times, roughly containing the GT box; use the greedy foreground region generation algorithm to fuse these enlarged regions: The greedy foreground region generation algorithm will increase the rough detection results As input, select the minimum size bounding box A as the starting point; For any remaining detection box B, the greedy foreground region generation algorithm searches for the smallest convex envelope C containing A and B. If the sum of the areas of A and B is greater than C, then A is updated with C and the foreground region is generated from Remove B; Repeat this process until no bounding box B satisfies the condition |A|+|B|≥ C; at this point, take A as A gathering area in Repeat the above steps until is an empty set, and the final fused region set is obtained .

3. The method for detecting dense and similar objects in aerial images according to claim 1, characterized in that: The specific implementation process of step (1) of the foreground area image generation process is as follows: Through the greedy foreground area generation algorithm, each aerial image is represented as several clustered areas of different sizes. In order to balance the sizes of these clustered areas, for small targets, first estimate The average size in the image is then converted to a fixed size to increase the aggregation area that is smaller than the set threshold.

4. The method for detecting dense and similar objects in aerial images according to claim 1, characterized in that: The specific implementation process of step (1) of the foreground area image generation process is as follows: using the PHSPPOG method, each chip is placed in an image without overlapping, and the required height of the image is minimized, and the clustered areas are spliced ​​into a mosaic image to form a foreground area image.

5. The method for detecting dense and similar objects in aerial images according to claim 4, characterized in that: The classification process of the "three-wheeled vehicle" and "non-three-wheeled vehicle" classifier is as follows: First, all objects belonging to "tricycle" are fed into the "tricycle" and "non-tricycle" classifier, which is composed of Resnet18 and fully connected layers. Then, all objects belonging to "tricycle" are fed into the "tricycle" and "awning-tricycle" classifiers, which are also composed of Resnet18 and fully connected layers. Finally, the easily confused classes "tricycle" and "awning-tricycle" were classified.

6. The method for detecting dense and similar objects in an aerial image according to any one of claims 1 to 5, characterized in that: The coarse detector is a TPH detector, which has made the following three improvements based on YOLOv5: First, TPH continues to use the CSPDarknet53 and PANet networks as the backbone and neck, respectively, in the initial version. In the head part, TPH adds a detection head for small object detection. TPH contains a total of four detection heads, one for detecting micro, small, medium, and large objects. Secondly, replace the original prediction head with Transformer; Finally, in order to find the region of interest in an image with a large coverage area, TPH introduces a convolutional attention module to generate attention maps in the channel direction and spatial direction in sequence; At the same time, in order to improve the overall TPH detection accuracy and the detection accuracy of easily confused classes, the easily confused classes are classified into one category when training TPH.

Images