An efficient object detection method for aerial images based on dense area perception
Through a method based on dense area perception, the low-resolution and high-resolution features of aerial images are extracted, and fine detection is carried out in combination with global context information, which solves the problems of dense area information loss and resource waste, and achieves efficient small-objective detection.
Patent Information
- Application Number
- CN202210071009.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-21
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2042-01-21
AI Technical Summary
When processing aerial images, the global context information of dense areas is seriously lost, which affects detection performance. The area positioning module and the target detection module are independent, and they fail to effectively use the target density to guide detection.
A highly efficient aerial image object detection method based on dense area perception is proposed. By acquiring images in real time and inputting them into the trained object detection model, low-resolution and high-resolution features are extracted, and regional focus cropping is used using global rough detection results and density maps, fine detection is carried out in combination with global context information, and the final detection results are finally obtained through the fusion of detection results.
Effectively distinguish dense areas, use global context information to assist in fine detection, improve small-object detection performance, reduce waste of computing resources, and improve detection speed and model efficiency.
Smart Images
Figure CN114445705B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to an efficient target detection method for aerial images based on dense area perception. Background Art
[0002] Object detection technology for aerial images is used to find objects or areas of interest in aerial images, and plays an increasingly important role in many scenarios such as urban security, smart transportation, and industrial inspections. Compared with conventional images, there are more small objects in aerial images, and high-resolution images are required to accurately detect small objects. However, directly applying existing detection models to high-resolution images will bring huge computational complexity and extremely high memory requirements. One solution is to first cut the high-resolution image into several small images, and then detect them separately.
[0003] However, the existing technology is still not perfect in selecting and processing dense areas or targets. The loss of global context information in dense areas caused by cropping seriously affects the detection performance. In addition, in the existing methods, the region positioning module and the target detection module are independent of each other, and the density of targets in the positioning region has not been taken seriously as a guide for the subsequent detection module. Summary of the invention
[0004] In order to solve the problems of difficulty in detecting dense small targets in aerial images and waste of computing resources when processing aerial high-resolution images in the above prior art, the present invention proposes an efficient aerial image target detection method based on dense area perception, the method comprising: acquiring an image to be detected in real time, inputting the image to be detected into a trained target detection model, and obtaining a detection result; marking the detection result;
[0005] The process of training an object detection model includes:
[0006] S1: Obtain an aerial image dataset, preprocess the data in the dataset, and divide the preprocessed dataset into a training set and a test set;
[0007] S2: Input the aerial images in the training set into the input layer of the target detection model to extract the low-resolution and high-resolution features of the aerial images;
[0008] S3: Input the low-resolution image into the global roughness detector and the convolutional neural network for density estimation respectively to obtain the global roughness detection result and the initial feature map;
[0009] S4: extract the global context information of the initial features to obtain a global feature map;
[0010] S5: Integrate the initial feature map to obtain the number of targets; use the global rough result to guide the tuning of the initial feature map, and obtain a mask map reflecting the target density distribution after binarization;
[0011] S6: Map the mask image to the high-resolution image to obtain a high-resolution target focus area;
[0012] S7: cropping the high-resolution original global image according to the high-resolution target focus area guidance; inputting the high-resolution target area image, the number of targets and the global feature map obtained after the cropping operation into the regional fine detector to obtain the regional fine detection result;
[0013] S8: Fusing the global rough detection result map and the regional fine detection result map to obtain the final detection result;
[0014] S9: Calculate the loss function of the model and continuously adjust the parameters of the model. When the loss function is minimized, the model training is completed.
[0015] Preferably, the process of preprocessing the data includes: deleting redundant data, and performing image enhancement processing on the deleted data.
[0016] Preferably, the process of processing the low-resolution image using a global rough detector includes: separating the object of interest from the background, and determining the category of the object and its spatial position, wherein the global rough detector can be any general advanced object detection model.
[0017] Preferably, the process of inputting the low-resolution image into the convolutional neural network for processing includes: using a convolution module composed of residual blocks to perform preliminary extraction on the low-resolution image to obtain multi-scale features with image semantic information; using a construction layer to perform jump link fusion on the multi-scale features to obtain a density map representing the distribution of target positions in the aerial image.
[0018] Preferably, the process of extracting global context information of the density map includes: using a convolutional neural network to encode multi-scale features in the process of generating the density map to obtain a global feature map containing semantic information such as image scene.
[0019] Preferably, the formula for integrating the density map is:
[0020] N S =∫∫ S F(X i ,θ)dxdy
[0021] Among them, N S represents the number of predicted targets in region S, F represents the dense region perception function, X iRepresents the pixel estimation in the density map, θ represents the hyperparameter set of the dense area perception module, x represents the horizontal coordinate of the pixel in the image, and y represents the vertical coordinate.
[0022] Preferably, the process of using the global roughness result to guide the tuning of the density map includes: using a Gaussian blur kernel to convert the bounding box in the global roughness detection result into a predicted target distribution map; then superimposing the predicted target distribution map on the density map generated by the convolutional neural network for tuning, and binarizing the tuned density map to obtain a mask map for guiding regional focusing.
[0023] Preferably, the process of obtaining a regional fine detection result map includes: after obtaining a mask map for guiding regional focusing through a density estimation module, mapping the mask map to an original high-resolution global image, and cropping it to obtain a corresponding high-resolution focused area image, and inputting the high-resolution focused area image into a fine detector to obtain a regional fine detection result.
[0024] Preferably, the process of fusing the global rough detection result map and the regional fine detection result map includes: sorting all prediction boxes in the global rough detection result and the regional detection result by confidence, and gradually removing the prediction boxes with lower confidence according to the overlap ratio between the prediction boxes under the guidance of the target density distribution.
[0025] Preferably, the expression of the model loss function is:
[0026]
[0027] in, represents the loss value between the model prediction value and the true value, N represents the number of images, M represents the number of target categories, and y ic represents the sign function (when the sample i category is equal to category c, it takes 1, otherwise it takes 0), p ic represents the predicted probability that sample i belongs to category c, P id represents the set of points predicted as targets in the density map corresponding to sample i, G id Represents the set of true values of the target points in the density map corresponding to sample i.
[0028] Beneficial effects of the present invention:
[0029] The present invention effectively distinguishes dense areas (such as target gathering areas) in the image by utilizing a method based on dense area perception, and then utilizes the global context information extracted during the rough detection of the entire image to assist in the fine detection of the dense areas. On the one hand, because dense areas can obtain more computing resources, higher resolution inputs and detectors with more complex structures can be used during processing. At the same time, thanks to the assistance of global context information, the performance of small target detection in dense areas can be significantly improved. On the other hand, because only lightweight detectors are needed for non-dense areas to obtain better results, the use of detectors with simpler structures in the processing of non-dense areas can greatly reduce the resources required for computing, improve the detection speed of the target detection model and reduce the video memory occupancy required in the detection process, avoid unnecessary waste, and effectively utilize limited computing resources, thereby realizing the practical application of efficient target detection models on high-resolution aerial images. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 A framework diagram of a target detection method based on dense area perception of the present invention. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] An efficient target detection method for aerial images based on dense area perception includes: acquiring images to be detected in real time, inputting the images to be detected into a trained target detection model to obtain detection results; and marking the detection results.
[0033] A specific real-time way to train object detection models, such as Figure 1 As shown, the training process includes:
[0034] S1: Obtain an aerial image dataset, preprocess the data in the dataset, and divide the preprocessed dataset into a training set and a test set;
[0035] S2: Input the aerial images in the training set into the input layer of the target detection model to extract the low-resolution and high-resolution features of the aerial images;
[0036] S3: Input the low-resolution image into the global roughness detector and the convolutional neural network respectively to obtain a global roughness detection result map and a density map of target distribution;
[0037] S4: extract the global context information of the density map to obtain the global feature map;
[0038] S5: Integrate the density map to obtain the number of targets; use the global rough result to guide the tuning of the density map, and obtain a mask map indicating the target distribution status after binarization;
[0039] S6: Map the mask image to the high-resolution image to obtain a high-resolution target focus area;
[0040] S7: cropping the high-resolution original global image according to the high-resolution target focus area guidance; inputting the high-resolution target area image, the number of targets and the global feature map obtained after the cropping operation into the regional fine detector to obtain the regional fine detection result;
[0041] S8: Fusing the global rough detection result map and the regional fine detection result map to obtain the final detection result;
[0042] S9: Calculate the loss function of the model and continuously adjust the parameters of the model. When the loss function is minimized, the model training is completed.
[0043] The process of preprocessing data includes: deleting redundant data and performing image enhancement processing on the deleted data.
[0044] The object detection model includes a dense area perception module, a global context learning module, an object detection module and detection result fusion.
[0045] Specifically, the dense area perception module includes: Aiming at the problem that dense areas in aerial images are difficult to detect, this method improves the classic network in the field of crowd counting and constructs a dense area perception network suitable for aerial high-resolution images. First, by estimating the scale of each type of target, a category-based Gaussian kernel is generated to narrow the gap between targets of different categories, and then the Gaussian kernel is used to convolve on the low-resolution image (LR) to generate a density map reflecting the target distribution. At the same time, based on the generated density map, the detection results on the low-resolution image (LR) can be combined to better locate the dense area in the image for subsequent refined processing and detection. In addition, by integrating the predicted density map, the number of targets in the corresponding area will be obtained to guide the detector to obtain better detection performance. Finally, these dense areas are mapped to the high-resolution image (HR) for cropping to obtain a series of independent dense area images, which will be sent to the detector in the next stage.
[0046] Global context learning module. The cropping operation of the dense region perception part causes the independence of each region. In order to avoid the loss of global context information in the image, we designed a global context learning method. The global features of the low-resolution image are extracted through a convolutional neural network, and then input into the detector for learning together with the high-resolution dense region image cropped in the previous stage, so as to achieve better target detection effect.
[0047] Object detection module. This part can be improved based on any advanced lightweight object detection network. Its main function is to detect low-resolution global images and cropped high-resolution dense area images respectively, and then output the corresponding detection results. In order to better solve the problems of large object scale changes and large number of small objects in aerial images, the loss function introduces the number of objects predicted by the dense area perception part as a guide, and optimizes the detection results by setting different weights for different object categories.
[0048] Detection result fusion. The global detection results output by the target detection part are matched and mapped with the detection results of the dense area and then fused to obtain the final detection results on the high-resolution image. In this process, we especially consider the best detection results that can be obtained for targets of different sizes in images of different resolutions, and adaptively select the best detection results in different images through learnable parameters, thereby improving the target detection performance for aerial high-resolution images.
[0049] The process of processing a low-resolution image using a global rough detector includes separating an object of interest from the background and determining the class and spatial location of the object, wherein the global rough detector can be any general advanced object detection model.
[0050] The process of inputting a low-resolution image into a convolutional neural network for processing includes: using a convolution module composed of residual blocks to perform preliminary extraction on the low-resolution image to obtain multi-scale features with image semantic information; then using the multi-scale features to construct hierarchical jump links to fuse feature information and obtain its corresponding density map.
[0051] The process of extracting global context information of the density map includes: using a convolutional neural network to encode multi-scale features in the process of generating the density map, and obtaining a global feature map containing semantic information such as image scenes.
[0052] The formula for integrating the density map is:
[0053] N S =∫∫ S F(X i ,θ)dxdy
[0054] Among them, NS represents the number of predicted targets in region S, F represents the dense region perception function, X i and θ represent the hyper-parameter sets of the pixel estimation and dense area perception modules in the density map, respectively. x represents the horizontal coordinate of the pixel in the image, and y represents the vertical coordinate.
[0055] The process of using the global roughness result to guide the tuning of the density map includes: using a Gaussian blur kernel to convert the bounding box in the global roughness detection result into a predicted target distribution map; then superimposing the predicted target distribution map on the density map generated by the convolutional neural network for correction, and binarizing the corrected density map to obtain a mask map for guiding regional focusing.
[0056] The process of obtaining the regional fine detection result includes: after obtaining the mask map guiding the regional focus through the density estimation module, mapping it to the original high-resolution global image, cropping it to obtain the corresponding high-resolution focus area image, and then sending it to the fine detector to obtain the regional fine detection result.
[0057] The process of fusing the first detection result map and the second detection result map includes: sorting all prediction boxes in the global rough detection result and the regional detection result according to confidence, and gradually removing prediction boxes with lower confidence according to the overlap ratio between the prediction boxes under the guidance of the target density distribution.
[0058] The expression of the model loss function is:
[0059]
[0060] in, represents the loss value between the model prediction value and the true value, N represents the number of images, M represents the number of target categories, and y ic represents the sign function (when the sample i category is equal to category c, it takes 1, otherwise it takes 0), p ic represents the predicted probability that sample i belongs to category c, P id represents the set of points predicted as targets in the density map corresponding to sample i, G id Represents the set of true values of the target points in the density map corresponding to sample i.
[0061] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An efficient aerial image target detection method based on dense area perception, characterized in that: include: Acquire the image to be detected in real time, input the image to be detected into the trained target detection model, and obtain the detection result; Marking the test results; The process of training an object detection model includes: S1: Obtain an aerial image dataset, preprocess the data in the dataset, and divide the preprocessed dataset into a training set and a test set; S2: Input the aerial images in the training set into the input layer of the target detection model to extract low-resolution images and high-resolution features of the aerial images; S3: inputting the low-resolution image into the global roughness detector and the convolutional neural network respectively, obtaining a global roughness detection result map and a density map of target distribution; the process of processing the low-resolution image using the global roughness detector includes: determining the target in the low-resolution image, marking the determined target; separating the marked target from the background; determining the separated target category and its spatial position; S4: extract the global context information of the density map to obtain the global feature map; S5: Integrate the density map to obtain the number of targets; and use the global rough detection result map to optimize the density map, and perform binarization on the optimized density map to obtain a low-resolution target area mask map; S6: Map the low-resolution target area mask image to the high-resolution feature map to obtain a high-resolution target area image; S7: cropping the high-resolution target area image to obtain an independent focus area image; inputting the focus area image, the number of targets and the global feature map into the second detector to obtain a regional fine detection result map; S8: Fusing the global rough detection result map and the regional fine detection result map to obtain the final detection result; S9: Calculate the loss function of the model and continuously adjust the parameters of the model. When the loss function is minimized, the model training is completed.
2. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The process of preprocessing data includes: deleting redundant data and performing image enhancement processing on the deleted data.
3. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The process of inputting low-resolution images into a convolutional neural network for processing includes: using a convolution module composed of residual blocks to perform preliminary extraction on the low-resolution image to obtain multi-scale features with image semantic information; using a construction layer to perform jump link fusion on the multi-scale features to obtain a density map representing the distribution of target positions in the aerial image.
4. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The process of extracting global context information of the density map includes: using a convolutional neural network to encode multi-scale features in the process of generating the density map, and obtaining a global feature map containing semantic information of the image scene.
5. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The formula for integrating the density map is: N S =∫∫ S F(X i ,θ)dxdy Among them, N S represents the number of predicted targets in region S, F represents the dense region perception function, X i Represents the pixel estimation in the density map, θ represents the hyperparameter set of the dense area perception module, x represents the horizontal coordinate of the pixel in the image, and y represents the vertical coordinate.
6. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The process of using the global roughness result to guide the tuning of the density map includes: using a Gaussian blur kernel to convert the bounding box in the global roughness detection result into a predicted target distribution map; then superimposing the predicted target distribution map on the density map generated by the convolutional neural network for tuning, and binarizing the tuned density map to obtain a mask map for guiding regional focusing.
7. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The process of obtaining a regional fine detection result map includes: after obtaining a mask map guiding regional focusing through a density estimation module, mapping the mask map to an original high-resolution global image, cropping it to obtain a corresponding high-resolution focused area image, and inputting the high-resolution focused area image into a fine detector to obtain a regional fine detection result.
8. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The process of fusing the global coarse detection result map and the regional fine detection result map includes: sorting all the prediction boxes in the global coarse detection result and the regional detection result by confidence, and gradually removing the prediction boxes with lower confidence according to the overlap ratio between the prediction boxes under the guidance of the target density distribution.
9. The method for efficient aerial image target detection based on dense area perception according to claim 1, characterized in that: The expression of the model loss function is: in, represents the loss value between the model prediction value and the true value, N represents the number of images, M represents the number of target categories, and y ic represents the sign function (when the sample i category is equal to category c, it takes 1, otherwise it takes 0), p ic represents the predicted probability that sample i belongs to category c, P id represents the set of points predicted as targets in the density map corresponding to sample i, G id Represents the set of true values of the target points in the density map corresponding to sample i.