Aerial target counting and detection method based on lightweight density estimation network

Abstract

The application discloses a kind of based on lightweight density estimation network's aerial target counting and detection method, belong to target detection field, comprising: the target density estimation network is obtained to the target density map by inputting aerial image data, the element value in target density map is summed, obtain the target quantity in aerial image data;Local area cropped image block is obtained by cutting high-density area;Respectively on global aerial image data and local area cropped image block are carried out target detection, the detection frame of local area cropped image block is mapped to global aerial image data according to positional relationship, and the detection frame of global aerial image data and the detection frame of mapped local area cropped image block are fused by soft non-maximum suppression algorithm Processing, obtain the target detection result of aerial image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a method for aerial target counting and detection based on a light-weight density estimation network. BACKGROUND

[0002] Target detection is a basic and difficult task in the field of computer vision. Existing detection methods can be divided into four categories: template matching-based methods, knowledge-based methods, object image analysis-based methods and machine learning-based methods. Deep learning algorithm is a new type of computing model, which has stronger generalization ability and feature expression ability compared with traditional object detection and positioning methods. However, the large image, complex image background and uneven size and quantity distribution of training samples make the detection task more challenging. In view of the complexity of real scene, such as small target area, large scale change, target occlusion, unbalanced positive and negative samples, real-time detection and a series of problems, researchers have proposed many detection models based on deep learning. Although these models have achieved great success in natural scene image detection, their performance is not ideal when they are directly used for aerial image detection. The target detection in high-resolution aerial images is a challenging task, and the main reasons are as follows:

[0003] 1. Aerial images usually have different resolutions and contain a large number of instance targets, and the overall distribution is sparse. The collected objects are smaller than general objects, and the complex scene often makes the feature information of the target be submerged, which makes the aerial image very complex;

[0004] 2. The collected data is affected by light, geographical location, terrain, climate and other natural factors, and the pixel distribution is complex, which leads to a significant increase in the difficulty of aerial image target detection;

[0005] 3. Due to the influence of shooting angle, there will be occlusion or blockage of objects in dense object scenes, and the true value of these objects is not complete, which is not conducive to the training of the model;

[0006] 4. It cannot be guaranteed that each type of object appears balanced in the data set in the aerial image, so the data set is unevenly distributed, which brings great challenges to the detection algorithm.

[0007] For high-resolution aerial images, the common method is to crop the picture into uniform small blocks and perform target detection on these small blocks, which has a high recall rate and a low omission rate. However, uniform cropping will cause uniform crops to have more background pixels, and cannot adapt to the bounding box resolution of different types of crops. In addition, this method cannot utilize semantic information for cropping, resulting in most of the blocked images having only background or large objects that may be cut into two or more different image blocks, causing false positives or false negatives. SUMMARY

[0008] The present application provides a kind of aerial target counting and detection method based on light density estimation network, by spatial position quantity distribution feature acquisition high-density cropped image block is enlarged detection, significantly improve the detection precision of aerial target.

[0009] The first aspect of the present application provides an aerial target counting and detection method based on a light density estimation network, comprising the following steps:

[0010] Obtain the aerial image data to be detected, input the aerial image data to be detected into the trained target density estimation network, extract features from the aerial image data to be detected, generate a target density map based on the feature information of the aerial image data to be detected, and obtain the target spatial position distribution feature, sum the element values in the target density map to obtain the number of targets in the aerial image data to be detected;

[0011] According to the target spatial position distribution feature of the target density map, the region where the target distribution density in the aerial image data to be detected is higher than the preset value is calculated, and the local region cropped image block is cropped;

[0012] Respectively, the aerial image data to be detected and the local region cropped image block are detected, the detection frame of the local region cropped image block is mapped to the aerial image data to be detected according to the positional relationship, and the detection frame of the aerial image data to be detected and the mapped local region cropped image block is fused by a soft non-maximum suppression algorithm, to obtain the target detection result of the aerial image data to be detected.

[0013] Optionally, in an embodiment of the present application, before obtaining the aerial image data to be detected, the target density estimation network is constructed, comprising:

[0014] Based on the GhostNetV2 network, the network is modified for density generation task, the convolution layer and the full connection layer used for classification at the end of the network are removed, the initial processing layer and the first 16 bottleneck layers of the network are retained, and the stride of the 6th bottleneck layer is modified to 1, as the front-end feature extraction network of the target density estimation network.

[0015] The 4-fold, 8-fold and 16-fold down-sampling P2 layer feature map, P3 layer feature map and P4 layer feature map output by the feature extraction network are respectively expanded in receptive field by using the receptive field module;

[0016] The P4 layer feature map processed by the receptive field module is extracted by the feature extraction module, and after up-sampling, the P3 layer feature map processed by the receptive field module is fused in channel cascade to generate a new P3 layer feature map, and the feature extraction module is continued to extract the feature, and then the P2 layer feature map processed by the receptive field module is fused in channel cascade to generate a new P2 layer feature map;

[0017] The P2 layer feature map and the P3 layer feature map are fused by the receptive field expansion module composed of multiple different hollow bottleneck layers to generate the final output P2 layer feature map and P3 layer feature map;

[0018] The output P2 layer feature map and P3 layer feature map are respectively reduced in dimension by using 1x1 convolution to generate P2 layer level prediction density map and P3 layer level prediction density map.

[0019] Optionally, in an embodiment of the present application, training the target density estimation network comprises:

[0020] Obtaining training aerial image data, and labeling the training aerial data;

[0021] Processing the target frame labeled by the training aerial image data, obtaining the center point of the target frame, and generating the corresponding density map of the training aerial image data by using the labeled information;

[0022] Inputting the training aerial image data and the corresponding density map into the target density estimation network to complete forward propagation, calculating the loss value of the target density estimation network according to the forward propagation result and the real density map of the target density estimation network, updating the target density estimation network according to the loss value, and completing the update of the target density estimation network through multiple rounds of iteration to obtain the trained target density estimation network.

[0023] Optionally, in an embodiment of the present application, processing the target frame labeled by the training aerial image data, obtaining the center point of the target frame, and generating the corresponding density map of the training aerial image data by using the labeled information comprises:

[0024] Converting the image including the center point position label of the target in the training aerial image data into a target position information matrix;

[0025] Generating a target density map by using a geometric self-adaptive Gaussian kernel on the target position information matrix.

[0026] Optionally, in an embodiment of the present application, the loss value of the target density estimation network is calculated according to the forward propagation result and the real density map of the target density estimation network, so as to update the target density estimation network according to the loss value, comprising:

[0027] calculating a first MSE loss of the forward propagated P2 layer feature map result and the 4 times down-sampling real density map, and a second MSE loss of the forward propagated P3 layer feature map result and the 8 times down-sampling real density map;

[0028] summing the first MSE loss and the second MSE loss by combining the loss balance coefficient to obtain the loss value of the target density estimation network;

[0029] updating the parameters of the target density estimation network by reverse propagation gradient descent using the AdamW optimizer and the cosine annealing learning rate adjustment strategy on the loss value.

[0030] Optionally, in an embodiment of the present application, according to the target density map, the target distribution density higher than the preset value in the aerial image data to be detected is calculated using the target spatial position distribution feature, and a local area cropped image block is obtained by cropping, comprising:

[0031] using a multi-scale sliding window to slide on the target density map without overlapping, calculating the total density value of all positions in the sliding window, comparing the total density value in the sliding window with a pre-set size sliding window threshold, if greater than or equal to the size sliding window threshold, setting the density value of the current sliding window coverage area to 1, if less than the size sliding window threshold, setting the density value of the current sliding window coverage area to 0, generating a density mask map of the local area cropped image block;

[0032] using a depth-first search algorithm to divide the density mask map into connected blocks, obtaining a plurality of connected blocks with internal density values of 1, for each connected block, calculating its minimum circumscribed rectangle region, and according to each minimum circumscribed rectangle region, cropping the corresponding region position on the aerial image data to be detected to generate the local area cropped image block.

[0033] The second aspect embodiment of the present application provides a kind of aerial target counting and detection device based on lightweight density estimation network, comprising:

[0034] The generating module is configured to obtain aerial image data to be detected, input the aerial image data to be detected into the trained target density estimation network to perform feature extraction on the aerial image data to be detected, generate a target density map according to feature information of the aerial image data to be detected to obtain a target spatial position distribution feature, and sum element values in the target density map to obtain a target quantity in the aerial image data to be detected.

[0035] The cropping module is configured to calculate, according to the target spatial position distribution feature of the target density map, a region in which a target distribution density in the aerial image data to be detected is higher than a preset value, and crop a local region cropped image block.

[0036] The detecting module is configured to perform target detection on the aerial image data to be detected and the local region cropped image block respectively, map a detection frame of the local region cropped image block to the aerial image data to be detected according to a positional relationship, and perform fusion processing on the detection frame of the aerial image data to be detected and the mapped detection frame of the local region cropped image block through a soft non-maximum suppression algorithm to obtain a target detection result of the aerial image data to be detected.

[0037] Optionally, in an embodiment of the present application, the device further comprises a constructing module configured to construct the target density estimation network, specifically, modifying a GhostNetV2 network for a density generation task, removing a convolution layer and a full connection layer used for classification at the end of the network, retaining a starting processing layer and the first 16 bottleneck layers of the network, and modifying a stride of the sixth bottleneck layer to 1 as a front-end feature extraction network of the target density estimation network.

[0038] The receptive field module is configured to expand receptive fields of the P2 layer feature map, the P3 layer feature map and the P4 layer feature map output by the front-end feature extraction network by 4 times, 8 times and 16 times respectively.

[0039] The feature extraction module is configured to extract features of the P4 layer feature map processed by the receptive field module, perform upsampling on the extracted features, and perform channel cascading fusion with the P3 layer feature map processed by the receptive field module to generate a new P3 layer feature map, and continue to extract features of the new P3 layer feature map, perform upsampling on the extracted features, and perform channel cascading fusion with the P2 layer feature map processed by the receptive field module to generate a new P2 layer feature map.

[0040] The receptive field expansion module composed of the plurality of different hollow rate hollow bottleneck layers is configured to generate final output P2 layer feature maps and P3 layer feature maps.

[0041] The P2 layer prediction density map and the P3 layer prediction density map are respectively generated by using a 1x1 convolution to reduce the dimension of the output P2 layer feature map and the P3 layer feature map.

[0042] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the aerial target counting and detection method based on the lightweight density estimation network as described in the above embodiments.

[0043] The fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to perform the aerial target counting and detection method based on the lightweight density estimation network as described in the above embodiments.

[0044] The aerial target counting and detection method based on the lightweight density estimation network of the embodiments of the present application constructs a lightweight aerial image density estimation network GDNet, which is suitable for density estimation of aerial images, has a lightweight network and fast calculation speed, and can generate high-quality aerial image density estimation maps, thereby providing accurate spatial quantity and position distribution feature information for subsequent adaptive cropping of local image blocks based on original high-resolution aerial images. The high-density cropped image blocks are obtained through the spatial position quantity distribution features to perform enlargement detection, thereby significantly improving the detection accuracy of aerial targets.

[0045] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the following drawings in which:

[0047] Figure 1 A flowchart of an aerial target counting and detection method based on a lightweight density estimation network according to an embodiment of the present application is provided.

[0048] Figure 2 A process schematic diagram of an aerial target counting and detection method based on a lightweight density estimation network according to an embodiment of the present application is provided.

[0049] Figure 3 A network structure diagram of a lightweight aerial image density estimation network GDNet according to an embodiment of the present application is provided.

[0050] Figure 4 A receptive field module structure diagram in the GDNet according to an embodiment of the present application is provided.

[0051] Figure 5 A structural diagram of a feature extraction module in a GDNet according to an embodiment of the present application is provided.

[0052] Figure 6 A structural diagram of a receptive field expansion module in a GDNet according to an embodiment of the present application is provided.

[0053] Figure 7 A partial density map effect generated by a GDNet according to an embodiment of the present application is provided.

[0054] Figure 8 A target density map according to an embodiment of the present application is provided.

[0055] Figure 9 A schematic diagram of a density map generated by a GDNet according to an embodiment of the present application is provided.

[0056] Figure 10 A structural diagram of an aerial target counting and detecting device based on a lightweight density estimation network according to an embodiment of the present application is provided.

[0057] Figure 11 A structural diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0059] As shown in Figure 1 and Figure 2 The aerial target counting and detecting method based on a lightweight density estimation network includes the following steps:

[0060] In step S101, aerial image data to be detected is acquired, and the aerial image data to be detected is input into a trained target density estimation network to extract features of the aerial image data to be detected, generate a target density map according to feature information of the aerial image data to be detected, obtain target spatial position distribution features, sum element values in the target density map, and obtain a target quantity in the aerial image data to be detected.

[0061] In an embodiment of the present application, a target density estimation network is constructed, the aerial image data is input into the trained target density estimation network, the target density map corresponding to the aerial image data is obtained, and then the target spatial position distribution feature is obtained, so as to obtain the high-density cropped image block through the spatial position number distribution feature for magnification detection, thereby significantly improving the detection accuracy of the aerial target.

[0062] In an embodiment of the present application, as shown in Figure 3 Before acquiring the aerial image data to be detected, a target density estimation network GDNet is constructed, including the following steps:

[0063] Based on the GhostNetV2 network, the corresponding modification is made for the density generation task, the convolution layer and the full connection layer used for classification at the end of the network are removed, the starting processing layer and the first 16 bottleneck layers of the network are retained, and the stride of the 6th bottleneck layer is modified to 1, which is used as the front-end feature extraction network of the target density estimation network;

[0064] The 4 times, 8 times and 16 times down-sampling P2 layer feature map, P3 layer feature map and P4 layer feature map output by the feature extraction network are respectively expanded by using the receptive field module to expand the receptive field of the feature map, and the structure of the receptive field module is as shown in Figure 4 ;

[0065] The P4 layer feature map processed by the receptive field module is extracted by the feature extraction module, and then is up-sampled and fused with the P3 layer feature map processed by the receptive field module in the channel level to generate a new P3 layer feature map, and the feature extraction module is extracted by the feature extraction module, and then is up-sampled and fused with the P2 layer feature map processed by the receptive field module in the channel level to generate a new P2 layer feature map, and the structure of the feature extraction module is as shown in Figure 5 ;

[0066] The P2 layer feature map and the P3 layer feature map are fused by the receptive field expansion module composed of multiple different hollow bottleneck layers to generate the final output P2 layer feature map and P3 layer feature map, and the structure of the receptive field expansion module is as shown in Figure 6 ;

[0067] The P2 layer feature map and the P3 layer feature map are respectively used to reduce the dimension by using the 1x1 convolution to generate the P2 layer prediction density map and the P3 layer prediction density map.

[0068] It can be understood that in subsequent target detection, the P3 layer density prediction map is used for processing, and the P2 layer prediction density map is used for assisting in reducing the loss function when training the target density estimation network.

[0069] The backbone network adopted by the application is a lightweight network with small calculation amount and high speed; the rear-end processing is performed by using a cavity convolution, the receptive field of each layer feature map is expanded; the feature enhancement module is used to perform the feature pyramid fusion on the multi-layer feature maps, and the context information of each layer feature map is increased.

[0070] In an embodiment of the application, training the target density estimation network comprises:

[0071] Obtaining training aerial image data and labeling the training aerial data;

[0072] Processing the target frame labeled by the training aerial image data, obtaining the center point of the target frame, and generating the corresponding density map of the training aerial image data by using the labeling information;

[0073] Inputting the training aerial image data and the corresponding density map into the target density estimation network to complete the forward propagation, calculating the loss value of the target density estimation network according to the forward propagation result and the real density map of the target density estimation network, updating the target density estimation network according to the loss value, completing the update of the target density estimation network through multiple iterations, and obtaining the trained target density estimation network.

[0074] In an embodiment of the application, processing the target frame labeled by the training aerial image data, obtaining the center point of the target frame, and generating the corresponding density map of the training aerial image data by using the labeling information comprises: converting the image including the center point position label of the target in the training aerial image data into a target position information matrix; and generating the target density map by the geometric self-adaptive Gaussian kernel through the target position information matrix. The partial density map effect generated by the GDNet is shown in Figure 7 .

[0075] In an embodiment of the application, calculating the loss value of the target density estimation network according to the forward propagation result and the real density map of the target density estimation network, and updating the target density estimation network according to the loss value comprises:

[0076] Calculating the first MSE loss of the P2 layer feature map result of the forward propagation and the 4 times down-sampling real density map, and the second MSE loss of the P3 layer feature map result of the forward propagation and the 8 times down-sampling real density map;

[0077] Summing the first MSE loss and the second MSE loss by combining the loss balance coefficient to obtain the loss value of the target density estimation network;

[0078] Updating the parameters of the target density estimation network by using the AdamW optimizer and the cosine annealing learning rate adjustment strategy for the reverse propagation gradient descent of the loss value.

[0079] Specifically, in order to train the target density estimation network, the embodiment of the application uses a UAV to collect aerial image data and labels a target frame to obtain a target detection data set.

[0080] The target frame of the obtained UAV aerial target detection image is processed to obtain a target frame center point, and then a corresponding density map is generated using the labeling information, in the following specific manner:

[0081] First, the picture with the center point position label of the aerial target is converted into a density map, in the following specific manner: if there is a target center position at a pixel point, it is represented as 1, and an image with N target center position labels can be represented as a function:

[0082]

[0083] The matrix containing the target position information is used to generate a target density map through a geometric adaptive Gaussian kernel, in the following formula:

[0084]

[0085] wherein is a Gaussian kernel, x i is the target center point position, is the average distance from the current target center point to the K nearest targets.

[0086] The processed aerial target image and the aerial target density map are input into the network to complete the forward propagation. Then, the result of the forward propagation is calculated with the real density map input into the network to update the model parameters, including the following:

[0087] The MSE loss of the P2 layer result of the forward propagation and the 4 times down-sampling real density map, and the MSE loss of the P3 layer result of the forward propagation and the 8 times down-sampling real density map are calculated, in the following specific formula:

[0088]

[0089]

[0090] wherein D(X i ; Θ) is the feature map calculated by the forward propagation of the i-th picture, D i represents the real density map of the i-th picture, L(Θ) p2 , L(Θ) p3 respectively represent the MSE losses calculated by the P2 layer and the P3 layer, and the final network total loss is:

[0091] L(Θ)=λL(Θ) p2 +L(Θ) p3

[0092] wherein λ is a loss balance coefficient.

[0093] The calculated loss is used to update the model parameters by backpropagation gradient descent using the AdamW optimizer and the cosine annealing learning rate adjustment strategy, thereby completing one iteration, and the training of the target density estimation network is completed by multiple iterations to a specified number of epochs. The number of iterations of the present application can be set to 300 times.

[0094] The embodiments of the present application can sum each element value in the target density map to obtain a count value, and the count value is used as the target quantity value in the aerial image data to be detected. Therefore, the embodiments of the present application can more accurately estimate and count the targets in the aerial scene, and have low network parameter quantity, fast calculation speed, and low MAE and MSE, and high accuracy and stability.

[0095] It can be understood that in the embodiments of the present application, after the aerial image data to be detected is input into the trained target density estimation network, a target density map of the aerial image data to be detected can be obtained, which can represent the spatial position distribution feature of the target. Figure 7 For a aerial image with target density, Figure 8 For Figure 7 The corresponding density map of the corresponding aerial image. Each element point in the target density map represents the probability (or number) of the existence of the target at that point, and the element value is high in the target dense area.

[0096] Specifically, in the annotation data for training the target density estimation network, the position of the target center point is annotated as 1, and the positions of the other points are annotated as 0. The discrete state value of the annotation is converted into a continuous value by a Gaussian function, so that the value of the area closer to the target center position is closer to 1, and the value of the area farther away is closer to 0. In the target area, the sum value is 1. Therefore, by summing the values of all pixel points in the density map, the sum value is obtained, which is the number of all targets in the aerial image data to be detected.

[0097] In step S102, according to the target density map, the region with a target distribution density higher than a preset value in the aerial image data to be detected is calculated using the target spatial position distribution feature, and a local region cropped image block is cropped.

[0098] In an embodiment of the present application, according to the target density map, the region with a target distribution density higher than a preset value in the aerial image data to be detected is calculated using the target spatial position distribution feature, and a local region cropped image block is cropped, comprising:

[0099] The multi-scale sliding window is used to slide on the target density map without overlapping, the sum of density values of all positions in the sliding window is calculated, the total density value in the sliding window is compared with the preset size sliding window threshold, if greater than or equal to the size sliding window threshold, the density value of the current sliding window coverage area is set to 1, if less than the size sliding window threshold, the density value of the current sliding window coverage area is set to 0, and a density mask map of the local region cropped image block is generated.

[0100] The density mask map is divided into connected blocks by using a depth-first search algorithm, a plurality of connected blocks with internal density values of 1 are obtained, for each connected block, a minimum circumscribed rectangle region is calculated, and a local cropped image block is generated according to the corresponding region position on the original image. Figure 9

[0101] Specifically, a target density estimation network is used to obtain a density map of the aerial image data to be detected, for the density map of the training set, a label is used, and for the test set, the picture of the test set is input into the trained lightweight density estimation network GDNet to obtain a P3 level feature map, and then the density map of the original picture size is obtained through 8 times of bilinear interpolation upsampling.

[0102] The target box size in the data is clustered by using a differential evolution clustering algorithm, a plurality of target box sizes are obtained, a multi-scale sliding window is used to slide on the obtained density map without overlapping, then the sum of density values of all positions in the sliding window is calculated, the total density value in the sliding window is compared with the preset size sliding window threshold T, if greater than T, the density value of the current sliding window coverage area is set to 1, if less than the threshold, the density value of the sliding window coverage area is assigned to 0, and thus a density mask map required for subsequent cropping of a local image block is generated.

[0103] On the obtained density mask map, the density mask map is divided into connected blocks by using a depth-first search algorithm, a plurality of connected blocks with internal density values of 1 are obtained, then for each connected block, a minimum circumscribed rectangle region is calculated, and finally a local cropped image block is generated according to the corresponding region position on the original image.

[0104] In step S103, target detection is performed on the aerial image data to be detected and the local region cropped image block respectively, the detection box of the local region cropped image block is mapped to the aerial image data to be detected according to the positional relationship, and the detection box of the aerial image data to be detected and the mapped detection box of the local region cropped image block are fused by using a soft non-maximum suppression algorithm, so that the target detection result of the aerial image data to be detected is obtained.​

[0105] The cropped local region image block is added to the training set to obtain a data enhanced target detection training set. Each picture in the test set is cropped to obtain a local region cropped image block to form a target detection local region cropped image block test set. Any target detector is used to train on the data enhanced training set. The trained target detector is used to infer and predict the pictures in the original test set and the local region cropped image block test set respectively to obtain global picture detection frames and local region cropped image block detection frames. Finally, the local region cropped image block detection frames are mapped back to the original picture, and a soft non-maximum suppression algorithm is used to fuse all the detection frames containing the global picture detection frames and the local region cropped image block detection frames mapped back to the original picture to obtain final target detection frames and output final detection results.

[0106] According to the aerial target counting and detection method based on the lightweight density estimation network provided in the embodiment of the present application, a lightweight aerial target density estimation network GDNet is constructed, feature extraction is performed on an original image using the GDNet, a corresponding density map is generated based on the original image feature information, and spatial position distribution features of the aerial target are obtained. According to the generated aerial target region density map, the spatial position number distribution feature information of the aerial target is used to calculate a region with high target distribution density in the original image, and then the region is cropped and extracted into a local region cropped image block. A target detection algorithm is used to detect the original complete picture and the cropped local cropped image block respectively to obtain corresponding detection frames. The detection frames on the local cropped image block are mapped back to the original image according to the positional relationship, and a soft non-maximum suppression algorithm is used to fuse the original image detection frames and the mapped local cropped image block detection frames to obtain final detection results. Thus, the present application obtains high-density cropped image blocks through spatial position number distribution features for magnification detection, and the detection accuracy of the aerial target is significantly improved.

[0107] Secondly, the aerial target counting and detection device based on the lightweight density estimation network provided in the embodiment of the present application is described with reference to the accompanying drawings.

[0108] Figure 10 The aerial target counting and detection device based on the lightweight density estimation network provided in the embodiment of the present application is shown in a structural schematic diagram.

[0109] As shown in Figure 10 The aerial target counting and detection device 10 based on the lightweight density estimation network includes a generation module 100, a cropping module 200, and a detection module 300.

[0110] The generation module 100 is configured to obtain aerial image data to be detected, input the aerial image data to be detected into the trained target density estimation network, perform feature extraction on the aerial image data to be detected, generate a target density map according to feature information of the aerial image data to be detected, obtain a target spatial position distribution feature, and sum element values in the target density map to obtain a target quantity in the aerial image data to be detected.

[0111] The cropping module 200 is configured to calculate a region in which a target distribution density in the aerial image data to be detected is higher than a preset value according to the target spatial position distribution feature of the target density map, and crop a local region cropped image block.

[0112] The detection module 300 is configured to perform target detection on the aerial image data to be detected and the local region cropped image block respectively, map a detection frame of the local region cropped image block to the aerial image data to be detected according to a positional relationship, and perform fusion processing on the detection frame of the aerial image data to be detected and the mapped detection frame of the local region cropped image block through a soft non-maximum suppression algorithm to obtain a target detection result of the aerial image data to be detected.

[0113] In an embodiment of the present application, the aerial target counting and detection device 10 based on the lightweight density estimation network further comprises a construction module configured to construct the target density estimation network, specifically, modifying a GhostNetV2 network for a density generation task, removing a convolution layer and a fully connected layer used for classification at the end of the network, retaining a starting processing layer and the first 16 bottleneck layers of the network, and modifying a stride of the sixth bottleneck layer to 1 as a front-end feature extraction network of the target density estimation network.

[0114] The 4-fold, 8-fold and 16-fold down-sampling P2 layer feature map, P3 layer feature map and P4 layer feature map output by the front-end feature extraction network are respectively expanded in receptive field by using a receptive field module;

[0115] The P4 layer feature map processed by the receptive field module is extracted by a feature extraction module, up-sampled, and then fused with the P3 layer feature map processed by the receptive field module in a channel level to generate a new P3 layer feature map, and the new P3 layer feature map is extracted by the feature extraction module, up-sampled, and then fused with the P2 layer feature map processed by the receptive field module in a channel level to generate a new P2 layer feature map;

[0116] The P2 layer feature map and the P3 layer feature map are fused by a receptive field expansion module composed of multiple hollow bottleneck layers with different hollow rates to generate final output P2 layer feature map and P3 layer feature map;

[0117] The P2 layer prediction density map and the P3 layer prediction density map are respectively generated by using 1x1 convolution to reduce the dimension of the output P2 layer feature map and the P3 layer feature map.

[0118] In an embodiment of the present application, the aerial target counting and detection device 10 based on the lightweight density estimation network further comprises a training module.

[0119] The training module is configured to obtain training aerial image data and label the training aerial data.

[0120] The target frame center point is obtained by processing the target frame labeled on the training aerial image data, and the density map corresponding to the training aerial image data is generated by using the labeling information.

[0121] The training aerial image data and the corresponding density map are input into the target density estimation network to complete the forward propagation, the loss value of the target density estimation network is calculated according to the forward propagation result and the real density map of the target density estimation network, the target density estimation network is updated according to the loss value, the update of the target density estimation network is completed through multiple iterations, and the trained target density estimation network is obtained.

[0122] In an embodiment of the present application, the target frame labeled on the training aerial image data is processed to obtain the target frame center point, and the density map corresponding to the training aerial image data is generated by using the labeling information, which comprises:

[0123] The image including the center point position label of the target in the training aerial image data is converted into a target position information matrix.

[0124] The target position information matrix is converted into a target density map by a geometric self-adaptive Gaussian kernel.

[0125] Optionally, in an embodiment of the present application, the loss value of the target density estimation network is calculated according to the forward propagation result and the real density map of the target density estimation network, and the target density estimation network is updated according to the loss value, which comprises:

[0126] The first MSE loss of the P2 layer feature map result of the forward propagation and the 4 times down-sampling real density map, and the second MSE loss of the P3 layer feature map result of the forward propagation and the 8 times down-sampling real density map are calculated.

[0127] The first MSE loss and the second MSE loss are summed by combining the loss balance coefficient to obtain the loss value of the target density estimation network.

[0128] The target density estimation network parameters are updated by using the AdamW optimizer and the cosine annealing learning rate adjustment strategy to perform the back propagation gradient descent on the loss value.

[0129] In an embodiment of the present application, according to the target density map, the region with a target distribution density higher than a preset value in the aerial image data to be detected is calculated by using the target spatial position distribution feature, and a local region cropped image block is cropped, comprising:

[0130] A multi-scale sliding window is adopted to slide on the target density map without overlapping, the sum of density values of all positions in the sliding window is calculated, the total density value in the sliding window is compared with a preset size sliding window threshold, if greater than or equal to the size sliding window threshold, the density value of the current sliding window coverage region is set to 1, if less than the size sliding window threshold, the density value of the current sliding window coverage region is set to 0, and a density mask map of the local region cropped image block is generated;

[0131] A depth-first search algorithm is used to divide the density mask map into connected blocks, a plurality of connected blocks with an internal density value of 1 are obtained, for each connected block, a minimum circumscribed rectangle region is calculated, and a local region cropped image block is cropped from the corresponding region position on the aerial image data to be detected according to each minimum circumscribed rectangle region.

[0132] It should be noted that the foregoing explanation and description of the embodiment of the aerial target counting and detection method based on the lightweight density estimation network also applies to the embodiment of the aerial target counting and detection device based on the lightweight density estimation network, which will not be described here.

[0133] According to the aerial target counting and detection device based on the lightweight density estimation network provided in the embodiment of the present application, based on the lightweight aerial target density estimation network GDNet constructed, the GDNet is used for feature extraction of an original image, a corresponding density map is generated based on the original image feature information, and the spatial position distribution feature of the aerial target is obtained, according to the generated aerial target region density map, the spatial position number distribution feature information of the aerial target is used to calculate the region with a high target distribution density in the original image, and then the region is cropped and extracted into a local region cropped image block, the original complete picture and the cropped local cropped image block are detected by using a target detection algorithm respectively, and corresponding detection frames are obtained, the detection frames on the local cropped image block are mapped back to the original image according to the positional relationship, and the original image detection frame and the mapped local cropped image block detection frame are post-processed by using a soft non-maximum suppression algorithm, and finally the detection result is obtained. The aerial target detection method provided in the present application obtains a high-density cropped image block by using the spatial position number distribution feature for magnification detection, and the detection accuracy of the aerial target is significantly improved.

[0134] Figure 11 The electronic device provided in the embodiment of the present application is shown in a structural schematic diagram. The electronic device can include:

[0135] The memory 1001, the processor 1002 and the computer program stored in the memory 1001 and capable of running on the processor 1002.

[0136] The processor 1002 implements the aerial target counting and detection method based on the lightweight density estimation network provided in the above embodiments when executing the program.

[0137] Further, the electronic device further comprises:

[0138] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002.

[0139] The memory 1001 is used for storing the computer program capable of running on the processor 1002.

[0140] The memory 1001 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0141] If the memory 1001, the processor 1002 and the communication interface 1003 are independently implemented, the communication interface 1003, the memory 1001 and the processor 1002 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 10 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or only one type of bus.

[0142] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can complete communication between each other through an internal interface.

[0143] The processor 1002 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0144] The embodiment also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the aerial target counting and detection method based on the lightweight density estimation network.

[0145] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0146] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0147] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing the specified logical functions or steps, and the preferred embodiments of the present application also include the possibility that the described functions may be implemented by alternative hardware, such as through the use of application specific integrated circuits (ASICs). The described processes may, in some embodiments, be performed by a computer system in response to the execution of one or more sequences of one or more instructions contained in a system memory. Such instructions can be read into the system memory from another computer readable medium, such as a storage device or a network link. Execution of the sequences of instructions contained in the system memory can cause the computer system to perform the described processes.

[0148] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if implemented in hardware, any of the following technologies known in the art or their combinations can be used: discrete logic circuit with logic gate circuit for implementing logical functions on data signals, application specific integrated circuit with suitable combination of logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0149] Those skilled in the art can understand that all or part of the steps of the foregoing method embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium. When the programs are executed, the steps of the method embodiments or a combination thereof are included.

Claims

1. A method for aerial target counting and detection based on a lightweight density estimation network, characterized in that, Includes the following steps: Aerial image data to be detected is acquired, and the aerial image data to be detected is input into a trained target density estimation network to extract features from the aerial image data to be detected. A target density map is generated based on the feature information of the aerial image data to be detected in order to obtain the spatial distribution features of the targets. The element values ​​in the target density map are summed to obtain the number of targets in the aerial image data to be detected. Based on the target spatial location distribution characteristics of the target density map, calculate the regions in the aerial image data to be detected where the target distribution density is higher than a preset value, and crop out the local region cropped image blocks. Target detection is performed on the aerial image data to be detected and the cropped image block of the local region, respectively. The detection box of the cropped image block of the local region is mapped onto the aerial image data to be detected according to the positional relationship. The detection box of the aerial image data to be detected and the mapped detection box of the cropped image block of the local region are fused by a soft nonmaximum suppression algorithm to obtain the target detection result of the aerial image data to be detected. Before acquiring the aerial image data to be detected, the target density estimation network is constructed, including: Based on the GhostNetV2 network, modifications were made for the density generation task. The final convolutional and fully connected layers used for classification were removed, while the initial processing layer and the first 16 bottleneck layers were retained. The stride of the 6th bottleneck layer was modified to 1, which was then used as the front-end feature extraction network of the target density estimation network. The receptive field of the feature maps at layers P2, P3, and P4, which are downsampled by 4x, 8x, and 16x respectively, is expanded using the receptive field module. The P4 layer feature map processed by the receptive field module is then upsampled and fused with the P3 layer feature map processed by the receptive field module through channel concatenation to generate a new P3 layer feature map. The feature map is then upsampled and fused with the P2 layer feature map processed by the receptive field module through channel concatenation to generate a new P2 layer feature map. The final output P2 layer feature map and P3 layer feature map are generated by a receptive field expansion module composed of multiple cascaded void bottleneck layers with different void ratios. The output P2 layer feature maps and P3 layer feature maps are reduced in dimensionality using 1x1 convolutions to generate P2 layer prediction density maps and P3 layer prediction density maps, respectively.

2. The method according to claim 1, characterized in that, Training the target density estimation network includes: Acquire training aerial image data and annotate the training aerial image data; The target bounding boxes labeled in the training aerial image data are processed to obtain the center point of the target bounding boxes, and the density map corresponding to the training aerial image data is generated using the labeling information. The training aerial image data and the corresponding density map are input into the target density estimation network to complete forward propagation. The loss value of the target density estimation network is calculated based on the forward propagation result and the true density map of the target density estimation network. The target density estimation network is updated based on the loss value. The target density estimation network is updated through multiple iterations to obtain a trained target density estimation network.

3. The method according to claim 2, characterized in that, The target bounding boxes labeled in the training aerial image data are processed to obtain the center points of the target bounding boxes, and a density map corresponding to the training aerial image data is generated using the labeling information, including: The training aerial image data, including the target's center point location label, is converted into a target location information matrix. The target location information matrix is ​​used to generate a target density map using a geometrically adaptive Gaussian kernel.

4. The method according to claim 2, characterized in that, Calculating the loss value of the target density estimation network based on the forward propagation results and the true density map of the target density estimation network, and updating the target density estimation network based on the loss value, includes: Calculate the first MSE loss of the P2 layer feature map result after forward propagation and the 4x downsampled true density map, and the second MSE loss of the P3 layer feature map result after forward propagation and the 8x downsampled true density map; The loss value of the target density estimation network is obtained by summing the first MSE loss and the second MSE loss using the loss balance coefficient. The target density estimation network parameters are updated by backpropagation gradient descent using the AdamW optimizer and cosine annealing learning rate adjustment strategy on the loss value.

5. The method according to claim 1, 3, or 4, characterized in that, Based on the target density map, the regions in the aerial image data to be detected with target distribution density higher than a preset value are calculated using the target spatial location distribution features, and cropped to obtain local region cropped image blocks, including: A multi-scale sliding window is used to slide non-overlappingly on the target density map. The sum of the density values ​​at all positions within the sliding window is calculated. The total density value within the sliding window is compared with a pre-set size sliding window threshold. If it is greater than or equal to the size sliding window threshold, the density value of the area covered by the current sliding window is set to 1. If it is less than the size sliding window threshold, the density value of the area covered by the current sliding window is set to 0. A density mask map of the cropped image patch of the local region is generated. The density mask image is divided into connected components using a depth-first search algorithm to obtain multiple connected components with an internal density value of 1. For each connected component, its minimum circumscribed rectangle region is calculated. Based on each minimum circumscribed rectangle region, the corresponding region position on the aerial image data to be detected is cropped to generate the local region cropped image block.

6. An aerial target counting and detection device based on a lightweight density estimation network, characterized in that, include: The generation module is used to acquire aerial image data to be detected, input the aerial image data to be detected into a trained target density estimation network to extract features from the aerial image data to be detected, generate a target density map based on the feature information of the aerial image data to obtain the spatial distribution features of the targets, and sum the element values ​​in the target density map to obtain the number of targets in the aerial image data to be detected. The cropping module is used to calculate the region in the aerial image data to be detected where the target distribution density is higher than a preset value based on the target spatial location distribution characteristics of the target density map, and crop the region to obtain a cropped image block. The detection module is used to perform target detection on the aerial image data to be detected and the local region cropped image block respectively, map the detection box of the local region cropped image block to the aerial image data to be detected according to the positional relationship, and perform fusion processing on the detection box of the aerial image data to be detected and the mapped detection box of the local region cropped image block through a soft nonmaximum suppression algorithm to obtain the target detection result of the aerial image data to be detected; The construction module is used to construct the target density estimation network. Specifically, it is based on the GhostNetV2 network and modified for the density generation task. The convolutional layer and fully connected layer used for classification at the end of the network are removed, the network's initial processing layer and the first 16 bottleneck layers are retained, and the stride of the 6th bottleneck layer is modified to 1, which serves as the front-end feature extraction network of the target density estimation network. The receptive field of the feature maps at layers P2, P3, and P4, which are downsampled by 4x, 8x, and 16x respectively, is expanded using the receptive field module. The P4 layer feature map processed by the receptive field module is then upsampled and fused with the P3 layer feature map processed by the receptive field module through channel concatenation to generate a new P3 layer feature map. The feature map is then upsampled and fused with the P2 layer feature map processed by the receptive field module through channel concatenation to generate a new P2 layer feature map. The final output P2 layer feature map and P3 layer feature map are generated by a receptive field expansion module composed of multiple cascaded void bottleneck layers with different void ratios. The output P2 layer feature maps and P3 layer feature maps are reduced in dimensionality using 1x1 convolutions to generate P2 layer prediction density maps and P3 layer prediction density maps, respectively.

7. An electronic device, characterized in that, include: The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aerial target counting and detection method based on a lightweight density estimation network as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the aerial target counting and detection method based on a lightweight density estimation network as described in any one of claims 1-5.

Images