UAV Image Target Counting Method Based on Parameter Adaptive Dynamic Neural Network

Abstract

A method for counting drone image targets based on a parameter-adaptive dynamic neural network, the steps of which include: 1) An aerial photography device on the drone collects ground images; 2) The collected images are preprocessed; 3) A feature extraction network is used to detect targets in the processed images; 4) The detected targets are counted. In step 2), the scale is dynamically divided and the image is segmented according to the flight altitude and shooting depression angle of the drone; in step 3), a feature extraction network with enhanced spatial information is used for the input feature map to extract the position information of the targets. The dilation rate parameter of the convolution in the receptive field selection module after the feature extraction network is determined according to the segmented image. A dilation rate parameter adaptive strategy is designed to calculate the scale range and segment the image according to the flight state, and to adjust the dilation rate parameter in the receptive field selection module. This method significantly reduces the inference latency, improves the MAE and MSE, and also has excellent counting accuracy.

Description

Technical Field

[0001] The present invention belongs to the application technology of deep learning in the field of computer vision, and specifically relates to a method for counting drone image targets based on a parameter-adaptive dynamic neural network. Background Art

[0002] Embedded vision systems endow drones with environmental perception capabilities, and target counting is one of the key technologies for realizing drone perception capabilities. Drones equipped with embedded vision systems have the advantages of high mobility and flexible deployment. Combining with target counting technology, it expands multiple application scenarios such as crowd counting, animal counting, vehicle counting, and environmental surveys. For example, in the field of traffic monitoring, drones can complete tasks such as vehicle flow monitoring and traffic guidance in the air. Compared with fixed ground traffic monitoring, drones can meet the needs of wide-area coverage and real-time monitoring in modern traffic.

[0003] However, due to the limited computing and storage capabilities of drones, it is difficult to deploy relatively complex deep learning models for online target counting. A natural step is to lightweight the model.

[0004] In addition, considering the dynamic deployment of drones and the changes in flight states, compared with ground monitoring, drone aerial image processing also faces some special problems:

[0005] (1) Due to the dynamic deployment of drones, aerial targets are often submerged in complex and changing backgrounds (see Fig. 1(a)), which distracts the model's attention to the area of interest and reduces the counting accuracy [1]. Although methods such as binary masks [2] and data augmentation [3] can reduce the interference of the background on the target, they require the support of additional modules, increasing the processing burden. Context information fusion [4] is also a feasible solution, but it requires a strong correlation between the target and the background (e.g., the same image background), which does not conform to the constantly changing background in aerial photography.

[0006] (2) Due to the changes in the flight altitude and shooting depression angle of the UAV, even for targets with the same physical size, there are often huge differences in pixels in the image, ranging from several to thousands [5] (see Fig. 1(b)). As a solution, the image pyramid model [6] inputs images with multi-level downsampling into the model to extract multi-scale information. Lin et al. [7] proposed the feature pyramid, which takes into account the shallow representation information and deep semantic information, facilitating the extraction of multi-scale information and thus alleviating the problem of uneven target scales. These methods for fusing multi-scale features consume a large amount of computing and video memory resources and are difficult to be directly deployed on the UAV platform. The sample-dependent dynamic neural network [8] is a feasible method for dealing with image scale changes, mainly including dynamic architectures and dynamic parameters. When processing samples of different scales, the dynamic architecture activates different layers or branches through strategies such as early exit [9], skip layer

[10] , and mixture of experts

[11] to improve the operation efficiency. However, due to the inability to obtain sufficient confidence, the model does not have the condition for early exit or skip layer in many cases, which is equivalent to automatically degenerating into a static neural network for "complete inference". For dynamic parameters, representative solutions include parameter adjustment

[12] , weight prediction

[13] , and dynamic features

[14] , etc. These methods improve the model expression ability by transforming the model parameters, but how to design the pre-network to guide parameter adjustment still needs further exploration.

[0007] (3) Compared with the images collected by ground monitoring, the aerial photography of the UAV has a wider coverage area. Especially, the large difference in the distribution of counting targets in the image brings serious isolated cluster problems

[15] (as shown in Fig. 1(c)). In the densely distributed area, the estimation of the density map is often inaccurate, while in the sparsely distributed area, isolated points are easily ignored. Training the model with the Euclidean loss function cannot effectively reduce the influence of distribution differences

[16] . It is necessary to redesign a loss function to balance the contribution of the losses in the dense and sparse areas to the total loss. Summary of the Invention

[0008] Considering the limited resources of the UAV platform and the particularity of aerial photography images, the present invention proposes a method for counting UAV image targets based on a parameter-adaptive dynamic neural network, achieving dual optimization of counting accuracy and inference latency.

[0009] The method for counting UAV image targets based on the parameter-adaptive dynamic neural network of the present invention includes the following steps: 1) The aerial photography device on the UAV collects ground images; 2) Preprocess the collected UAV images; 3) Use the feature extraction network to detect targets in the processed images; 4) Count the detected targets (the method for target counting can use the commonly used density estimation method in the prior art).

[0010] In step 2), the image is dynamically scaled and segmented based on the drone's flight altitude and shooting angle. In step 3), a feature extraction network with spatial information enhancement is used on the input feature map to extract the target's location information. The expansion rate parameter of the convolution in the receptive field of the feature extraction network is determined based on the segmented image.

[0011] An adaptive strategy for the expansion rate parameters is designed to calculate the scale range and segment the image according to the flight state, and to adjust the expansion rate parameters of the receptive field output by the feature extraction network.

[0012] The main contributions of this invention include:

[0013] First, to address problem (1), a feature extraction network with spatial information enhancement is constructed, and the attention mechanism is used to aggregate horizontal and vertical feature information, thereby enhancing the position representation of the target in the entire image and reducing background interference.

[0014] Second, for problem (2), we explore the adaptive method of expansion rate parameters.

[0015] In the image preprocessing stage, the scale and segmentation of the image are dynamically divided according to the flight altitude and shooting angle to avoid ranging overhead, and then the convolution expansion rate parameter is determined based on the segmented image.

[0016] During the model training and inference stages, a receptive field selection module with adjustable expansion rate parameters is designed, which makes the trained model scale-generalizable without increasing redundant calculations.

[0017] Third, to address problem (3), a block-level target distribution-aware loss function is designed, and the k-root method is used to improve the Euclidean loss function to balance the proportion of losses in areas with different target distributions in the aerial image in the total loss, thereby reducing the negative impact of target distribution differences on counting.

[0018] Experimental results on multiple authoritative datasets demonstrate that our proposed method significantly reduces model inference latency while achieving counting accuracy approaching that of static models relying on powerful basic neural networks like VGG and ResNet. Compared to mainstream lightweight models, our method achieves both more accurate counting and similar inference latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1(a) to Figure 1(c) The characteristics of the drone aerial image are shown, where:

[0020] Figure 1(a) shows a changing background, Figure 1(b) shows a large scale span, and Figure 1(c) shows a large spatial distribution gap.

[0021] Figure 2 The invention discloses a UAV image target counting (model) framework based on a parameter-adaptive dynamic neural network.

[0022] Figure 3 It is a schematic diagram of attention enhanced by spatial information.

[0023] Figure 4(a) to Figure 4(c) They are respectively schematic diagrams of scale range division when the flight state changes.

[0024] Figure 5 They are schematic diagrams of each scale range.

[0025] Figure 6 It is an improved loss calculation framework.

[0026] Figure 7(a) to Figure 7(c) It is an experimental comparison graph on the CARPK dataset (the first row is the original image; the second row is the image segmentation; the third row is the predicted density map of the HLCNN

[26] model; the fourth row is the predicted density map of the method of the present invention).

[0027] Figure 8(a) to Figure 8(c) It is an experimental comparison graph on the PUCPR+ dataset (the first row is the original image; the second row is the image segmentation; the third row is the predicted density map of the HLCNN

[26] model; the fourth row is the predicted density map of the method of the present invention).

[0028] Figure 9(a) to Figure 9(c) It is an experimental comparison graph on the VisDrone2019 Vehicle dataset (the first row is the original image; the second row is the image segmentation; the third row is the ground truth density map; the fourth row is the predicted density map of the method in this paper). Detailed implementation manners

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation manners.

[0030] 1 Overview

[0031] Aiming at the problems of complex background interference, large-scale changes and target distribution differences in the target counting of UAV images, the present invention proposes a UAV image target counting method based on a parameter-adaptive dynamic neural network to achieve dual optimization of counting accuracy and inference latency.

[0032] Aggregate the horizontal and vertical weights of features through the attention module to enhance spatial position information and suppress background interference.

[0033] Calculate the range of each scale in the aerial image according to the UAV flight state parameters and segment the image to avoid the computational overhead caused by ranging.

[0034] After receiving the segmented image, the Switch function guides the adjustment of the convolutional dilation rate parameter to enhance the scale generalization ability of the model (i.e., the method of the present invention).

[0035] The traditional loss function is optimized by the k-root method to alleviate the problem of isolated clusters caused by the difference in target distribution in aerial images.

[0036] Experimental results on three authoritative datasets show that, compared with the CSRNet model, the method proposed in this invention significantly reduces the inference latency while the MAE is improved by about 29.4%-54.0% and the MSE is improved by about 28.6%-41.2%; compared with lightweight models (MCNN and MobileCount), the method proposed in this invention shows better performance in counting accuracy.

[0037] 2 Scheme Design

[0038] This section introduces the design and implementation of the method for counting drone image targets with a parameter-adaptive dynamic neural network. As Figure 2 shown, the method proposed in this invention includes three interdependent technical elements: (1) a feature extraction network with enhanced spatial information; (2) dilation rate parameter adaption; (3) a block-level target distribution-aware loss function.

[0039] 2.1 Feature Extraction Network with Enhanced Spatial Information

[0040] Mobilenet V3

[17] , as a lightweight feature extraction network, is suitable for deployment on devices with limited computing power. Considering the interference of the background in aerial images, this subsection improves the attention mechanism in Mobilenet V3, comprehensively considering the horizontal and vertical weights of features to enhance spatial information. The structure is shown in Figure 3 .

[0041] First, the input feature map r with dimensions C×G×W is horizontally compressed and vertically compressed, and the related operation functions are denoted as S1(·) and S2(·). The compression operation generates channel statistics in two directions through global average pooling layers, aggregating the feature map in the spatial dimension. The output of the c-th channel at height g is expressed as

[0042]

[0043] Similarly, the output of the c-th channel at width w is

[0044]

[0045] After the feature map is compressed, the generated horizontal feature and vertical feature are denoted as Z1 and Z2 respectively, that is

[0046]

[0047] where the dimensions of Z1 and Z2 are C×G×1 and C×1×W respectively.

[0048] Then, the intermediate results Z1 and Z2 are input into the excitation operation functions E1(·) and E2(·) to generate the horizontal weight T1 and the vertical weight T2, expressed as

[0049]

[0050] where F1 and F2 are 1×1 convolution operations, δ is batch normalization, and the activation function σ(·) is expressed as

[18]

[0051]

[0052] The purpose is to enhance the nonlinear expression ability of the model without increasing the computational overhead. Compared with the traditional activation functions that focus on improving accuracy, the σ of the present invention balances the calculation speed and the counting accuracy.

[0053] Finally, r is reweighted by T1 and T2 to obtain

[0054] R = U(r, T1, T2) = r × T1 × T2 (6)

[0055] The position information of the target is aggregated in R and enters the next layer of the model.

[0056] 2.2 Adaptive dilation rate parameter

[0057] Although the output R of Equation (b) aggregates the spatial position information, when the receptive field size is fixed, the model can only adapt to a certain scale range

[19] . For aerial images with multiple scales coexisting, this subsection designs an adaptive dilation rate parameter strategy, which integrates scale discrimination based on flight state perception, image segmentation based on scale discrimination, and receptive field selection based on scale perception. This strategy calculates the scale range according to the flight state and segments the image, and then selects the dilation rate parameter matching the scale in the convolution.

[0058] 2.2.1 Scale discrimination based on flight state perception

[0059] The scale of the target in the image is mainly determined by the distance from the target to the UAV camera. Scale calculation usually relies on ranging, and common methods include binocular cameras

[20] and deep learning ranging methods

[21] , which have high requirements for computing power. This subsection designs a simple scale discrimination method to judge the target scale by referring to the UAV flight height h and the depression angle β, without the need to rely on a dedicated ranging module, avoiding additional ranging overhead.

[0060] As shown in Figure 4, taking the point where the UAV projects on the ground as the origin, the ground as the horizontal axis, and the direction of the intersection of the visual midline and the ground relative to the origin as the positive direction of the horizontal axis, a rectangular coordinate system is established. When the UAV hovers at the height h, the coordinates of the ground targets with distances d1 and d2 from it are respectively and When the perspective angle of the drone is α and the depression angle is β, the coordinates of the intersection points of the lower line of sight and the upper line of sight with the ground are (τ1, 0) and (τ2, 0) respectively, where

[0061]

[0062] For a target located at (x, 0), the drone calculates the scale based on the flight altitude h and the depression angle β, as shown in Table 1. Taking a drone with a flight altitude h ∈ (0, d1) as an example, the images it captures can include three scales: small, medium, and large, corresponding to Figure 4(a) to Figure 4(c) Regions III, II, and I.

[0063] Table 1 Rules for dividing the scale range when the flight state changes

[0064]

[0065] 2.2.2 Image segmentation based on scale discrimination

[0066] Next, segment the image according to the result of scale discrimination. Figure 4(a) to Figure 4(c) The parts of Regions III, II, and I intersecting with the ground correspond to the proportions of the small, medium, and large scale regions in the aerial image, which are respectively denoted as ω1, ω2, and ω3. Given the flight altitude h and the depression angle β, the values of these proportions are shown in Table 2.

[0067] Table 2 Proportions of each scale when the flight state changes

[0068]

[0069] Taking Figure 5 the shown aerial scene as an example, the range of the flight altitude of the drone in it, and the intersection points of the upper line of sight and the lower line of sight with the ground is

[0070]

[0071] If the target in the image falls within the range it is regarded as a small-scale target; if within the range it is regarded as a medium-scale target; if within the range it is regarded as a large-scale target. According to the angles of each scale range in the perspective angle α, the proportions of the small, medium, and large scales can be calculated as

[0072]

[0073] Segment the image according to the above proportions, and input the segmented images into the feature extraction network and the Switch function respectively to guide the adjustment of the convolution parameters.

[0074] 2.2.3 Selection of scale-aware receptive fields

[0075] In this subsection, a scale-aware receptive field selection strategy is designed. The Switch function determines the convolutional dilation rate according to the image scale, and then obtains the optimal receptive field range.

[0076] Stack three convolutions with adjustable dilation rates to achieve a dynamic receptive field. The convolutional dilation rate is represented as a triple Q = (q1, q2, q3), where the three elements are relatively prime to make full use of each pixel information and avoid the grid effect

[22] . As Figure 2 shown in the scale-aware receptive field selection module, multiple sets of values of Q are set so that the receptive field of the model covers small, medium, and large scale ranges, which are respectively labeled as III, II, and I for distinction. When the Switch function recognizes that the input is an image of a specific scale, it guides the convolution to adjust to the matching dilation rate. q1, q2, and q3 are the dilation rates of the three convolutions respectively.

[0077] Dynamically selecting the receptive field not only allows the model to focus on the targets within a certain scale range, but also ensures that all scale training data can be learned. The trained parameters learn scale-invariant features, enabling the model to accurately detect the same type of targets at different scales. In addition, relying on the Switch function to adjust the model parameters can maintain the single-column structure of the model. Compared with the multi-column structure

[23] , it reduces the number of parameters of the model; compared with the multi-branch structure

[15] , it avoids redundant convolutional operations on the scale-mismatched branches.

[0078] The scale discrimination method for flight state perception uses the ranges of each scale in the image as the basis for image segmentation. The receptive field selection strategy adaptively adjusts the dilation rate parameters according to the image scale, thereby expanding the receptive field coverage range without increasing the model size and being able to adapt to various scale ranges of the image.

[0079] 2.3 Construction of the block-level target distribution-aware loss function

[0080] To balance the proportion of the loss in the dense and sparse regions in the total loss, this section constructs a block-level target distribution-aware loss function to enable the model to cope with the fluctuations in the target spatial distribution.

[0081] Figure 6 The loss calculation framework is given. First, the predicted entire density map and the ground-truth density map are uniformly divided into m blocks, and the corresponding blocks are labeled as A i＝1,2,…,m , a block-level constraint is established for the density map to roughly divide the dense and sparse regions; then, the Euclidean loss and the counting loss of A i＝1,2,…,m are calculated and taken to the 1 / k power, so that the blocks with dense targets are suppressed, while the blocks with sparse targets are emphasized; finally, the losses of A i＝1,2,…,m after reweighting are summed to obtain the final loss of the density map, which is used to train the model.

[0082] Let \(F(X i,j ; \Theta)\) and represent the predicted density map and the ground-truth density map of the \(j\)-th block of the \(i\)-th image, and represent the predicted count and the ground-truth count of the \(j\)-th block of the \(i\)-th image. The improved loss function is expressed as

[0083]

[0084] where \(N\) represents the total number of samples, \(\Theta\) is the parameter to be trained, and \(\lambda\) is the weight of the count loss. The optimal value of \(k\) will be determined through experiments later.

[0085] 3 Experimental Design and Result Analysis

[0086] The method proposed in this invention is implemented in the Pytorch framework, using Adam to optimize the parameters and setting the learning rate to \(1e - 5\). The model parameters are randomly initialized by a Gaussian distribution, with a mean of zero and a standard deviation of \(0.01\).

[0087] The mean absolute error (MAE) and the mean square error (MSE) are used as evaluation metrics, which respectively reflect the accuracy and robustness of the network.

[0088]

[0089]

[0090] where \(N\) is the total number of image samples, represents the ground-truth density map count of the \(i\)-th image, represents the predicted density map count of the \(i\)-th image.

[0091] According to the literature

[19] , the target pixel sizes in each scale range are defined (large scale: less than \(32\times32\); medium scale: \(32\times32\) to \(96\times96\); small scale: greater than \(96\times96\)), and \(d1\) and \(d2\) are set to 40 meters and 100 meters respectively, so that when the drone shoots vertically downward, no matter what height it is at, the target scale conforms to the above definition. In the receptive field selection strategy, three sets of values are set as parameter options for the dilation rate:

[0092] · \(Q=(1,1,1)\) is used to identify large scales.

[0093] · \(Q=(1,2,3)\) is used to identify medium scales.

[0094] · \(Q=(3,4,5)\) is used to identify small scales.

[0095] Using a continuous dilation rate can make the model smoothly connect the scale changes. \(\lambda\) in Equation (7) is set to \(0.1\).

[0096] Table 3 shows the structure of the feature extraction network. Among them, Conv2d represents a 3×3 convolutional layer. Each Bneck describes an operation with n identical, stride s, channel expansion ratio t, and output channels c, and sets whether to use spatial information enhanced attention.

[0097] Feature extraction network structure after optimizing the attention mechanism

[0098]

[0099] To objectively evaluate the performance of the proposed scheme of this invention, multiple public datasets were selected for comparative analysis, and ablation experiments were designed to prove the effectiveness of each module and its contribution to the overall performance.

[0100] 3.1 Results under different datasets

[0101] A comparative experiment was conducted on the value of parameter k in Equation (7) on the VisDrone2019 Vehicle dataset

[15] . Table 4 shows the result comparison of the optimal value of parameter k in the block-level target distribution awareness loss function. It can be seen from Table 4 that the errors of MAE and MSE are the smallest when k = 4, and this value is used in subsequent experiments.

[0102] CARPK and PUCPR+

[24] are drone vehicle counting datasets, which contain nearly 90,000 cars captured from different parking lots. There are up to 331 cars in a single image, as few as no cars, and both crowded and sparse vehicle counting scenarios.

[0103] Table 4 Optimal value of parameter k

[0104]

[0105] Table 5 lists the comparison results of the proposed method of this invention and the baseline method on the CARPK and PUCPR+ datasets. Compared with the lightweight model LMSFFNet

[28] with multi-branch scale information fusion, the MAE and MSE tested in the PUCPR+ dataset of the method of this invention are reduced by 10.5% and 2.9% respectively, while maintaining a relatively low number of parameters and computational complexity. This shows that when processing images with small target scale fluctuations, the receptive field selection strategy proposed in this invention can effectively avoid the negative impact of scale mismatch on counting accuracy. Compared with LMSFFNet, the MAE of the method proposed in this invention on the CARPK dataset is reduced by 5.1%, but the MSE is inferior to LMSFFNet, indicating that when there are interference regions in the image, the prediction results of the model have certain fluctuations. Although HLCNN

[26] is better than the method of this invention in terms of MAE and MSE, it is difficult to be directly deployed on the drone platform due to its large number of model parameters.

[0106] Comparison on the CARPK and PUCPR+ datasets

[0107]

[0108] Some of the predicted density maps obtained by the method proposed in this invention on the CARPK dataset are shown in Figure 7(a) to Figure 7(c) . Figures 7(a) and 7(b) show that the proposed method can accurately predict both crowded and sparse scenarios. In the original image of Figure 7(c), there is a shadow covering the vehicle target, while in the predicted density map of the model, not all the vehicles in the shadow area are marked. It can be seen that insufficient light is one of the reasons why the MSE of the method proposed in Table 5 is slightly higher than that of the existing methods. Figure 8(a) to Figure 8(c) The predicted density maps obtained by the proposed method on the PUCPR+ dataset are given, which are relatively close to the effect of Figure 7(a) to Figure 7(c) .

[0109] VisDrone2019 Vehicle is a public object detection dataset containing categories such as cars, vans, trucks, and buses for vehicle counting. To be as close as possible to the actual scenario and increase the challenge of counting, the images contain severely occluded or truncated objects. The images are taken from drones at different flight heights and shooting angles of depression, covering a multi-scale range.

[0110] The comparison results on the VisDrone2019 Vehicle dataset are shown in Table 6. The MAE of the method proposed in this invention is inferior to the most accurate VCNet

[29] , but due to the depthwise separable convolution in Mobilenet V3, it has fewer parameters and computational amounts compared to ordinary convolution. Compared with the lightweight model MobileCount

[30] , when the parameters and computational amounts of the method proposed in this invention are comparable, the MAE and MSE are reduced by 13.5% and 18.9% respectively.

[0111] Table 6 Comparison on the Vi s D r o ne 2019 V e hi c l e dataset

[0112]

[0113] Figure 9(a) to Figure 9(c)Some predicted density maps of the model in the VisDrone2019 Vehicle dataset are given. There is an interfering background in the original image of Figure 9(a), but the vehicle position is accurately displayed in the predicted density map, indicating that the attention module effectively eliminates the interference. The original image of Figure 9(b) has extreme scale changes. The method proposed in this invention accurately segments the image according to the scale range and dynamically and adaptively adjusts the dilation rate parameter in the convolution to generate a high-quality density map. The original image of Figure 9(c) has a significant drop in vehicle distribution. The predicted density map accurately displays scattered vehicles, and the pixel colors are darker, which confirms that the proposed block-level distribution-aware loss function effectively increases the proportion of counting errors in sparse areas in the loss function.

[0114] 3.2 Ablation Experiment

[0115] Next, ablation experiments were conducted on the VisDrone2019 vehicle and CARPK datasets to analyze the impact of different modules on model performance. Table 7 shows the ablation test results. Compared with the Mobilenet V3 baseline method, the proposed method shows significant improvement in counting accuracy, with MAE and MSE decreasing by 33.6% and 38.1%, respectively. On the other hand, after removing the spatial information enhancement attention module alone, the MAE and MSE of the proposed method increased by 15.2% and 18.2%, respectively. After removing the expansion rate parameter adaptation module alone, the MAE and MSE increased by 40.3% and 45.5%, respectively. After removing the block-level target distribution perception loss function alone, the MAE and MSE increased by 6.5% and 7.1%, respectively. Similar results were obtained from the ablation experiment on the CARPK dataset.

[0116] Table 7 Comparison of ablation experiment results

[0117]

[0118] The above ablation experiments demonstrate the effectiveness and rationality of the proposed solution. Because it enables the model to perceive scale and dynamically adjust parameters, the adaptive expansion rate parameter makes the most significant contribution to improving counting accuracy.

[0119] 4 Summary

[0120] This paper proposes a method for counting targets in drone images based on a parameter-adaptive dynamic neural network. By dynamically adjusting model parameters based on input, the method improves aerial image counting accuracy while maintaining a low computational load. The proposed scenario is replicable and universal, unrestricted by the drone's flight altitude or shooting angle. The proposed expansion rate parameter adaptation scheme is scalable and can be further applied to target detection in large-scale, changing scenarios.

[0121] References

[0122] [1] Xiao Jinsheng, Zhang Shuhao, Chen Yunhua, et al. Remote Sensing Image Object Detection with Bidirectional Feature Fusion and Feature Selection [J]. Acta Electronica Sinica, 2022, 50(2): 267-272.

[0123] [2] Jiang X, Zhang L, Xu M, et al. Attention Scaling for Crowd Counting [C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 4706-4715.

[0124] [3] Duan C, Wei Z, Zhang C, et al. Coarse-grained Density Map Guided Object Detection in Aerial Images [C]. Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 2789-2798.

[0125] [4] Liu W, Salzmann M, Fua P. Context-aware Crowd Counting [C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 5099-5108.

[0126] [5] Gao G, Liu Q, Wang Y. Counting from Sky: A Large-scale Data Set for Remote Sensing Object Counting and a Benchmark Method [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5): 3642-3655.

[0127] [6] Najibi M, Singh B, Davis LS. Autofocus: Efficient multi-scale inference[C]. Proceedings of the IEEE / CVF international conference on computer vision. 2019:9745-9755.

[0128] [7] Lin TY, Dollár P, Girshick R, et al. Feature pyramid networks for object detection[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:2117-2125.

[0129] [8] Han Y, Huang G, Song S, et al. Dynamic neural networks: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(11):7436-7456.

[0130] [9] Huang G, Chen D, Li T, et al. Multi-scale dense networks for resource efficient image classification[J]. arXiv preprint arXiv:1703.09844, 2017.

[0131]

[10] Veit A, Belongie S. Convolutional networks with adaptive inference graphs[C]. Proceedings of the European Conference on Computer Vision (ECCV). 2018:3-18.

[0132]

[11] Eigen D, Ranzato M A, Sutskever I. Learning factored representations in a deep mixture of experts[J]. arXiv preprint arXiv:1312.4314, 2013.

[0133]

[12] Chen Y, Dai X, Liu M, et al. Dynamic convolution: Attention over convolution kernels[C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:11030-11039.

[0134]

[13] Denil M, Shakibi B, Dinh L, et al. Predicting parameters in deep learning[J]. Advances in neural information processing systems, 2013, 26.

[0135]

[14] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018:7132-7141.

[0136]

[15] Bai H, Wen S, Gary Chan S H. Crowd counting on images with scale variation and isolated clusters[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision Workshops. 2019:18-27.

[0137]

[16] Cheng Z Q, Li J X, Dai Q, et al. Learning spatial awareness to improve crowd counting[C]. Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6152 - 6161.

[0138]

[17] Howard A, Sandler M, Chu G, et al. Searching for mobilenetv3[C]. Proceedings of the IEEE / CVF international conference on computer vision. 2019: 1314 - 1324.

[0139]

[18] Ramachandran P, Zoph B, Le Q V. Searching for activation functions[J]. arXiv preprint arXiv:1710.05941, 2017.

[0140]

[19] Li Y, Chen Y, Wang N, et al. Scale-aware trident networks for object detection[C]. Proceedings of the IEEE / CVF international conference on computer vision. 2019: 6054 - 6063.

[0141]

[20] Sun X, Jiang Y, Ji Y, et al. Distance measurement system based on binocular stereo vision[C]. IOP Conference Series: Earth and Environmental Science. IOP Publishing, 2019, 252(5): 052051.

[0142]

[21] Chang J R, Chang P C, Chen Y S. Attention-aware feature aggregation for real-time stereo matching on edge devices[C]. Proceedings of the Asian Conference on Computer Vision. 2020.

[0143]

[22] Wang P, Chen P, Yuan Y, et al. Understanding convolution for semantic segmentation[C]. 2018 IEEE winter conference on applications of computer vision (WACV). 2018:1451-1460.

[0144]

[23] Li Y, Zhang X, Chen D. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes[C]. Proceedings of the IEEE conference on computer vision and pattern recognition. 2018:1091-1100.

[0145]

[24] Hsieh M R, Lin Y L, Hsu W H. Drone-based object counting by spatially regularized regional proposal network[C]. Proceedings of the IEEE international conference on computer vision. 2017:4145-4153.

[0146]

[25] Gao G,Liu Q,Hu Z,et al.PSGCNet:A pyramidal scale and globalcontext guided network for dense object counting in remote-sensing images[J].IEEE Transactions on Geoscience and Remote Sensing,2022,60:1-12.

[0147]

[26] Kilic E,Ozturk S.An accurate car counting in aerial images basedon convolutional neural networks[J].Journal of Ambient Intelligence andHumanized Computing,2021:1-10.

[0148]

[27] Zhang Y,Zhou D,Chen S,et al.Single-image crowd counting viamulti-column convolutional neural network[C].Proceedings of the IEEEconference on computer vision and pattern recognition.2016:589-597.

[0149]

[28] Yi J,Shen Z,Chen F,et al.A Lightweight MultiScale Feature FusionNetwork for Remote Sensing Object Counting[J].IEEE Transactions on Geoscienceand Remote Sensing,2023.

[0150]

[29] Zhang J, Qiao J J, Wu X, et al. Vehicle Counting Network with Attention-based Mask Refinement and Spatial-awareness Block Loss[C]. Proceedings of the 29th ACM International Conference on Multimedia. 2021:2889-2898.

[0151]

[30] Wang P, Gao C, Wang Y, et al. MobileCount: An efficient encoder-decoder framework for real-time crowd counting[J]. Neurocomputing, 2020, 407:292-299.

Claims

1. A method for counting targets in UAV images based on a parameter-adaptive dynamic neural network, comprising the following steps: 1) UAV-mounted aerial photography equipment collects ground images; 2) Preprocessing the collected drone images; 3) performing target detection on the processed image using a feature extraction network; 4) counting the detected targets; wherein, in step 2), dynamically dividing the scale and segmenting the image based on the UAV's flight altitude and the shooting angle; in step 3), extracting the target's location information using a feature extraction network enhanced with spatial information on the input feature map; wherein, the expansion rate parameter of the convolution in the receptive field selection module after the feature extraction network is determined based on the segmented image; Design an adaptive strategy for the expansion rate parameter, calculate the scale range and segment the image based on the flight state, and adjust the expansion rate parameter in the receptive field selection module; In step 2), 2.1) Scale discrimination of flight state perception The scale of the target in the image is determined by the distance between the target and the drone camera. The scale discrimination method is to judge the target scale by referring to the drone's flight altitude h and pitch angle β. A rectangular coordinate system is established with the point where the drone is projected on the ground as the origin, the ground as the horizontal axis, and the direction of the intersection of the visual midline and the ground relative to the origin as the positive direction of the horizontal axis. When the drone is hovering at a height h, the coordinates of the ground targets at distances d1 and d2 from it are respectively and When the UAV viewing angle is α and the pitch angle is β, the coordinates of the intersection of the lower and upper sight lines with the ground are (τ1,0) and (τ2,0), respectively. For the target at (x, 0), the drone calculates the scale based on the flight altitude h and the pitch angle β. Assuming that the flight altitude of a drone is h∈(0, d1), the images it collects include three scales: small, medium, and large. The corresponding field of view coverage areas of the three scales are areas III, II, and I respectively. The scale range division rules when the flight state changes are shown in Table 1: Table 1 2.2) Image segmentation based on scale discrimination results The intersection of regions III, II, and I in Table 1 with the ground corresponds to the proportion of small, medium, and large scale areas in the aerial image, which are represented by ω1, ω2, and ω3 respectively. Given the flight altitude h and the pitch angle β, the values of these scale proportions are shown in Table 2: Table 2 Segment the image according to the ratio in Table 2; Selection of the expansion rate parameter in the receptive field selection module after the feature extraction network in step 3) The Switch function is used to determine the convolution expansion rate in the receptive field module based on the image scale obtained in step 2); in the receptive field selection module, three convolutions with adjustable expansion rates are stacked to achieve a dynamic receptive field; the convolution expansion rate is represented as a triple Q = (q1, q2, q3), where the three elements are mutually prime; multiple groups of values for Q are set so that the receptive field covers small, medium, and large scales; when the Switch function recognizes that the input is an image of a specific scale, it guides the convolution to adjust to a expansion rate that matches it.

2. The method for counting targets in drone images based on parameter adaptive dynamic neural network according to claim 1 is characterized by: The input of the feature extraction network in step 3) is the segmented image obtained in step 2); the feature extraction network adopts the improved Mobilenet V3 feature extraction network, and the improvement method is as follows: the attention mechanism in the Mobilenet V3 feature extraction network is improved; in the feature extraction network: 3.1) The input feature map r of dimension C×G×W is compressed horizontally and vertically, with the operation functions represented as S1(·) and S2(·) respectively. The compression operation generates channel statistics in two directions through a global average pooling layer, aggregating the feature map in the spatial dimension. The output of the cth channel at height g is expressed as The output of the cth channel at width w is expressed as After the feature map is compressed, the generated horizontal and vertical features are represented as Z1 and Z2 respectively, that is, Among them, the dimensions of Z1 and Z2 are C×G×1 and C×1×W respectively; W is the width, G is the height, and C is the number of channels; 3.2) Input the intermediate results Z1 and Z2 into the excitation operation functions E1(·) and E2(·) to generate the horizontal weight T1 and vertical weight T2, which are expressed as Among them, F1 and F2 are 1×1 convolution operations, δ is batch normalization, and the activation function σ(·) is expressed as 3.3) r is re-weighted by T1 and T2 to obtain R=U(r,T1,T2)=r×T1×T2 (6) The target location information is aggregated in R as the output of the feature extraction network.

3. The method for counting targets in drone images based on parameter adaptive dynamic neural network according to claim 1 is characterized by: In the step 4), a density estimation-based method is used to perform target counting.

4. The method for counting targets in drone images based on parameter adaptive dynamic neural network according to claim 1 is characterized by: Steps 2) to 4) constitute the drone image target counting model. During model training, a block-level target distribution perception loss function is used, and the k-root method is used to improve the Euclidean loss function to balance the loss of areas with different target distributions in the aerial image. Specifically, the following is the contribution of the loss to the total loss: First, the predicted density map and the true value density map are evenly divided into m blocks, and the corresponding blocks are marked as A. i＝1,2,…,m , establish block-level constraints on the density map and roughly divide the sparse and dense areas; Then, find A i＝1,2,…,m The Euclidean loss and counting loss are combined to the power of 1 / k, so that blocks with dense targets are suppressed and blocks with sparse targets are valued; Finally, for A i＝1,2,…,m The reweighted losses are summed to obtain the final loss of the density map, which is used to train the model; Let F(X i,j ; E) and Represent the predicted density map and the true density map of the jth block of the i-th image, respectively. and Represent the predicted count and true count of the jth block of the i-th image respectively; The loss function is expressed as Where N represents the total number of samples, E is the training parameter, and λ is the weight of the counting loss; the optimal value of k is determined through experiments.

Images