A method and system for crowd density estimation

Abstract

The application relates to the technical field of computer vision, and discloses a crowd density estimation method and system. The method adopts an improved adjacent space-time perception network to analyze crowd information of an image including a crowd; wherein the crowd information analysis includes crowd counting and / or crowd positioning. The application solves the problem of low accuracy of crowd density estimation in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a crowd density estimation method and system. BACKGROUND

[0002] With the increasing of urban population, the flow of people in public areas is increasing, and the monitoring and management of the crowd are facing great challenges and pressures. The method of crowd density estimation and crowd positioning based on the aerial view of the unmanned aerial vehicle can help the relevant departments better detect the crowd density of various places, so as to effectively avoid the occurrence of stampede and trampling events.

[0003] In the method of crowd density estimation and crowd positioning based on images:

[0004] The LCOI method adopts the MSEA distance function, and performs density estimation by minimizing the regularized risk quadratic loss function. However, this method does not use deep learning to calculate the crowd count. The Multi-column CNN method uses multiple neural networks to detect the features of the people in the image, and each neural network detects the features of the head of different sizes. This method is greatly limited by the detection environment in actual detection.

[0005] In the convolutional layer of the neural network, the image structure between different positions and different features has different statistical characteristics, and using the same convolution kernel may not be able to fully capture these features. Therefore, the Switch-CNN uses a convolution kernel group, each convolution kernel group contains multiple convolution kernels, and each convolution kernel group has its own weight parameters. In the process of model training, an attention mechanism is used to learn to select which convolution kernel to use, so as to evaluate the crowd density in different dense scenes. However, this method is not strong in handling the relevance between different regions of the same image.

[0006] The SANet introduces a scale aggregation mechanism to improve the traditional crowd counting method, aiming to solve the trade-off between counting accuracy and computational efficiency at different scales. However, this method uses multiple networks to process features of different scales, and different scene datasets also have different scales, so this method cannot flexibly select a suitable scale for model training and inference.

[0007] In the HGNN, the graph neural network is used to process the spatial relationship in the crowd dense area, and the convolutional neural network is used to capture the visual features of the crowd. This method needs to be reasonably designed and adjusted in actual application in terms of how to determine the boundary and weight between nodes, and how to handle partial occlusion and other problems.

[0008] ConvLSTM-nt, a spatio-temporal modeling method for crowd counting in videos. This method aims to improve the accuracy of the crowd counting task by considering the temporal information and spatial relationships in the video. The main idea of this method is to treat each frame in the video sequence as a time step, and use spatio-temporal modeling techniques to capture the temporal and spatial features of the crowd. However, in the task of crowd counting, different frames in the video may have different numbers of crowds. This can lead to the problem of unbalanced data samples.

[0009] In the adjacent spatio-temporal perception network (STNNet), although multi-scale detection and multi-task learning are achieved, the feature extraction capability of the CNN-based neural network for the input image still needs to be improved. It directly uses MSE to calculate the density loss function, ignoring the problem of sample distribution.

[0010] In summary, in the existing methods, the following problems exist: ① It is not possible to correlate the information of different local regions in the same image, which is crucial for neural network learning to learn more target information and background information; ② The neural network is limited and cannot achieve multi-scale target detection through a single neural network; ③ The unbalanced distribution of data samples leads to poor generalization of the network; ④ The performance of the CNN-based neural network is limited. In addition, most of the images taken by drones for aerial photography are 1K or 2K resolution images, while Backbones such as VGG, ResNet, etc. built by CNN require input images to be 224*224 or 512*512. If the high-resolution image is directly scaled down, a large amount of information will be lost. SUMMARY

[0011] To overcome the shortcomings of the prior art, the present application provides a crowd density estimation method and system, which solves the problem of low accuracy in the prior art.

[0012] The technical scheme adopted by the present application to solve the above problems is:

[0013] A crowd density estimation method uses an improved adjacent spatio-temporal perception network to analyze crowd information in images containing crowds; wherein the crowd information analysis includes crowd counting and / or crowd positioning.

[0014] As a preferred technical scheme, the following steps are included:

[0015] S1, network model construction: constructing an improved adjacent spatio-temporal perception network; wherein the improved adjacent spatio-temporal perception network model includes a backbone network, a neck, a density map output head, and a positioning map output head;

[0016] S2, loss function setting: set the density loss function of the improved adjacent spatio-temporal perception network model for crowd density detection, and / or, set the positioning loss function of the improved adjacent spatio-temporal perception network model for crowd positioning;

[0017] S3, crowd information analysis: train the improved adjacent spatio-temporal perception network model, analyze the crowd density detection by using the density evaluation index, and / or, analyze the crowd positioning by using the positioning evaluation index.

[0018] As a preferred technical solution, in step S1, the SwinTransformer model is used as the backbone network of the improved adjacent spatio-temporal perception network model.

[0019] As a preferred technical solution, in step S1, the bottleneck composite multi-scale prior module is used to replace the convolution layer in the original adjacent spatio-temporal perception network model FPN structure, and the replaced FPN structure is used as the neck of the improved adjacent spatio-temporal perception network model.

[0020] As a preferred technical solution, in step S2, the density loss function is:

[0021]

[0022]

[0023] wherein, represents the optimal transport loss, represents the unnormalized real density map function value, represents the unnormalized predicted density map function value, |·|1 represents the summation of one-dimensional feature maps, represents the wasserstein distance function, α * represents the real value distribution matrix, β * represents the predicted value distribution matrix, <·> represents the cost matrix function, <·> is used to measure the difference between the two, P and Q represent two different distributions, represents the wasserstein distance between distribution P and distribution Q, π represents the transport plan or matrix, π specifies the mass that should be moved from each element of P to each element of Q, π i,j represents the element of the transport plan, represents the mass that should be moved from the i-th element of P to the j-th element of Q, c i，j represents the mass cost from the i-th element of P to the j-th element of Q, min π represents the minimization operation on the transport plan.

[0024] As a preferred technical solution, in step S2, the density loss function formula after introducing the total variable loss is:

[0025]

[0026]

[0027] wherein, represents the total variable loss, f TV (·) represents the L1 norm loss function, L1Loss represents the L1 norm loss, i represents the index value of the current variable, n represents the total number of variables, y i represents the i-th real value, x i represents the i-th input value, f(x i ) represents the i-th predicted value.

[0028] As a preferred technical solution, in step S2, the positioning loss function is:

[0029]

[0030] wherein, represents the positioning loss function, represents the predicted point position label, represents the real point position label, represents the predicted point offset, represents the real point offset, l represents the layer number of the feature map, L represents the total number of layers of the feature map, i represents the index of the horizontal coordinate, V l represents the width of the feature map, j represents the index of the vertical coordinate, H l represents the height of the feature map, represents the classification loss of the label value of the predicted point and the real point at the (i, j) position in the l-th layer, S k (i, j, l) represents the predicted probability value of the (i, j) position in the l-th layer, represents the regression loss of the offset of the predicted point and the real point at the (i, j) position in the l-th layer, represents the label value of the predicted point at the (i, j) position in the first layer of the classification subgraph, represents the label value of the real point at the (i, j) position in the l-th layer of the classification subgraph, represents the offset of the predicted point at the (i, j) position in the l-th layer of the regression subgraph, represents the offset of the real point at the (i, j) position in the l-th layer of the regression subgraph.

[0031] As a preferred technical solution, in step S3, the density evaluation index is calculated by using the mean absolute error, and the formula is:

[0032]

[0033] Wherein, MAE represents the average error between the predicted value and the true value, i represents the number of videos, j represents the number of frames, M represents the total number of videos, N i represents the number of frames in the ith video, Z i，j represents the number of real people in the jth frame image of the ith video, represents the number of predicted people in the jth frame image of the ith video.

[0034] As a preferred technical solution, in step S3, the positioning evaluation index adopts the following method: using a greedy algorithm to calculate the positional relationship between each predicted point and each real point under different thresholds.

[0035] A crowd density estimation system for implementing the crowd density estimation method, comprising the following modules connected in turn:

[0036] The network model construction module is used to construct an improved adjacent space-time perception network, wherein the improved adjacent space-time perception network model comprises a backbone network, a neck, a density map output head and a positioning map output head.

[0037] The loss function setting module is used to set the density loss function of the improved adjacent space-time perception network model for crowd density detection, and / or set the positioning loss function of the improved adjacent space-time perception network model for crowd positioning.

[0038] The crowd information analysis module is used to train the improved adjacent space-time perception network model, analyze the crowd density detection using the density evaluation index, and / or analyze the crowd positioning using the positioning evaluation index.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] (1) Compared with the adjacent space-time perception network (STNNet) using VGG as the backbone network, the present application uses SwinTransformer as the backbone network, which can more efficiently extract image features and correlate the information of each local area of the image;

[0041] (2) In the feature fusion process of the spatial pyramid, the present application adds a BottleNeckCSP multi-scale prior module, which can realize cross-stage information interaction, reduce the dimension and improve the learning ability of the model to the features of the image;

[0042] (3) The optimal transport loss and the total variable loss are used for density loss calculation, compared with the density loss calculation of the STNNet model using only the MSE, the application can better measure the distribution relationship between the real density map and the predicted density map, and the total variable loss can better help the model to back propagation. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Structure diagram of the improved STNNet model;

[0044] Figure 2 Partially enlarged view of the Figure 1

[0045] Figure 3 Partially enlarged view of the Figure 1

[0046] Figure 4 Partially enlarged view of the Figure 1

[0047] Figure 5 Density heat map of the model prediction result under the DroneCrowd dataset scene;

[0048] Figure 6 Positioning heat map of the model prediction result under the DroneCrowd dataset scene;

[0049] Figure 7 Positioning result map of the model prediction under the DroneCrowd dataset scene. DETAILED DESCRIPTION

[0050] The application will be further described in detail below in combination with the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0051] Embodiment 1

[0052] As shown in the Figures 1 to 7 application provides an algorithm for estimating the crowd density under the aerial view of the unmanned aerial vehicle, and simultaneously provides the corresponding data processing method and model training method, and the scheme is as follows:

[0053] 1. Data loading method based on high-pixel image of unmanned aerial vehicle aerial photography: solves the problem that the CNN model cannot fully extract and learn the features of high-pixel images.

[0054] ​​​2. The improved network model based on the adjacent spatio-temporal perception network (STNNet) can realize more efficient feature extraction capability, cross-stage information fusion capability and multi-scale target reasoning capability, making it perform better in detecting high-pixel unmanned aerial vehicle aerial image, as follows:

[0055] (1) The VGG backbone network of the adjacent spatio-temporal perception network (STNNet) is modified to a backbone network based on SwinTransformer, which solves the problems of ① limited CNN feature extraction capability; ② different region information interaction; ③ high-pixel image causing excessive CNN parameter quantity.

[0056] (2) Based on the neck of the adjacent spatio-temporal perception network (STNNet) model, the BottleNeckCSP module is introduced, which helps the network model to realize feature dimension reduction and enhancement, and preliminary cross-stage information transmission of feature maps.

[0057] (3) The image pyramid part of the adjacent spatio-temporal perception network (STNNet) is improved to help the network model better perform multi-scale information fusion and realize multi-scale target detection task.

[0058] 3. The model optimization method based on the adjacent spatio-temporal perception network (STNNet) introduces Optimal Transport Loss and Total Variable Loss to promote network learning and parameter iteration, solving the sample distribution problem ignored by the adjacent spatio-temporal perception network method.

[0059] The application provides a multi-task network model for multi-scale residual processing and multi-scale feature fusion based on SwinTransformer as a backbone network, which is designed for crowd density detection and crowd positioning in unmanned aerial vehicle aerial scene. The network model has multiple functions, including realizing spatio-temporal information interaction between images, multi-scale crowd detection, crowd density reasoning and crowd positioning information reasoning. Through this multi-task network model, the crowd density detection and crowd positioning tasks can be efficiently processed simultaneously in unmanned aerial vehicle aerial application, providing strong support for urban management and public safety field.

[0060] The present application is based on the improvement of the adjacent spatio-temporal perception network (STNNet) model, and the evaluation is carried out using the rich DroneCrowd dataset of unmanned aerial vehicle aerial images. The main goal is to enhance the feature extraction capability of the model, realize the fusion of image features, and strengthen the information interaction of different receptive fields, while improving the multi-scale target detection and crowd counting and positioning capability. Through these improvements, we hope to more accurately infer the position and quantity information of different scale targets when dealing with crowd targets in unmanned aerial vehicle aerial images. Therefore, the optimization of the present application aims to make the neural network model perform better in complex scenes, especially in the multi-scale crowd detection and positioning task involving unmanned aerial vehicle aerial images.

[0061] Figure 1 The structure of the improved adjacent spatio-temporal perception network (STNNet) model is shown, mainly including four parts: the backbone network composed of SwinTransformer, the network neck part for feature fusion, the density detection head, and the positioning detection head.

[0062] Figure 1 In the middle, Patch Partition represents the feature map cutting module, which divides the input picture of (H, W, 3) into (4, 4) small blocks, and the size of the divided picture is (H / 4, W / 4);

[0063] Linear Embedding represents the linear embedding module, which divides the input feature map into multiple blocks, and then performs linear transformation on each block to generate a high-dimensional feature representation; this way helps the model to capture image features of different scales and levels, thereby improving the performance of the model;

[0064] Patch Merging represents the image block information integration module, and realizes the dimension reduction of the feature map;

[0065] Density_map represents the density map output by the model, which is used for crowd density detection;

[0066] Classification_map represents the classification subgraph, which is used to realize the classification of people and background;

[0067] Regression_map represents the regression subgraph, which records the offset of the predicted position of the person.

[0068] Figure 5 The density heat map of the model prediction result in the DroneCrowd dataset scene is shown, and the crowd area predicted by the network model is the focused area.

[0069] Figure 6The positioning heat map of the model prediction result in the DroneCrowd dataset scenario is shown, and the crowd area predicted by the network model is the focus area.

[0070] Figure 7 The positioning result of the model prediction in the DroneCrowd dataset scenario is shown. The solid origin represents the real crowd point, and the hollow circle represents the position information of each person in the image predicted by the network.

[0071] I. Construction of network model:

[0072] The input size of the network model is [B, 3, 512, 512], where B represents the batch-size (the number of samples transmitted into the network model per batch), and the network model is composed of four parts: backbone network, neck, density map output head, and positioning map output head.

[0073] 1. Constructing the backbone network: The pixel of the unmanned aerial vehicle aerial image is large, the information is rich, and the scale of the detected target is uneven, so an excellent network structure is needed to extract the basic features. In the structure of the adjacent spatiotemporal perception network (STNNet), the backbone network uses the stage 2, stage 3, and stage 4 of the VGG-16 model as the basic feature maps of three scales, but the traditional convolutional neural network model such as VGG has fewer parameters than the transformer model, and the ability of image feature extraction is lacking. The present application will replace the conventional convolutional neural network with a transformer structure; on the other hand, when processing high-pixel input images, the computational load of the transformer structure will increase in a square level. In order to solve these two problems at the same time, the present application uses the SwinTransformer model to extract features from the input image, which has the following advantages: ① SwinTransformer is an advanced model based on transformer, which can better capture global context information and extract more accurate features when processing images compared to the backbone network of convolutional neural network; ② SwinTransformer uses a self-attention module (self-attention) to effectively establish long-range dependencies, which is very important for processing large-scale images and capturing long-distance relationships in images. In contrast, traditional convolutional neural networks are limited by the receptive field, resulting in a decline in performance when processing large images; ③ SwinTransformer uses a window attention mechanism, which can maintain computational and memory efficiency when processing images; ④ SwinTransformer uses a sliding window mechanism, which can effectively interact local information and global information, improving the learning ability of the model.

[0074] In this process, we mainly rely on the multi-head self-attention structure and the shifted-window mechanism in it. Among them, the self-attention module helps the model to capture the relevance between pixels or regions, and automatically learns the importance and correlation degree between different regions in the image, thereby improving the understanding ability of the image; the shifted-window helps the model to learn the information across the window. The calculation formula of self-attention is as follows:

[0075]

[0076]

[0077] In the formula, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. Q, K, and V are matrices that can be iteratively learned by performing dot product on the feature map output by the previous layer network and the linear transformation matrix WQ, WK, and WV; Q, K, and V have the same size and dimension; the vector inner product is performed on Q and K, and in order to prevent the inner product from being too large, it is divided by the square root of d, where d represents the column number of the Q and K matrices; softmax represents cross-entropy.

[0078] Since the SwinTransformer is a hierarchical network structure, in the present application, the model inference results of Stage1, Stange2 and Stange3 (Stage 1, Stage 2, and Stage 3) of the SwinTransformer are output. When the input of the network model is an image of [B, 3, 512, 512], three feature maps of different scales are obtained, and the shapes are [B, 192, 128, 128], [B, 384, 64, 64], and [B, 768, 32, 32], respectively. Then, the three feature maps are subjected to deconvolution and upsampling operations to change their shapes to [B, 192, 256, 256], [B, 384, 128, 128], and [B, 768, 64, 64], respectively, and finally three basic feature maps f1, f2, and f3 are obtained.

[0079] 2. Constructing a model neck: the application introduces a bottle neck composite multi-scale prior module (BottleNeckCSP) in the neck part of a space-time perception network (STNNet) model, realizes the improvement of the information interaction ability of different scale feature maps, and thus enhances the detection ability of the model for different scale targets. In order to verify the effectiveness of the module, the DroneCrowd dataset is adopted, which contains unmanned aerial vehicle aerial image, and the scale of the crowd target is uneven. By introducing the bottle neck composite multi-scale prior module (BottleNeckCSP), the model can process multi-scale targets with richer image features, including the detection of large targets and small targets, and effectively improve the detection performance. Therefore, by integrating the unmanned aerial vehicle aerial image and the DroneCrowd dataset with rich image features, and combining the bottle neck composite multi-scale prior module (BottleNeckCSP), the application successfully enhances the ability of the network model in different scale target detection.

[0080] Firstly, the extracted f1, f2, f3 are input into the bottle neck composite multi-scale prior module (BottleNeckCSP), and a large number of residual (Residual) structures in it help to realize the dimension reduction and enhancement of the features and the preliminary cross-stage information transmission of the feature maps, and new feature maps f1, f2, f3 are obtained. When the feature map input into the residual structure is x, the structure thereof can be represented by the following formula:

[0081] x1=x

[0082] x2=f cnn (x)

[0083] Output=f activation (x1+x2)

[0084] In the formula, x represents an input feature map, x1 represents a feature map same as x, f cnn represents a module composed of multiple convolution layers, x2 represents an output feature map after inputting x into the convolution module, f activation represents an activation function (such as ReLu, Sigmoid, LekyReLu, etc.), and Output represents a Residual structure output feature map.

[0085] In the BottleNeckCSP, the size of the input feature map is the same as the size of the output feature map and the number of channels. Then f1, f2, f3 are input into the image space pyramid of the FPN structure for feature fusion. The smallest scale f3 feature map is transformed into the same scale as f2 by convolution layer and up-sampling layer, and is fused with f2 to obtain g2. The obtained g2 is again learned by convolution layer, ReLu layer and BottleNeckCSP to obtain new g2. The new g2 is converted into the same scale as f1 by up-sampling layer and fused with f1 in the channel to finally obtain the feature map g1. g1 is again learned by convolution layer, ReLu layer and BottleNeckCSP to obtain new g1. After the above series of operations, the Neck layer outputs three scale feature maps g1, g2, f3, and their shapes are [B, 192, 256, 256], [B, 384, 128, 128], [B, 768, 64, 64] respectively.

[0086] 3. Density map output head: This module processes the feature maps g1, g2, f3 output by the neck structure, and outputs three scale single-channel density maps, whose shapes are [B, 1, 256, 256], [B, 1, 128, 128], [B, 1, 64, 64] respectively.

[0087] 4. Positioning map output head: The positioning output head contains two branches: classification branch and regression branch. The same feature map processing is performed in both branches: first, the feature maps g1, g2, f3 output by the neck are respectively processed by a convolution layer with an output channel number of 2 to obtain three feature maps with shapes [B, 2, 256, 256], [B, 2, 128, 128], [B, 2, 64, 64], and then the scale is converted to [B, 2, 256, 256] after up-sampling processing. Finally, channel fusion and convolution processing are performed to output classification feature maps and regression feature maps with a scale of [B, 2, 256, 256], which are denoted as classification-map and regression-map respectively. In the 0th dimension channel of the classification feature map, the class features of the detected target are represented, and in the 1st dimension channel, the class features of the background are represented. In the 0th dimension channel of the regression feature map, the position offset of the x coordinate point is represented, and in the 1st dimension channel, the position offset of the y coordinate point is represented.

[0088] II. Loss function:

[0089] (1) Density loss function:

[0090] In the adjacent spatio-temporal aware network (STNNet), different input images will produce different output density maps, but may draw the same crowd counting result. The existing method is to sum the density map output by the network to obtain the predicted crowd number, and then calculate the mean square error (MSE) loss with the true result. However, this method ignores the position distribution information of the crowd in each image. In order to solve this problem, the density estimation loss function in the DM-Count method is adopted in the present invention, which is composed of two parts: optimal transport loss and total variable loss. By introducing these loss functions, we can better utilize the position information in the density map, thereby further optimizing the prediction result of the crowd number.

[0091] Among them, the optimal transport loss is used to measure the loss between two probability distributions, which is based on the optimal transport theory and aims to measure the minimum cost required to transform one distribution into another. In the present invention, the two distributions are the density map predicted by the network and the density map generated by the real coordinate points, respectively. The optimal transport loss is adopted to make the network produce effective gradient in the iteration process and can be used to match the difference between the predicted density map and the real density map. Its formula is as follows:

[0092]

[0093]

[0094] In the formula, and are the unnormalized real density map and predicted density map function values, which are divided by the total amount of each to convert them into probability density functions.

[0095] Wasserstein distance represents the Wasserstein distance, which is a loss function used to measure the difference between two probability distributions.

[0096] Since only the optimal transport loss is used, only the dense scene achieves significant results, and the sparse crowd scene performs poorly, so the total variable loss is introduced to increase the stability of the model training process, and its formula is as follows:

[0097]

[0098]

[0099] In the formula, and are the unnormalized real density map and predicted density map function values; f TVdenotes the absolute error loss (L1 Loss) for measuring the absolute error between the real density map and the predicted density map.

[0100] (2) Positioning loss function:

[0101] The positioning loss function contains two parts: classification loss and regression loss. In the present application, each predicted point is matched with the real labeled point, and if the predicted point is within the adjacent range of the real point, it is marked as a positive sample and marked as 1 in the predicted mask map, and the rest of the points are predicted as background and marked as 0. The positioning loss formula is as follows:

[0102]

[0103] wherein and are the label values of the predicted point and the real point at the (i, j) position in the lth layer; and are the offsets of the predicted point and the real point at the (i, j) position in the lth layer. Finally, the log-likelihood loss is calculated to calculate the mean square error (MSE) and, only for the positive predicted points.

[0104] III. Evaluation index:

[0105] (1) Density evaluation index:

[0106] The density estimation evaluation index is calculated using the mean absolute error (MAE). The formula is as follows:

[0107]

[0108] In the formula, y i represents the predicted value, and x i represents the true value. MAE represents the average error between the predicted value and the true value. In the prediction task of the density map, the single-channel density map output by the neural network is summed for each pixel point value, which is the current predicted number of people. The present application will predict and evaluate a series of video frames, and the MAE calculation formula in the present application is as follows:

[0109]

[0110] In the formula, M represents the total number of video frames, N i represents the number of frames in the ith video, Z i，j and represent the real number of people and the predicted number of people in the jth frame of the ith video, respectively.

[0111] (2) Positioning evaluation index:

[0112] The final goal of crowd positioning is to predict the positioning information of all people in each image. The classification map predicted by the network model is subjected to non-maximum suppression (NMS) processing to obtain the coordinates of the predicted points and the confidence of the predicted points. The NMS here is slightly different from the NMS used in the target detection algorithm. The NMS in the target detection algorithm will filter each detection box to ensure that each target corresponds to one predicted box output. The NMS here is a filtering operation based on pixel points, and finally outputs a positioning coordinate for each predicted point. Since the output classification map is a two-channel feature map, the 0th channel represents the class information of the people, and the 0th channel of the classification map will be subjected to NMS filtering. The specific method is to find the position information of the peak point of each 8*8 region and the value of the peak point in the 0th channel of the classification map, which represents the confidence.

[0113] Secondly, the position relationship between each predicted point and each real point under different threshold values is calculated using a greedy algorithm. The obtained predicted points and real points are associated with each other, and the evaluation index is denoted as L-AP, and the average evaluation result is denoted as L-mAP. When calculating L-AP, three representative threshold values [10, 15, 20] are selected, and the final results are denoted as L-AP@10, L-AP@15, and L-AP@20. Finally, L-mAP is calculated, which is represented as the average value of L-AP values calculated by 25 threshold values (1, 2, 3, … 25 pixels).

[0114] Four, performance comparison:

[0115] The present application uses the DroneCrowd dataset for model training and testing, which contains 24,000 drone aerial images for training and 9,600 images for testing. Through this dataset, we ensure that the network model provided by the present application has sufficient training and evaluation data when processing drone aerial images, thereby improving the performance of the model in this specific field. It is worth mentioning that the STNNet is the first network model based on the DroneCrowd dataset for crowd density estimation and positioning. Therefore, when verifying all the evaluation indexes of the present application, we compare the evaluation results of the STNNet on the DroneCrowd dataset as the benchmark.

[0116] (1) Comparison of mean absolute error (MAE) results:

[0117] When the network model only contains the density detection head, the density evaluation result of all scenes is improved by 3.23 compared with the adjacent spatiotemporal perception network (STNNet) based on the MAE index; the large target scene is improved by 7.73, the small target scene is reduced by 0.02, the overcast scene is improved by 5.11, the sunny scene is improved by 6.63, the night scene is reduced by 10.31, the crowded scene is improved by 8.21, and the sparse scene is improved by 0.07.

[0118] When the network model contains the density detection head and the positioning detection head at the same time, the density evaluation index of all scenes is improved by 5.62 compared with the adjacent spatiotemporal perception network (STNNet); the large target scene is improved by 6.05, the small target scene is improved by 5.29, the overcast scene is improved by 6.92, the sunny scene is improved by 5.32, the night scene is improved by 2.62, the crowded scene is improved by 7.69, and the sparse scene is improved by 4.51.

[0119] (2) Comparison of L-AP results:

[0120] The positioning evaluation index is compared with the adjacent spatiotemporal perception network (STNNet), when the network model only contains the density detection head, L-mAP is improved by 1.8, L-AP@10 is improved by 3.37, L-AP@15 is reduced by 0.95, and L-AP@20 is improved by 0.64; when the network model contains the density detection head and the positioning detection head at the same time, L-mAP is improved by 9.62, L-AP@10 is improved by 11.23, L-AP@15 is improved by 12.94, and L-AP@20 is improved by 13.09.

[0121] The comparison results of the network model and the STNNet model in the DroneCrowd dataset reflect the advancement and feasibility of the present application.

[0122] The analysis reason is that the present application has the following advantages:

[0123] (1) Compared with the adjacent spatiotemporal perception network (STNNet) using VGG as the backbone network, the present application uses SwinTransformer as the backbone network, which can more efficiently extract image features and correlate the information of each local area of the image;

[0124] (2) In the feature fusion process of the spatial pyramid, the present application adds a BottleNeckCSP multi-scale prior module, which can realize cross-stage information interaction, reduce the dimension, and improve the learning ability of the model to the image features;

[0125] (3) The application adopts OT Loss and TV Loss for density loss calculation, compared with the density loss calculation of the adjacent spatio-temporal perception network (STNNet) model adopting only MSE, the application can better measure the distribution relationship between the real density map and the predicted density map, and the TV Loss can better help the model to perform back propagation.

[0126] Embodiment 2

[0127] As Figures 1 to 7 shown, as a further optimization of embodiment 1, on the basis of embodiment 1, the present embodiment further comprises the following technical features:

[0128] The application relates to the technical field of computer vision, and provides an innovative method for solving the problems of crowd density estimation and positioning in unmanned aerial vehicle (UAV) aerial images. Traditional methods have limitations in complex scenes and changing light conditions, and a more efficient and accurate solution is needed. The application aims to improve the adjacent spatio-temporal perception network (STNNet) model, and the improved network model comprises a backbone network, a neck, a density detection head and a positioning detection head. The key innovations of the application include the following aspects:

[0129] (1) The SwinTransformer is adopted as the backbone network, instead of the original VGG backbone network, to solve the limitations of the convolutional neural network in the receptive field and feature extraction.

[0130] (2) The BottleNeckCSP is introduced to replace the ordinary convolution operation in the traditional network, helping the network model to realize feature dimension reduction and enhancement, and preliminary cross-stage information transmission of the feature map.

[0131] (3) The optimal transport loss (OT Loss) and the total variation loss (TV Loss) are adopted as the loss function for density estimation, solving the problem of the traditional mean square error loss (MSE Loss) in neglecting the sample distribution, and improving the stability of the network model in processing sparse crowd scenes.

[0132] Through the improved network model, experiments on the DroneCrowd dataset prove that the method achieves better results and higher accuracy.

[0133] In summary, the application provides a multi-task network model based on SwinTransformer as a backbone network for multi-scale residual processing and multi-scale feature fusion, which is designed for crowd density detection and crowd positioning in the UAV aerial scene. The network model has multiple functions, including realizing the interaction of space-time information between images, multi-scale crowd detection, crowd density inference, and crowd positioning information inference. Through this multi-task network model, the crowd density detection and crowd positioning tasks can be efficiently processed simultaneously in UAV aerial applications, providing strong support for urban management and public safety.

[0134] I. Data loading method:

[0135] 1. Annotation data preprocessing:

[0136] The application trains, validates and tests the model based on the DroneCrowd dataset. In the DroneCrowd dataset, each image has 1920*1080 pixels, and the crowd position is represented by annotating the center point coordinates of each head. Based on the annotated human coordinate information, a real point regression map and a density map need to be created.

[0137] ① Real regression map creation:

[0138] The application uses the real point coordinates (x, y) of each image as the basis and creates a series of offsets for each coordinate point. For each point coordinate (x, y), the application generates 12 relative offset points, which are:

[0139] (x-5, y), (x-3, y), (x-1, y), (x+1, y), (x+3, y), (x+5, y), (x, y-5), (x, y-3), (x, y-1), (x, y+1), (x, y+3), (x, y+5), and other coordinate points are set to (0, 0). Based on the original image size, the values and coordinates of these points are scaled by 1 / 2, 1 / 4, and 1 / 8 to generate single-channel regression maps of the original image size, which are (960, 540), (480, 270), and (240, 135) respectively.

[0140] ② Real density map creation:

[0141] For the creation of the real density map, the application scales the real coordinate points in each image by 1 / 2, 1 / 4, and 1 / 8 of the original coordinate size, and creates original Figure 1 / 2, 1 / 4, 1 / 8 size full 0 single channel map, the shape is (960, 540), (480, 270), (240, 135) respectively. According to the coordinates of the corresponding scaled size and the full 0 single channel map, the value at the coordinate point is modified to 1, and the value at other positions is 0.

[0142] Finally, the generated regression map and density map of the three scales are respectively scaled to the shape of (512, 512), (256, 256), (128, 128).

[0143] 2、Data augmentation:

[0144] During the model training process, the present application adopts two data augmentation methods to enrich the original data set and improve the robustness of the model. Each time the model is iterated, one of the two methods is randomly selected as the data augmentation method for this iteration. The following is a detailed description of the two methods:

[0145] ① Random cropping and scaling: in this data augmentation method, when the model iteration starts, the present application will randomly crop 1 / 4 of the original image size, i.e. a region of (960, 540), and scale it to the shape of (512, 512) as the input of the neural network model. At the same time, in order to maintain the consistency of the data, the present application will crop the regression map and density map according to the same proportion and position to ensure the correspondence between the input data and the original image region.

[0146] ② Split and random selection: in this data augmentation method, the present application divides the original image and the corresponding regression map and density map into four parts with an area of (960, 540) each. In each model iteration, the present application will randomly select one of the four parts and scale it to the size of (512, 512) as the input of the network model. In this way, the model can learn the features and details of different image regions, enhancing the model's ability to adapt to different scenes and crowd density distribution.

[0147] By adopting these two data augmentation methods, the present application can provide more diverse and rich training data for the model. Each time the model is iterated, one of the two methods is randomly selected, so that the model can be trained under different data transformations, improving the model's generalization ability and robustness to various scenes. In this way, the model can better cope with images of various scales, positions and crowd densities, improving the model's performance in practical applications.

[0148] II. Model training:

[0149] 1. Obtain the DroneCrowd data set and obtain the corresponding label for each image, and create a real density map and a real regression map corresponding to each image.

[0150] 2. The dataset and its labels are processed and loaded according to the data loading method provided in this invention. During model training, 24,000 training images from the DroneCrowd dataset are selected and fed into the network for training.

[0151] 3. Based on the output of the model, the density loss and localization loss proposed in this invention are used to calculate the loss value, and backpropagation and gradient descent of the parameters are performed to iterate through all the parameters of the model.

[0152] 4. The model is tested on the test set during each iteration. Training is stopped when overfitting is observed on the test set, and the model that performs best on the test set is selected as the final model result.

[0153] In addition, during model training, the stochastic gradient descent algorithm with momentum (Adam) is used as the model optimization strategy, with a learning rate of 1e and a weight decay of 1e. At the same time, cosine annealing is used to adjust the learning rate during model training, with the cosine annealing restart cycle set to 50. The model training cycle is 60. The model training uses an RTX Guadro 6000 graphics card, with a batch size of 4 per graphics card.

[0154] III. Result Testing:

[0155] Based on the DroneCrowd dataset, each input image to the network model is a quarter of the original image. Therefore, in testing, the original image is cropped into four 2x2 parts, and each part is sequentially input into the model for prediction. The prediction result for each image is the sum of the results of the four parts.

[0156] As described above, the present invention can be implemented well.

[0157] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0158] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method of crowd density estimation, characterized by, The improved adjacent space-time perception network is used for crowd information analysis of an image including a crowd; wherein the crowd information analysis includes crowd density detection and / or crowd positioning, and includes the following steps: S1, network model construction: constructing an improved adjacent space-time perception network; wherein the improved adjacent space-time perception network model includes a backbone network, a neck, a density map output head, and a positioning map output head; S2, loss function setting: setting a density loss function of the improved adjacent space-time perception network model for crowd density detection, and / or setting a positioning loss function of the improved adjacent space-time perception network model for crowd positioning; In step S2, the positioning loss function is: wherein, represents a localization loss function, represents a predicted point position label, represents a real point position label, represents a predicted point offset, represents a real point offset, l represents a layer number of a feature map, L represents a total number of layers of the feature map, i represents an index of a horizontal coordinate, W l represents a width of the feature map, j represents an index of a vertical coordinate, H l represents a height of the feature map, represents a classification loss of a label value of a predicted point and a real point at a (i, j) position in an l-th layer, S k (i, j, l) represents a predicted probability value of the (i, j) position in the l-th layer, represents a regression loss of an offset of the predicted point and the real point at the (i, j) position in the l-th layer, represents a label value of the predicted point at the (i, j) position in the l-th layer of a classification subgraph, represents a label value of the real point at the (i, j) position in the l-th layer of the classification subgraph, represents an offset of the predicted point at the (i, j) position in the l-th layer of a regression subgraph, represents an offset of the real point at the (i, j) position in the l-th layer of the regression subgraph; S3, crowd information analysis: training the improved adjacent space-time perception network model, analyzing crowd density detection using a density evaluation index, and / or analyzing crowd positioning using a positioning evaluation index.

2. The crowd density estimation method of claim 1, wherein, In step S1, a SwinTransformer model is used as the backbone network of the improved adjacent space-time perception network model.

3. The crowd density estimation method of claim 2, wherein, In step S1, a bottle-neck composite multi-scale prior module is used to replace the convolution layer in the original FPN structure of the adjacent space-time perception network model, and the replaced FPN structure is used as the neck of the improved adjacent space-time perception network model.

4. The crowd density estimation method of claim 1, wherein, In step S2, the density loss function is: W(P,Q) = min π ∑ i,j π i,j ·c i,j ; wherein, represents the optimal transport loss, z represents the unnormalized real density map function value, represents the unnormalized predicted density map function value, | · |1represents the summation over the one-dimensional feature map, represents the Wosternan distance function, a* represents the true value distribution matrix, b* represents the predicted value distribution matrix, <·> represents the cost matrix function, <·> is used to measure the difference between the two, P and Q represent two different distributions, represents the Wosternan distance between distribution P and distribution Q, p represents the transmission plan or matrix, p specifies the mass that should be moved from each element of P to each element of Q, p i,j represents the element of the transmission plan, represents the mass that should be moved from the i-th element of P to the j-th element of Q, c i,j represents the mass cost from the i-th element of P to the j-th element of Q, min π represents the minimization operation on the transmission plan.

5. The method of claim 1, wherein, In step S2, the density loss function formula after introducing the total variable loss is: wherein, represents the total variable loss, f TV (·) represents an L1 norm loss function, L1Loss represents an L1 norm loss, i represents an index value of a current variable, n represents a total variable number, y i represents an i-th real value, x i represents an i-th input value, f(x i ) represents an i-th predicted value.

6. The crowd density estimation method of claim 1, wherein, In step S3, the density evaluation index is calculated using the mean absolute error, and the formula is: where MAE represents the average error between the predicted value and the true value, i represents the number of videos, j represents the number of frames, M represents the total number of videos, N i represents the number of frames in the i-th video, Z i,j represents the number of real people in the j-th frame image of the i-th video, represents the number of predicted people in the j-th frame image of the i-th video.

7. The method of claim 1, wherein, In step S3, the positioning evaluation index is calculated using the following method: using a greedy algorithm to calculate the positional relationship between each predicted point and each real point under different threshold values.

8. A crowd density estimation system, characterized by, A crowd density estimation method for implementing any one of claims 1 to 7, comprising the following modules connected in sequence: A network model construction module is used to construct an improved adjacent space-time perception network; wherein the improved adjacent space-time perception network model includes a backbone network, a neck, a density map output head, and a positioning map output head; A loss function setting module is used to set a density loss function of the improved adjacent space-time perception network model for crowd density detection, and / or set a positioning loss function of the improved adjacent space-time perception network model for crowd positioning; A crowd information analysis module is used to train the improved adjacent space-time perception network model, analyze crowd density detection using a density evaluation index, and / or analyze crowd positioning using a positioning evaluation index.

Images