Crowd counting and localization method based on pixel distance map and four-dimensional dynamic convolution

Abstract

The present invention discloses a crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution, comprising the following steps: 1. marking and smoothing the center points of crowd heads to generate a real pixel distance map; 2. cropping the pixel distance map as input to an FDDNet network and preprocessing it using a feature dimension enhancement module; 3. extracting high-dimensional features of the image using a multi-stage feature extraction module to obtain four feature maps with different resolutions; 4. upsampling and fusing the feature maps; 5. inputting the feature map into a dilated convolution regression head of the FDDNet, and processing it through a dilated convolution layer, a batch normalization layer, and a ReLU activation function layer to obtain a 256×256×1 predicted pixel distance map; 6. training the FDDNet network model; 7. inputting a crowd image into the trained FDDNet to obtain a predicted pixel distance map and finding its local maximum to obtain pedestrian coordinates; 8. the number of local maximums is the number of people, thereby improving the counting accuracy and positioning effect of crowds in dense areas.

Description

Technical Field

[0001] The present invention relates to image processing and computer vision, and in particular to a crowd counting and positioning method based on pixel distance graph and four-dimensional dynamic convolution. Background Art

[0002] As the global population continues to grow, stampedes frequently occur in large-scale events at home and abroad. In order to solve the management and safety problems of densely populated places, crowd counting and positioning has become an important research direction in the field of computer vision. Crowd counting and positioning can realize real-time monitoring and analysis of the number, flow and distribution of people, help urban management departments predict and plan crowd flow, reduce congestion and safety hazards, and improve urban management efficiency. It can be widely used in densely populated public places such as shopping malls, airports, railway stations or subway stations.

[0003] Deep learning-based algorithms can learn the high-dimensional feature information of crowd images through multi-layer network structures. Among various deep learning networks related to crowd analysis, convolutional neural networks (CNNs) perform extremely well. CNNs can effectively learn distinguishing features from crowd analysis datasets and extract prior knowledge, becoming an important component of crowd counting tasks.

[0004] In 2015, Zhang et al. were inspired by the FCN network and designed the first crowd counting network MCNN based on neural network. Its three-column design corresponds to the different scale features in the crowd scene. The three-column network is trained separately and then reasoned together after training, achieving good counting results. In 2016, Li et al. combined VGG16 with dilated convolution and designed a single-column method that can extract features of different scales. This method uses the first 13 layers of VGG16 as the backbone network and adds 6 layers of dilated convolution layers to increase the model receptive field and extract multi-scale information of crowd scenes. At that time, it achieved the best results in crowd counting tasks. The advent of CSRnet promoted the development of human With the rapid development of group counting, GUO et al. added the attention mechanism to the crowd counting method and proposed DADnet. This method also uses the first 13 layers of VGG16 as the backbone. DADnet designed a scale-aware attention in the regression stage, using different expansion scales to discover visual environment clues, avoiding the checkerboard effect caused by the addition of dilated convolution. Subsequently, Gao et al. proposed the SCAR method, adding a more comprehensive attention mechanism to the neural network. SCAR includes two modules, namely SVM and CAM. SVM can encode the pixel-level context information of the entire image, and CAM can extract different feature information between different channels.

[0005] With the development of Transformers in the field of vision, Liang et al. proposed the TransCrowd method, which uses the Transformer's self-attention mechanism to effectively extract crowd semantic information. This is the first Transformer-based crowd counting research. It reframes the crowd counting problem from the perspective of Transformer-based sequence counting. Experimental results show that this model achieves very good performance in dense crowd counting tasks. Tian et al. proposed the CCTrans method, which first blocks the feature map, then calculates self-attention within each block, and finally calculates global self-attention at a key point in each block. This method greatly reduces the computational complexity of the Transformer and also achieves good counting performance.

[0006] In 2018, Laradji et al. proposed four novel loss functions: Image-level loss, Point-level loss, Split-level loss, and False Possitive loss. These four loss functions are better suited for crowd localization tasks, enabling the model to learn the center of pedestrian heads based solely on point labels. This method achieved excellent crowd localization results on four public datasets. Subsequently, Liu et al. proposed a deep detection method, PSDNN, which uses a progressively more challenging curriculum learning strategy to enable the model to generate the most appropriate pseudo-annotated bounding boxes based solely on point labels, thereby detecting pedestrian locations and completing the crowd localization task. In 2019, Sam et al. proposed using Gaussian kernel convolution to process head point annotation images into dense real-valued matrices for supervised model training. They then extracted local maxima from the predicted density map for crowd localization. However, in subsequent research, Sam et al. found that using the crowd density map generated by a regression model for subsequent crowd localization was redundant. Therefore, they proposed a pixel-level binary classification method, LSC-CNN, to predict pedestrian locations directly from crowd scenes to crowd localization. Since the density map supervised learning method can better utilize the rich spatial information, foreground and background information in the image, the use of density map supervised learning has become a new trend in crowd localization tasks.

[0007] Although the above methods have achieved great success in the field of crowd counting and positioning, they still have the following two problems: First, these methods cannot obtain the accurate location and size of individuals. The main reason is that in densely populated areas, the Gaussian spots of nearby people seriously overlap, making the local maximum value inconsistent with the position of the individual, resulting in poor counting and positioning effects; Second, the mainstream crowd positioning network uses a static convolutional neural network, which has difficulty in obtaining prior knowledge of crowd images and has reduced adaptability to different input images, resulting in poor positioning effect. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution, which improves the counting accuracy and positioning effect of crowds in dense areas.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution includes the following steps:

[0011] Step 1: Obtain a training sample dataset. First, annotate the center points of the heads of the target people in the training sample dataset at the pixel level to obtain pixel annotation points representing pedestrians. Then, smooth the pixel annotation points to generate a true pixel distance map for calculating the training loss.

[0012] Step 2: First, crop the real pixel distance map to a size of 256×256×3 and use it as the input data of the FDDNet network. Then, use the feature dimension enhancement module of the FDDNet network to preprocess it into an image of size 64×64×32;

[0013] Step 3: Use the multi-stage feature extraction module of the FDDNet network to extract the high-dimensional features of the 64×64×32 image obtained in step 2, and obtain four feature maps with different resolutions;

[0014] Step 4: Upsample the four feature maps of different resolutions to a feature map of 1 / 4 the size of the original training sample image through bilinear interpolation, and then fuse all the obtained feature maps together to obtain a feature map of size 64×64×32;

[0015] Step 5: Input the 64×64×32 feature map obtained in step 4 into the dilated convolution regression head of the FDDNet network. After processing through the dilated convolution layer, batch normalization layer, and ReLU activation function layer, a 256×256×1 predicted pixel distance map is obtained.

[0016] Step 6: Calculate the mean square error loss between the real pixel distance map and the predicted pixel distance map. After iterative training, optimize the parameters of the FDDNet network model.

[0017] Step 7: First, input a crowd image into the trained FDDNet network model to obtain the predicted pixel distance map. Then, the local maximum detection algorithm LMDS is used to calculate the local maximum of the predicted pixel distance map to obtain the specific coordinates of each pedestrian.

[0018] Step 8: Sum the number of local maxima obtained in step 7 to obtain the number of people in the crowd image.

[0019] Furthermore, the step 1 specifically includes the following steps:

[0020] Step 1.1: Obtain RGB images of different times, locations, and head sizes from multiple different scenes as training sample datasets. Annotate the head center points of the target population in the training sample dataset at the pixel level to obtain pixel annotation points; annotate non-head center points to obtain non-head center annotation points.

[0021] Step 1.2: Calculate the Euclidean distance between the non-head center annotation point and the pixel annotation point and retain the minimum value. The calculation formula is as follows:

[0022]

[0023] Where B represents all target head center annotation points, P(x,y) represents the distance between any non-head center annotation point (x,y) in the image and its nearest pixel annotation point, and the value range of P(x,y) is h and w represent the length and width of the image respectively;

[0024] Step 1.3: Introduce the inverse function to restrict the distribution of the Euclidean distance P(x,y) to the interval (0,1). The true pixel distance map I′ is defined as follows:

[0025]

[0026] Furthermore, the step 3 specifically includes the following steps:

[0027] Step 3.1: Perform the first stage of feature extraction on the 64×64×32 feature map output by the feature dimension enhancement module, perform convolution on the image, and keep the size of the feature map at 64×64×32.

[0028] Step 3.2: Downsample the 64×64×32 feature map output from the first stage to extract features. Keep the network branch from step 3.2. After downsampling, the feature map size is 32×32×64, and perform feature extraction and fusion on the feature map in the second stage.

[0029] Step 3.3: Downsample the feature map of the 32×32×64 branch output by the second stage and retain the network branches of steps 3.2 and 3.3 to obtain a feature map of size 16×16×128. Then perform feature extraction and fusion on the feature map in the third stage.

[0030] Step 3.4: Downsample the 16×16×128 feature map output by the third stage and retain the three network branches of steps 3.2 to 3.4 to obtain a feature map of size 8×8×256. Perform feature extraction and fusion of the feature map in the fourth stage to obtain feature maps of different resolutions.

[0031] Furthermore, steps 3.2 to 3.4 all use four-dimensional dynamic convolution for downsampling, and the four-dimensional dynamic convolution is defined as:

[0032]

[0033] X out =(y1+y2+…+y n )X in

[0034] Where, ω n Represents each candidate convolution kernel W n The attention value, α n represents the attention value along the input channel direction, β n represents the attention value along the output channel direction, and λ n Represents the attention value on the convolution kernel space dimension, ω n , λ n ∈R k*k , C in and C out Represent the number of channels of the feature map before and after convolution, R is the natural number field, k represents the size of the convolution kernel, represents the multiplication operation along different dimensions of the kernel space, y n Represents the multiplication of all candidate convolution kernels with four-dimensional weights, y n With X in Perform convolution and calculate the output feature map X of the dynamic convolution layer out ;

[0035] The four-dimensional dynamic convolution adopts a four-way parallel attention mechanism to obtain the dynamic characteristics of the full dimension of the convolution kernel. The attention mechanism is expressed as:

[0036] AT(x)=Sigmoid(FC(RELU(FC(GAP(X in )))))

[0037] In the formula, Sigmoid() represents the activation function, FC() represents the fully connected layer, RELU() represents the activation function, GAP() represents the global average pooling, X in Represents the input feature map.

[0038] Furthermore, the feature maps of different resolutions obtained in step 3.4 are respectively 1 / 8, 1 / 16, and 1 / 32 of the sizes of the original images in the training sample dataset.

[0039] Furthermore, the expression of the bilinear interpolation in step 4 is:

[0040] f(x,y)≈(1-a)(1-b)f(x1,y1)+a(1-b)f(x2,y2)+abf(x3,y3)+(1-a)bf(x4,y4)

[0041] Where a = x-x1, b = y-y1, f(x,y) is the target pixel value, (x1,y1), (x2,y2), (x3,y3) and (x4,y4) are the four adjacent pixels around each pixel (x,y).

[0042] Furthermore, the expression of the batch normalization layer in step 5 is:

[0043]

[0044] Where x represents the input feature map, E[x] represents the mean of the input feature map, Var[x] represents the variance, and ε=10 -5 , γ and β are scaling and offset parameters respectively.

[0045] Furthermore, the expression of the ReLU function in step 5 is:

[0046] f(x)=max(0,x)

[0047] Where x is the input real number. When x>0, f(x)=x; when x≤0, f(x)=0.

[0048] Furthermore, the expression of the mean square error in step 6 is as follows:

[0049]

[0050] In the formula, y represents the true value of the sample, represents the predicted value of the model, and n represents the number of samples.

[0051] Furthermore, the specific process of step 7 is as follows:

[0052] Step 7.1. Use a 3×3 max pooling operation to obtain all local maxima in the pixel distance map to obtain candidate points. Compare the image containing the candidate points with the pixel distance map predicted by the network pixel by pixel, and retain the pixels that are the same between the two.

[0053] Step 7.2: By setting the parameter noise tolerance τ = 0.3, the maximum value among all local maxima in the retained pixel points is taken as the global maximum value, and the global maximum value is multiplied by the noise tolerance to obtain the adaptive threshold;

[0054] Step 7.3: Compare the global pixel value of the predicted pixel distance map with the adaptive threshold. All pixels less than the adaptive threshold are set to 0, while pixels greater than or equal to the adaptive threshold are set to 1. The position of the pixel 1 is the center position of the pedestrian's head predicted by the network.

[0055] Compared with the prior art, the present invention has the following technical effects:

[0056] The high-resolution network FDDNet based on four-dimensional dynamic convolution (FDDC) proposed in this paper can not only meet the high-resolution output requirements of crowd localization tasks, but also dynamically adjust network parameters according to different input images by linearly combining the attention of different dimensions of convolution kernels, thereby effectively capturing the fine-grained location information of the image; in addition, FDDNet is a lightweight network model with a size of only 67.9MB. Compared with mainstream networks, FDDNet has fewer parameters and can provide high-quality positioning maps, solving the problem of poor crowd counting and positioning effects caused by traditional Gaussian kernel smoothing due to label adhesion and overlap, complex crowd background, and large target scale changes.

[0057] The pixel distance map proposed in this paper is different from the traditional Gaussian smoothed density map. It mainly uses the inverse Euclidean distance to ensure that there are clear boundaries between adjacent heads in a dense crowd without any overlap, so the number of people can be accurately obtained. The local maximum detection algorithm used effectively obtains the coordinates of individuals by locating local maxima, and shows strong robustness to negative samples, thereby improving the positioning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow diagram of the present invention;

[0059] Figure 2 This is a structural diagram of the high-resolution network FDDNet based on four-dimensional dynamic convolution of the present invention;

[0060] Figure 3 It is a four-dimensional dynamic convolution structure diagram;

[0061] Figure 4 These are some visualization results of different methods on the NWPU dataset. DETAILED DESCRIPTION

[0062] The specific contents of the present invention are further explained in detail below with reference to the embodiments.

[0063] like Figure 1 As shown in FIG, a crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution includes the following steps:

[0064] Step 1. First, obtain RGB images of different times, different locations, and different head sizes from multiple different scenes as training sample data sets, and then annotate the head center points of the target people in the training sample images at the pixel level to obtain pixel annotation points (i.e., head center annotation points). The pixel annotation points are recorded as "1". The annotation of a head center point represents a pedestrian, and the other non-head center points are marked as "0". The sum of the pixel annotation points marked as 1 is the total number of people in the image; since the use of deep learning to predict the number of people in the image requires first training an optimal network model, but the discrete pixel annotation cannot calculate the loss with the output of the neural network, it is necessary to smooth the pixel annotation points to generate a real pixel distance map for calculating the training loss. The process of smoothing the pixel annotation points is: first calculate the Euclidean distance between each non-head center annotation point and each target head center annotation point and retain the minimum value. The calculation formula is as follows:

[0065]

[0066] Where B represents all target head center annotation points, P(x,y) represents the distance between any non-head center annotation point (x,y) in the image and its nearest head center annotation point, and the value range of P(x,y) is Where h and w represent the length and width of the image respectively;

[0067] Since the Euclidean distance P(x, y) varies greatly, its direct participation in network training may cause the model to be difficult to converge and it is impossible to directly generate a valid pixel distance map. Therefore, this embodiment introduces an inverse function to limit the distance distribution to the interval (0, 1) to facilitate model learning and optimization. The true pixel distance map is defined as follows:

[0068]

[0069] Where I′ represents the real pixel distance map. To avoid the denominator being 0, this embodiment adds a constant 1 to the Euclidean distance P(x, y) between pixels.

[0070] Step 2: Crop the real pixel distance map to a size of 256×256×3 and use the cropped image as Figure 2 The input data of the FDDNet network shown is preprocessed into a 64×64×32 image using the feature dimension enhancement module of the FDDNet network;

[0071] Step 3: Use the multi-stage feature extraction module of the FDDNet network to extract the high-dimensional features of the output data of the feature dimension enhancement module. The specific process is as follows:

[0072] Step 3.1: Perform the first stage of feature extraction on the 64×64×32 feature map output by the feature dimension enhancement module, perform convolution on the image, and keep the size of the feature map at 64×64×32.

[0073] Step 3.2: Downsample the 64×64×32 feature map output from the first stage to extract features. Keep the network branch from step 3.2. After downsampling, the feature map size is 32×32×64, and perform feature extraction and fusion on the feature map in the second stage.

[0074] Step 3.3: Downsample the feature map of the 32×32×64 branch output by the second stage and retain the network branches of steps 3.2 and 3.3 to obtain a feature map of size 16×16×128. Then perform feature extraction and fusion on the feature map in the third stage.

[0075] Step 3.4: Downsample the 16×16×128 feature map output by the third stage and retain the three network branches from steps 3.2 to 3.4 to obtain a feature map of size 8×8×256. Perform feature extraction and fusion of the fourth stage on the feature map to obtain feature maps of different resolutions.

[0076] Steps 3.2 to 3.4 are all done as follows Figure 3 The four-dimensional dynamic convolution shown in the figure is down-sampled. The traditional dynamic convolution only pays attention to one dimension of the convolution kernel. The convolution kernel proposed in this embodiment includes four dimensions, namely the number of convolution kernels, convolution kernel input, convolution kernel output and spatial size to construct a four-dimensional dynamic convolution. The four-dimensional dynamic convolution adopts a four-way parallel attention mechanism to obtain the dynamic characteristics of the full dimension of the convolution kernel. The attention mechanism is expressed as:

[0077] AT(x)=Sigmoid(FC(RELU(FC(GAP(X in )))))

[0078] In the formula, Sigmoid() represents the activation function, FC() represents the fully connected layer, RELU() represents the activation function, GAP() represents the global average pooling, X in Represents the input feature map;

[0079] The four-dimensional dynamic convolution is defined as follows:

[0080]

[0081] X out =(y1+y2+…+yn )X in

[0082] Where, ω n Represents each candidate convolution kernel W n The attention value, α n represents the attention value along the input channel direction, β n represents the attention value along the output channel direction, and λ n Represents the attention value on the convolution kernel space dimension, ω n , λ n ∈R k*k , R is the natural number field, k represents the size of the convolution kernel, C in and C out Represents the number of channels of the feature map before and after convolution, represents the multiplication operation along different dimensions of the kernel space, y n Represents the multiplication of all candidate convolution kernels with four-dimensional weights, y n With X in Perform convolution and calculate the output feature map X of the dynamic convolution layer out ;

[0083] Step 4: Upsample the feature maps of different resolutions output from the four stages of step 3, i.e., the feature maps of 1 / 8, 1 / 16, and 1 / 32 of the original training sample image, to a feature map of 1 / 4 of the original training sample image through bilinear interpolation, and fuse all these feature maps together to finally obtain a feature map of size 64×64×32. The calculation formula of the bilinear interpolation is expressed as:

[0084] f(x,y)≈(1-a)(1-b)f(x1,y1)+a(1-b)f(x2,y2)+abf(x3,y3)+(1-a)bf(x4,y4)

[0085] Where a = x-x1, b = y-y1, f(x,y) is the target pixel value. For each pixel point (x,y) in the new image, first find the four adjacent pixel points (x1,y1), (x2,y2), (x3,y3), (x4,y4) around it. These four points are the four pixels closest to the point (x,y) along the x and y directions on the original image of the training sample. Calculate the distance weight of each adjacent pixel point in the x and y directions, and perform a weighted average of the values ​​of the four neighboring pixels according to the distance weight.

[0086] Step 5: Input the 64×64×32 feature map obtained in step 4 into the dilated convolution regression head of the FDDNet network. After the dilated convolution, the batch normalization layer is used to alleviate the internal covariate shift problem caused by network updates, thereby improving the training speed and stability of the network. After the ReLU activation function layer, the predicted pixel distance map of 256×256×1 is finally obtained.

[0087] The batch normalization layer alleviates the internal covariate shift problem caused by network updates by standardizing the data within each batch to make the data distribution more stable, thereby improving the training and generalization capabilities of the model. Specifically, batch normalization can make the model more robust, allowing models trained on different batches to migrate to each other, alleviating the gradient vanishing problem and reducing the risk of overfitting. The calculation formula of the batch normalization layer is as follows:

[0088]

[0089] Where x represents the input feature map, E[x] represents the mean of the input feature map, Var[x] represents the variance, and ε is a very small positive value. In this embodiment, ε=10 -5 ,γ and β are scaling and offset parameters respectively;

[0090] The main advantage of the ReLU activation function layer is its linear and non-saturated properties, which makes the network easier to optimize. The expression of the ReLU function is:

[0091] f(x)=max(0,x)

[0092] Where x is the input real number. When the input x>0, the output value of the ReLU function is f(x)=x; when x≤0, the output value of the ReLU function is f(x)=0. The derivative of the ReLU function is 1 when x≥0 and 0 when x<0. Therefore, the calculation of the gradient is more efficient during back propagation, avoiding the problem of gradient disappearance.

[0093] Step 6: Use the mean square error loss for training, calculate the mean square error loss between the real pixel distance map and the predicted pixel distance map output in step 5, iteratively train the mean square error, and optimize the parameters of the FDDNet network model;

[0094] The advantage of using mean square error as the loss function in this embodiment is that it is a simple and intuitive evaluation method that is widely used in regression problems. It measures the distance between the predicted value and the true value. The smaller the value, the better the model fit. The calculation formula of mean square error is as follows:

[0095]

[0096] In the formula, y represents the true value of the sample, Represents the predicted value of the model, n represents the number of samples, and the formula means dividing the sum of the squares of the prediction errors of each data point by the number of samples n;

[0097] Step 7: Input a crowd image into the trained FDDNet network model to obtain the predicted pixel distance map. Then, the local maximum detection algorithm (LMDS) is used to calculate the local maximum of the predicted pixel distance map to obtain the specific coordinates of each pedestrian. The specific process is as follows:

[0098] Step 7.1. Use a 3×3 max pooling operation to obtain all local maxima in the pixel distance map, i.e., candidate points. Compare the image containing the candidate points with the pixel distance map predicted by the network pixel by pixel, and retain the pixels that are the same between the two.

[0099] Step 7.2: By setting the parameter noise tolerance τ = 0.3, the maximum value among all local maxima in the retained pixel points is taken as the global maximum value, and the global maximum value is multiplied by the noise tolerance to obtain the adaptive threshold;

[0100] Step 7.3: Compare the pixel value corresponding to each pixel in the predicted pixel distance map (i.e., the global pixel value) with the adaptive threshold. Set all pixels less than the adaptive threshold to 0, and set pixels greater than or equal to the adaptive threshold to 1. At this point, the position where the pixel is 1 is the center of the pedestrian's head predicted by the network.

[0101] Step 8: Sum the number of local maxima obtained in step 7 to obtain the number of people in the crowd image.

[0102] The effectiveness of the dense crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution proposed in this embodiment is verified through the following experiments:

[0103] (1) Dataset

[0104] In order to verify the counting effect of the present invention on dense crowds, experiments were conducted on three datasets: ShanghaiTech, UCF_QNRF and NWPU, where: the ShanghaiTech dataset consists of two parts, A and B, 300 images in Part A are used for training, and 183 images are used for testing; 400 images in Part B are used for training, and 316 images are used for testing, with the average resolution of PartA being 589×868 and the average resolution of PartB being 768×1024; the UCF-QRNF dataset has a total of 1535 images, including 1201 training sets and 334 test sets, with an average resolution of 2013×2902; the NWPU-Crowd dataset has a total of 5109 images, including 3109 training sets, 500 validation sets, and 1500 test sets, with an average resolution of 2191×3209.

[0105] (2) Experimental details

[0106] This paper is implemented based on the PyTorch framework and accelerated training is completed on an NVIDIA GeForce RTX 3090 graphics card with 24G of video memory. To prevent overfitting, horizontal flipping and random cropping are used to enhance the data, and the image cropping size in Part A and B of the Shanghai Tech dataset is 256×256, and the cropping size of the images in other datasets is 512×512. The Adam optimizer is used to optimize the network, the learning rate is set to 0.0001, the weight decay is set to 0.0005, and the batch size is 8 during training.

[0107] (3) Evaluation indicators

[0108] When measuring counting performance, this embodiment uses the mean absolute error (MAE) and mean square error (MSE) indicators to test the counting results and the generated density map results. The smaller the MAE and MSE results, the better the model counting effect. The mean absolute error and mean square error are calculated as follows:

[0109]

[0110]

[0111] Where N is the number of samples from the test set, Y i and are the predicted and true numbers of the i-th test image, respectively. The predicted count is obtained by summing the crowd density map output by the model;

[0112] When measuring the positioning performance, the precision, recall and F1-measure are used as the evaluation indicators of the positioning results. The estimated crowd head positioning point P is used in the indicator calculation. p and the true value annotation point P g Perform one-to-one matching association, when the prediction point P p With the marked point P g When the distance between them is less than the radius β of the circumscribed circle determined by the bounding box, the two points are considered to be matched and are called true positive TP. Otherwise, they are false positive FP. p If there is no labeled point matching it, it is called a false negative FN. The radius β of the circumscribed circle determined by the dataset bounding box is defined as follows:

[0113]

[0114] Where w and h are the height and width of the bounding box, respectively. Precision is the ratio of true positive samples to all matched points. Recall is the ratio of true positive samples to points that should have been matched. F1-measure is a comprehensive evaluation of precision and regression rate. α is a constant with a value of 1. The specific formula is as follows:

[0115]

[0116]

[0117]

[0118] (4) Ablation studies

[0119] To further verify the effectiveness of different modules in the FDDNet network proposed in this embodiment, an ablation experiment was conducted on the ShanghaiTechPartA dataset. Table 1 shows the results of experiments using FDDNet as the basic network, using Gaussian smoothed density map (Gaussian) and pixel distance map (Pixel-map) methods respectively. It can be seen that compared with the FDDNet+Gaussian method, the FDDNet+Pixel-map method improves MAE and MSE by 3.5% and 2.2% in the counting task, respectively. At the same time, it also achieves better results than the Gaussian smoothing method in the positioning task, further proving the effectiveness of the pixel distance map proposed in this embodiment.

[0120] Table 1 Pixel distance map ablation experiment results

[0121] Backbone MAE MSE F1(％) FDDNet+Gaussian 59.6 108.1 71.1 FDDNet+Pixel-map 57.5 104.4 77.0

[0122] Table 2 shows the experiments using pixel distance maps as labels, using DLMP-Net, CSRNet and high-resolution network FDDNet. It is observed that the results of using high-resolution network FDDNet as the baseline network are better than the above two methods, which proves the effectiveness of high-resolution network as the baseline network.

[0123] Table 2. Experimental results of high-resolution network ablation

[0124] method MAE MSE CSRNet 66.4 108.0 DLMP-Net 58.6 85.2 High-resolution network 58.1 80.3

[0125] Table 3 compares the classic dynamic convolution CondConv, DYConv and the four-dimensional dynamic convolution (FDDC) proposed in this embodiment. It can be concluded from Table 3 that CondConv and DYConv have certain performance improvements on the positioning results, but the four-dimensional dynamic convolution shows better prediction results. That is to say, compared with CondConv and DYConv, the positioning accuracy of the four-dimensional dynamic convolution is improved by 2.5% and 1.2% respectively. This shows that the four-dimensional dynamic convolution has stronger performance in the crowd positioning task and can more accurately capture the location information of the crowd.

[0126] Table 3. Four-dimensional dynamic convolution ablation experimental results

[0127] method Parameters(M) F1(％) HRNet+Pixelmap 66.5 73.0 +CondConv 69.8 74.8 +DYConv 69.4 76.1 +FDDC 67.9 77.0

[0128] (5) Comparison with existing technologies

[0129] To verify the effectiveness of the FDDNet network in this embodiment, experiments were conducted on three public datasets: ShanghaiTech, UCF-QNRF, and NWPU. The network was also compared with advanced technologies such as MCNN, CSRNet, DADnet, SCAR, SFCN+, SUA-Fully, MFP-Net, TransCrowd, and DLMP-Net. The results are shown in Tables 4 to 7.

[0130] Table 4 Comparison results of counting experiments using different technologies on Shanghai Tech and UCF-QNRF datasets

[0131]

[0132]

[0133] Table 5 Comparison results of counting experiments with different technologies on the NWPU dataset

[0134]

[0135] For the ShanghaiTech dataset: As can be seen from Table 4, FDDNet achieved the best performance in ShanghaiTech PartA, with a MAE of 57.5. Compared with the classic method CSRNet, MAE increased by 47.2% and MSE increased by 39.7%. This result fully demonstrates the effectiveness of the high-resolution network; in ShanghaiTech PartB, FDDNet's MAE index reached the optimal value of 6.8, which was 4.2% higher than the latest method DLMP-Net.

[0136] For the UCF-QRNF dataset: As can be seen from Table 4, FDDNet achieves the best counting performance, with MAE and MSE of 88.4 and 153.2 respectively. Compared with the latest method MFP-Net, MAE is improved by 21.0% and MSE is improved by 19.6%. Since the UCF-QRNF dataset contains a variety of crowd scenes and even unmanned scenes, it can be proved that the four-dimensional dynamic convolution proposed in this embodiment can effectively extract specific features to ensure the accuracy of counting.

[0137] For the NWPU-Crowd dataset: As can be seen from Table 5, FDDNet achieves the best performance in the crowd counting task, with MAE and MSE of 82.4 and 334.7 respectively. Compared with the latest method DLMP-Net, MAE is improved by 6.0% and MSE is improved by 22.4%; Figure 4 The visualization results of crowd counting by FDDNet on the NWPU-Crowd dataset are shown. As can be observed from the third column of the figure, FDDNet can predict clear target boundaries in both sparse and dense crowd areas, and can correctly handle the overlap between targets in densely populated areas. This dataset contains more scenes with huge scale changes. This is because the high-resolution network used can capture multi-scale features, thereby extracting richer detailed information, which is conducive to the feature extraction of small targets.

[0138] Table 6 Comparison results of positioning experiments using different technologies on the Shanghai Tech and UCF-QNRF datasets

[0139]

[0140] Table 7 Comparison results of positioning experiments with different technologies on the NWPU dataset

[0141]

[0142] As shown in Table 6, on the ShanghaiTech dataset, FDDNet achieves a 2.7% improvement in F1, a 4.3% improvement in Pre, and a 4.0% improvement in Rec over the best-performing crowd localization method, Tpcount, on ShanghaiTech Part A. Compared to TFaces, FDDNet achieves a 20.0% improvement in F1. On ShanghaiTech Part B, FDDNet achieves an 8.9% improvement in F1, a 7.1% improvement in Pre, and an 8.4% improvement in Rec over the state-of-the-art TpCount method. This significant improvement in all metrics over TFaces is due to FDDNet's ability to provide high-quality pixel-wise distance maps, which improves localization accuracy.

[0143] For the UCF-QRNF dataset, as shown in Table 6, FDDNet outperforms mainstream methods in various positioning indicators. Compared with the latest method TpCount, its F1 improves by 0.5%, Pre improves by 2.1%, and Rec improves by 3.2%. Compared with the mainstream crowd positioning methods TFaces, RALoc, and LS-CNN, F1 improves by 32.9%, 44.8%, and 6.2%, respectively. In addition, FDDNet also achieves the highest Pre and Rec.

[0144] For the NWPU-Crowd dataset: As shown in Table 7, FDDNet achieved the second highest Pre, and its F1 improved by 12.4% compared with the mainstream suboptimal method RALoc. In the test set, although Faster-RCNN still had the highest Pre, reaching 95.8%, its F1 was only 6.7%. Among all positioning methods, FDDNet achieved the best F1, which was 2.1% higher than the mainstream method TpCount. The experimental results fully demonstrated the advantages of FDDNet in crowd positioning tasks. Figure 4 The visualization results of crowd localization using FDDNet on the NWPU-Crowd dataset are shown. From the fourth column of the figure, it can be observed that the proposed method can accurately obtain the position coordinates of pedestrians in both sparse and dense crowd areas, proving the effectiveness of the local maximum detection algorithm design.

[0145] In summary, compared with the existing technology, the high-resolution network structure based on four-dimensional dynamic convolution in this embodiment can effectively capture the local and global information of the image. The use of void convolution enhances the network's ability to capture targets of different scales, thereby effectively solving the problem of low counting accuracy caused by the complex background of crowd density images.

Claims

1. A crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution, characterized in that: The steps include: Step 1: Obtain a training sample dataset. First, annotate the center points of the heads of the target people in the training sample dataset at the pixel level to obtain pixel annotation points representing pedestrians. Then, smooth the pixel annotation points to generate a true pixel distance map for calculating the training loss. Step 2: First, crop the real pixel distance map to a size of 256×256×3 and use it as the input data of the FDDNet network. Then, use the feature dimension enhancement module of the FDDNet network to preprocess it into an image of size 64×64×32; Step 3: Use the multi-stage feature extraction module of the FDDNet network to extract the high-dimensional features of the 64×64×32 image obtained in step 2, and obtain four feature maps of different resolutions. The specific steps include the following: Step 3.1: Perform the first stage of feature extraction on the 64×64×32 feature map output by the feature dimension enhancement module, perform convolution on the image, and keep the size of the feature map at 64×64×32. Step 3.2: Downsample the 64×64×32 feature map output from the first stage to extract features. Keep the network branch from step 3.

2. After downsampling, the feature map size is 32×32×64, and perform feature extraction and fusion on the feature map in the second stage. Step 3.3: Downsample the feature map of the 32×32×64 branch output by the second stage and retain the network branches of steps 3.2 and 3.3 to obtain a feature map of size 16×16×128. Then perform feature extraction and fusion on the feature map in the third stage. Step 3.4: Downsample the 16×16×128 feature map output by the third stage and retain the three network branches from steps 3.2 to 3.4 to obtain a feature map of size 8×8×256. Then, perform feature extraction and fusion on the feature map in the fourth stage to obtain feature maps of different resolutions. Steps 3.2 to 3.4 all use four-dimensional dynamic convolution for downsampling, and the four-dimensional dynamic convolution is defined as: Where, Represents each candidate convolution kernel The attention value, represents the attention value along the input channel direction, represents the attention value along the output channel direction, and Represents the attention value on the convolution kernel space dimension, , , , , C in and C out Represent the number of channels of the feature map before and after convolution, R is the natural number field, k represents the size of the convolution kernel, represents the multiplication operation along different dimensions of the kernel space, Represents the multiplication of all candidate convolution kernels with four-dimensional weights, and Perform convolution and calculate the output feature map of the dynamic convolution layer ; The four-dimensional dynamic convolution adopts a four-way parallel attention mechanism to obtain the dynamic characteristics of the full dimension of the convolution kernel. The attention mechanism is expressed as: Where, represents the activation function, represents the fully connected layer, represents the activation function, represents global average pooling, Represents the input feature map; Step 4: Upsample the four feature maps of different resolutions to a feature map of 1 / 4 the size of the original training sample image through bilinear interpolation, and then fuse all the obtained feature maps together to obtain a feature map of size 64×64×32; Step 5: Input the 64×64×32 feature map obtained in step 4 into the dilated convolution regression head of the FDDNet network. After processing through the dilated convolution layer, batch normalization layer, and ReLU activation function layer, a 256×256×1 predicted pixel distance map is obtained. Step 6: Calculate the mean square error loss between the real pixel distance map and the predicted pixel distance map. After iterative training, optimize the parameters of the FDDNet network model. Step 7: First, input a crowd image into the trained FDDNet network model to obtain the predicted pixel distance map. Then, the local maximum detection algorithm LMDS is used to calculate the local maximum of the predicted pixel distance map to obtain the specific coordinates of each pedestrian. Step 8: Sum the number of local maxima obtained in step 7 to obtain the number of people in the crowd image.

2. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The step 1 specifically includes the following steps: Step 1.1: Obtain RGB images of different times, locations, and head sizes from multiple different scenes as training sample datasets. Annotate the center points of the heads of the target people in the training sample datasets at the pixel level to obtain pixel annotation points. Mark the non-head center points to obtain non-head center marking points; Step 1.2: Calculate the Euclidean distance between the non-head center annotation point and the pixel annotation point and retain the minimum value. The calculation formula is as follows: Where, Represents all target head center annotation points, Represents any non-head center annotation point in the image The distance to the nearest pixel annotation point, The value range of , and Represent the length and width of the image respectively; Step 1.3: Introduce the inverse function to convert the Euclidean distance Distribution is limited to Interval, true pixel distance map The definition is as follows: 。 3. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The feature maps of different resolutions obtained in step 3.4 are 1 / 8, 1 / 16, and 1 / 32 of the original images in the training sample dataset, respectively.

4. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The expression of the bilinear interpolation in step 4 is: Where, , , is the target pixel value, , , and For each pixel The four adjacent pixels around .

5. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The expression of the batch normalization layer in step 5 is: Where, represents the input feature map, represents the mean of the input feature map, represents the variance, = , and are the scaling and offset parameters respectively.

6. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The expression of the ReLU function in step 5 is: Where, is the input real number, when the input hour, ;when hour, .

7. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The expression of the mean square error of step 6 is as follows: Where, represents the true value of the sample, represents the predicted value of the model, Represents the sample size.

8. The crowd counting and positioning method based on pixel distance map and four-dimensional dynamic convolution according to claim 1 is characterized in that: The specific process of step 7 is as follows: Step 7.

1. Use a 3×3 max pooling operation to obtain all local maxima in the pixel distance map to obtain candidate points. Compare the image containing the candidate points with the pixel distance map predicted by the network pixel by pixel, and retain the pixels that are the same between the two. Step 7.2: By setting the parameter noise tolerance , the maximum value among all local maxima in the retained pixels is taken as the global maximum value, and the global maximum value is multiplied by the noise tolerance to obtain the adaptive threshold; Step 7.3: Compare the global pixel value of the predicted pixel distance map with the adaptive threshold. All pixels less than the adaptive threshold are set to 0, while pixels greater than or equal to the adaptive threshold are set to 1. The position of the pixel 1 is the center position of the pedestrian's head predicted by the network.

Images