A dense crowd counting method based on neural network

Abstract

This paper provides a dense crowd counting method based on a neural network. To address the problem of counting based on density maps, a novel neighborhood summation-based strategy is introduced to improve the correlation between density map pixels, thereby improving the accuracy and robustness of counting in a simple and effective manner. To address the problem that the convolutional layer that aggregates image depth features cannot efficiently utilize inter-channel features, a global response normalization layer is introduced to alleviate the problem of information utilization between multi-channel features. The global response normalization layer increases competition between multi-channel information to ensure that the network retains more useful information when aggregating features from multiple channels.

Description

Technical Field

[0001] The present invention relates to the field of image processing based on deep learning, and in particular to a dense crowd counting method based on a neural network. Background Art

[0002] Counting, especially the problem of counting dense crowds, is an important issue that has received widespread attention in computer vision. Accurately estimating the distribution and number of people in a scene is of great significance in scenarios such as public safety, traffic management, and scenic area diversion. Currently, density map estimation is a commonly used method for counting dense crowds. Density map-based methods usually convert the object point annotations of the image into a density distribution map as supervision information for network training. In order to make the density distribution map closer to the distribution of real portraits, some methods will also choose to Gaussian blur the density map obtained by point annotation. Finally, the mean absolute error, mean square error, etc. are used as supervision functions to train the network.

[0003] Due to the large variations in density distribution and scale in crowd images, many methods focus on designing loss functions that effectively utilize information from density maps. Simple mean absolute error or mean squared error are insensitive to outliers in point annotations, while manual point annotation often contains inaccurate noisy annotations, which inevitably complicates counting. To improve robustness to noise, some methods incorporate solutions such as Bayesian loss and optimal transfer to combat annotation noise and scale variations in density distribution.

[0004] Density map-based methods have inherent limitations. In the density map output by the network model, the value of each pixel corresponds to the number of people in the corresponding area of ​​the original image. Summing the density map yields an estimate of the total number of people in the image. However, each pixel is an independent unit with no connections between them. This results in relatively limited image features learned by the network, making them susceptible to the influence of anomalies in the annotations. This makes it difficult to understand the distribution of target objects in complex scenes.

[0005] In addition, since the generation of density maps requires the aggregation of image depth features from multiple channels, a common method is to use a simple convolution layer to reduce the dimensionality of the depth features. However, due to the large number of depth feature channels, simple convolution cannot avoid the loss of information between channels, and the generated density map has poor stability. Summary of the Invention

[0006] To address the above-mentioned deficiencies in the prior art, the present invention aims to provide a dense crowd counting method based on a neural network. This method addresses the problem of density map counting by introducing a novel neighborhood summation strategy to improve the correlation between density map pixels, thereby improving the accuracy and robustness of the counting method in a simple and effective manner. Guided by the neighborhood summation method, the model can better understand the distribution and relationships of target objects by fully utilizing the information between neighborhood pixels in the density map. At the same time, neighborhood summation blurs the relative distances between targets, which is beneficial for combating outliers and noise in point annotations and improving the robustness of the model. To address the problem that the convolutional layer that aggregates image depth features cannot efficiently utilize inter-channel features, a global response normalization layer is introduced to alleviate the problem of information utilization between multi-channel features. The global response normalization layer increases competition between multi-channel information to ensure that more useful information is retained when the network aggregates features from multiple channels. Global response normalization mainly consists of three parts: global feature aggregation, feature normalization, and feature calibration.

[0007] According to a first aspect of the present invention, a method for counting dense crowds based on a neural network is provided, comprising:

[0008] Step 10: Obtain crowd images and corresponding labeled data, and perform preprocessing operations to obtain a training dataset.

[0009] Step 20: Use vgg-19 as the backbone network, delete the linear prediction layer and the last maximum pooling layer of vgg-19, and add a feature aggregation layer; add a global response normalization module in the feature aggregation layer; and obtain the crowd counting model

[0010] Among them, the processing of the global response normalization layer is:

[0011]

[0012] Among them, G(·) is a normalization function based on the L2 norm, is a standard division normalization function, γ and β are two learnable parameters; is the input feature map, is the feature map after global normalization.

[0013] Step 30: Train the crowd counting model according to the training data set to obtain a trained crowd counting model. The counting loss calculation process based on neighborhood summation guidance used in the training includes steps 31-34:

[0014] Step 31: Obtain the true density map of the corresponding crowd image through the labeled data of the training dataset Obtaining the predicted density map of a crowd image using a crowd counting model Where W and H are the width and height of the density map.

[0015] Step 32: D t and D p Perform normalization:

[0016]

[0017] Among them, D t and D p Represent the true density map and the predicted density map respectively, ||·||1 represents the L1 norm of a vector, ||D t ||1 and ||D p ||1 represents the real count and the real count respectively, ε is the stability factor, and ε=10 -6 , D′ t , D′ p That is, the normalized true density map and predicted density map.

[0018] Step 33: Calculate the counting loss under neighborhood summation guidance:

[0019]

[0020] in, represents neighborhood summation, that is, summing the pixel points in the density map and the pixels around it, ||·||1 represents the mean absolute error, represents the counting loss guided by neighborhood summation.

[0021] Step 34: By cascading multiple neighborhood summation operations, the pixel points are summed to obtain more neighborhood information. The expanded neighborhood summation is:

[0022]

[0023] Where i represents the number of steps of the multi-level neighborhood summation operation. When i = 0, that is, no neighborhood summation operation is performed, D′ ti =D′ t , D′ pi =D′ p , when i>0, n is the total number of steps in neighborhood summation.

[0024] Step 40: Input the image to be counted into the trained crowd counting model to obtain the predicted density map To D p Perform pixel-by-pixel summation, i.e.

[0025]

[0026] Where i∈[0, H-1], j∈[0, W-1] represents the coordinates of the pixel point, H, W represent the height and width of the density map respectively, D p (1, i, j) represents the value of the pixel in the density map, and count is the number of people in the image to be counted.

[0027] Furthermore, in the method provided by the present invention, step 10 also includes: step 11: scaling the crowd image using a bicubic interpolation algorithm to limit the narrow edge of the crowd image to within the acceptable range of the crowd counting model; step 12: processing the labeled data to obtain a labeled density map of the crowd image; step 13: randomly cropping the crowd image to a certain size to adapt to the batch training process.

[0028] Furthermore, the method provided by the present invention, step 20 also includes: the obtained crowd counting model includes a feature extraction layer and a feature aggregation layer; the feature extraction layer is a vgg19 backbone architecture, including a four-order architecture: the first and second orders include two convolutional layers and one maximum pooling layer, the third and fourth orders include four convolutional layers and one pooling layer, and the fifth order includes four convolutional layers; the feature aggregation layer includes: the number of channels extracted by the feature extraction layer is 512 image features, which is reduced to 128 channels after the first convolutional layer, processed by the first GELU activation function and the first global response normalization layer, and then reduced to 32 channels after the second convolutional layer, processed by the second GELU activation function and the second global response normalization layer, and then reduced to 1 channel after the third convolutional layer, and finally output through the ReLU activation function to obtain the final output predicted density map result.

[0029] Furthermore, the method provided by the present invention, step 30 also includes: during the training process, using images with a crop size of 512×512 for batch training, selecting Adam as the optimizer of network parameters, and the learning rate of 10 -5 , the weight decay is 10 -5 , the number of training cycles is set to 1000.

[0030] According to a second aspect of the present invention, a computer device is provided, comprising: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the method of the first aspect.

[0031] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the method of the first aspect is executed.

[0032] This invention introduces a neighborhood summation strategy into the counting loss, leveraging information between pixels and their neighbors to improve the network model's understanding of the distribution and relationships of target objects in the image. Furthermore, a global response normalization layer is added to the feature aggregation layer to enhance the ability to aggregate information from multi-channel features. Compared to existing technologies, the above-described technical solution conceived by this invention has at least the following beneficial effects:

[0033] 1. Design and construction of a neighborhood sum-based loss function, including obtaining a density map predicted by a neural network and annotated density maps; normalizing the predicted density map and the annotated density map using the L1 norm; obtaining a summed density feature map using the neighborhood summation method, and calculating a neighborhood sum-guided counting loss using the L1 loss;

[0034] 2. Cascade extension of neighborhood summation, including multi-level nested construction of neighborhood summation and a method for aggregating losses at different layers to obtain the total output loss;

[0035] 3. The design of a feature aggregation layer based on the global response normalization layer, including the method of adding a global response normalization layer to the feature aggregation layer added after the backbone network, using global response normalization to alleviate the loss and collapse problems of multi-channel feature aggregation.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] Figure 1 It is a schematic diagram of an overall architecture according to an exemplary embodiment.

[0039] Figure 2 is a schematic diagram of a neighborhood summation operation according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0041] The dense crowd counting method based on neural network proposed in this invention mainly consists of the following steps:

[0042] 1. Obtain crowd images and related annotation data and perform certain preprocessing operations, including: scaling the image using the bicubic interpolation algorithm to limit the narrow edge of the image to the acceptable range of the neural network; processing the point annotations to obtain the annotation density map of the dense crowd image; and randomly cropping the crowd image to a certain size to accommodate the batch training process;

[0043] 2. Use the general VGG-19 as the backbone network of the network; delete the linear prediction layer and the last maximum pooling layer of VGG-19, and add a feature aggregation layer; add a global response normalization module in the feature aggregation layer; the global response normalization layer can be expressed as follows:

[0044]

[0045] Among them, G(·) is a normalization function based on the L2 norm, is a standard division normalization function, γ and β are two learnable parameters. The final architecture diagram is as follows Figure 1 shown.

[0046] 3. During the training process, we used images with a crop size of 512×512 for batch training, and selected Adam as the optimizer for network parameters with a learning rate of 10. -5 , the weight decay is 10 -5 , the number of training cycles is set to 1000;

[0047] The counting loss based on neighborhood summation guidance proposed in this patent mainly consists of the following steps:

[0048] 1. Obtain the true density map of the input image through data set annotation Obtain the predicted density map of the image through the backbone neural network Where W and H are the width and height of the density map;

[0049] 2. D t and D p The following normalization process is performed, namely

[0050]

[0051] Among them, ||·||1 represents the L1 norm of a vector, ||D t ||1 and ||D p ||1 represents the real count and the real count respectively, ε is the stability factor, and ε=10 -6 , D′ t , D′ p That is, the normalized true density map and predicted density map;

[0052] 3. Finally, the counting loss guided by neighborhood summation is obtained by the following formula:

[0053]

[0054] in, represents neighborhood summation, that is, summing the pixel points in the density map and the pixels around it, ||·||1 represents the mean absolute error, represents the counting loss guided by neighborhood summation. In practical applications, neighborhood summation is mainly implemented through convolution. For example, a convolution with a kernel size of 3×3, with its weight set to 1 and bias set to 0, can sum the pixel values ​​of a pixel and its surrounding eight points. Figure 2 A diagram briefly illustrating the neighborhood sum operation.

[0055] The neighborhood summation operation is easy to extend. By cascading multiple neighborhood summation operations, the pixel summation can be used to obtain more neighborhood information. The extended neighborhood summation algorithm can be expressed as follows:

[0056]

[0057] Among them, when i=0, D′ ti =D′ t , D′ pi =D′ p , when i>0, The expanded neighborhood summation is more suitable for sparse density maps. However, when the crowd is relatively dense, the model's ability to perceive the target distribution will be reduced due to the larger receptive field. In practical applications, the appropriate number of layers should be selected based on the target distribution.

[0058] Unlike methods like Gaussian blurring, which blur the labeled density map, the neighborhood summation method does not directly affect the content of the density map learned by the model, but rather provides additional constraints on it. While methods like Gaussian blurring do not address the issue of isolated pixels in the density map, neighborhood summation can enhance the information exchange between pixels to a certain extent, enabling the network to learn richer distribution features.

[0059] We selected the method proposed in DM-Count as the backbone architecture of the network, added a global response normalization layer to the feature aggregation layer, added the proposed neighborhood summation-based counting loss to the loss function, and conducted relevant experiments on the ShanghaiTechPart A dataset. The relevant experimental results are as follows:

[0060] Table 1 Experimental results of ShanghaiTech Part A dataset

[0061]

[0062]

[0063] Among them, MSE and MAE represent the mean square error and mean absolute error of the model's predicted counts and labeled counts on the test set, respectively. MSE is sensitive to outliers and reflects the stability of the model's predictions. MAE is relatively stable and reflects the accuracy of the model's predictions. represents the application of neighborhood sum-guided counting loss, and GRN represents the application of a global response normalization layer.

[0064] The neighborhood sum-guided counting loss method improves both accuracy and stability on the ShanghaiTech Part A dataset compared to the baseline method DM-Count. Due to the relatively dense crowd images in the ShanghaiTech Part A dataset, performance decreases with increasing the number of layers, but still improves compared to the baseline. Applying a global response normalization layer significantly improves both model stability and accuracy, demonstrating that effectively leveraging multi-channel features can enhance the model's counting performance.

[0065] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A dense crowd counting method based on neural network, characterized in that: include: Step 10: Obtain crowd images and corresponding labeled data, and perform preprocessing operations to obtain a training dataset; Step 20: Use VGG-19 as the backbone network, delete the linear prediction layer and the last maximum pooling layer of VGG-19, and add a feature aggregation layer; Add a global response normalization layer to the feature aggregation layer to obtain a crowd counting model; Among them, the processing of the global response normalization layer is: Among them, G(·) is a normalization function based on the L2 norm, is a standard division normalization function, γ and β are two learnable parameters; is the input feature map, is the feature map after global normalization, where W and H are the width and height of the density map; Step 30: Train the crowd counting model according to the training data set to obtain a trained crowd counting model. The counting loss calculation process based on neighborhood summation guidance used in the training includes steps 31-34: Step 31: Obtain the true density map of the corresponding crowd image through the labeled data of the training dataset Obtaining the predicted density map of a crowd image using a crowd counting model Where W and H are the width and height of the density map; Step 32: D t and D p Perform normalization: Among them, D t and D p Represent the true density map and the predicted density map respectively, ||·||1 represents the L1 norm of a vector, ||D t ||1 and ||D p ||1 represents the real count and the real count respectively, ε is the stability factor, and ε=10 -6 , D' t 、D' p That is, the normalized true density map and predicted density map; Step 33: Calculate the counting loss under neighborhood summation guidance: in, represents neighborhood summation, that is, summing the pixel points in the density map and the pixels around it, ||·||1 represents the mean absolute error, represents the counting loss guided by neighborhood summation; Step 34: By cascading multiple neighborhood summation operations, the pixel points are summed to obtain more neighborhood information. The expanded neighborhood summation is: Where i represents the number of steps of the multi-level neighborhood summation operation. When i = 0, that is, no neighborhood summation operation is performed, D' ti =D' t ,D' pi =D' p , when i>0, n is the total number of steps for neighborhood summation; Step 40: Input the image to be counted into the trained crowd counting model to obtain the predicted density map To D p Perform pixel-by-pixel summation, i.e. Where i∈[0,H-1],j∈[0,W-1] represents the coordinates of the pixel point, H, W represent the height and width of the density map respectively, D p (1,i,j) represents the value of the pixel in the density map, and count is the number of people in the image to be counted.

2. The method according to claim 1, characterized in that Step 10 also includes: Step 11: Scale the crowd image using a bicubic interpolation algorithm to limit the narrow edge of the crowd image to within the acceptable range of the crowd counting model; Step 12: Process the labeled data to obtain a labeled density map of the crowd image; Step 13: Randomly crop the crowd images to a certain size to adapt to the batch training process.

3. The method according to claim 1, characterized in that Step 20 also includes: The obtained crowd counting model includes a feature extraction layer and a feature aggregation layer; the feature extraction layer is a vgg-19 backbone architecture, including a five-order architecture: the first and second orders include two convolutional layers and a maximum pooling layer, the third and fourth orders include four convolutional layers and a pooling layer, and the fifth order includes four convolutional layers; the feature aggregation layer includes: the number of channels extracted by the feature extraction layer is 512 image features, which is reduced to 128 channels after the first convolutional layer, processed by the first GELU activation function and the first global response normalization layer, and then reduced to 32 channels after the second convolutional layer, processed by the second GELU activation function and the second global response normalization layer, and then reduced to 1 channel after the third convolutional layer, and finally output by the ReLU activation function to obtain the final output predicted density map result.

4. The method according to claim 1, wherein Step 30 also includes: During the training process, we used images with a crop size of 512×512 for batch training, and chose Adam as the optimizer for network parameters with a learning rate of 10. -5 , the weight decay is 10 -5 , the number of training cycles is set to 1000.

5. A computer device, characterized in that: include: a memory for storing instructions; A processor, configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that Instructions are stored, and when the instructions are executed by a processor, the method according to any one of claims 1 to 4 is performed.

Images