Dense crowd counting method combining high-resolution CNN and lightweight transformer

Abstract

The application provides a dense crowd counting method combining a high-resolution CNN and a lightweight Transformer, and comprises the following steps: the scale size of a human head in a crowd image is calculated by using a fixed Gaussian kernel method to generate a supervised density map for network training; a crowd counting network based on a high-resolution feature extraction network HRNet and a lightweight Transformer is constructed; data augmentation is performed on a crowd data set, the constructed counting network is trained by using a training set, and an optimal model is screened and saved; the optimal network model obtained is tested by using a test set, and the final counting result of the picture crowd is obtained by accumulating and summing the pixel values of the network predicted density map. The application can not only maintain high-resolution output of crowd features, but also can fuse multi-scale information, improve the robustness of crowd counting, and significantly improve the convergence speed and generalization performance of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crowd counting, in particular to a dense crowd counting method combining high-resolution CNN and lightweight Transformer. BACKGROUND

[0002] Crowd counting aims to obtain the number of people in an image through computer vision technology. This technology has important academic research value and application value, and is widely used in public security, intelligent monitoring, epidemic prevention and control and other fields. It can also be extended to related fields such as cell microscopy and vehicle counting. At present, many crowd counting methods based on deep learning are being proposed. In particular, the density estimation method based on deep neural network. This method uses a carefully constructed network model to regress the density map, and then obtains the total number of people by integrating the predicted density map.

[0003] (I) Analysis of patent technologies related to crowd counting

[0004] The Chinese invention patent application with application number 201911161705.0, entitled "Crowd density estimation method", uses AlexNet network to divide the crowd picture data set into dense and sparse two categories according to the preset threshold, and then sends the two types of image density features into the corresponding feature extraction network respectively, so as to obtain better effective crowd density estimation features. However, this method needs to train AlexNet classification network, dense data feature extraction network and sparse data feature extraction network respectively, resulting in a complicated and complex calculation process, and the classification selection error of the picture data will seriously affect the counting accuracy.

[0005] The Chinese invention patent application with application number 202010170236.5, entitled "Dense crowd counting algorithm based on cascaded high-resolution convolutional neural network", uses cascaded high-resolution convolutional neural network CHRNet to extract high-resolution features of dense crowd images, and adopts a regional loss weighting method to optimize network parameters by using two loss functions of MSE and counting error. The disadvantage of this method is that the weights between different regions and the two different losses are set artificially, and the optimization quality of the density map is sensitive to the weight setting, and the counting result will deviate greatly due to improper weight setting.

[0006] The Chinese invention patent application with application number 202211557637.1, entitled "Crowd counting method and system based on multi-scale fusion convolutional network", proposes to use a multi-scale fusion convolutional network after the VGG16 primary feature extraction network to effectively extract multi-scale information. However, the backbone network of this method adopts a single column structure, which makes it difficult to realize the extraction and fusion of multi-level features, and the fusion method of the multi-scale fusion convolutional network is relatively simple, which is not conducive to effective modeling of high complexity image scenes.

[0007] (II) Analysis of Population Counting Research Based on Deep Neural Networks

[0008] Li et al. proposed a counting network called CSRNet based on the VGG-16 architecture (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1091-1100). This network expands its receptive range and feature extraction capabilities by adding dilated convolutional layers in the backend network. However, CSRNet contains a large number of pooling operations and convolutional layers with strides greater than 1, resulting in a density map size that is only 1 / 8 of the original input size. This may not be able to contain enough small-scale target feature information, thus limiting its performance in some complex scenes. For dense crowds or small-scale targets, low-resolution features are not conducive to accurate prediction.

[0009] Gao et al. introduced spatial and channel attention mechanisms into their proposed counting network SCAR (Neurocomputing, Vol. 363, pp. 1-8, 2019). Spatial attention is used to encode pixel-level contextual information for the entire image, improving the model's accuracy in predicting density maps at the pixel level; channel attention is used to extract different feature information, making the model more robust to noisy backgrounds. While these two attention mechanisms in the model can better focus on local details, they struggle to capture global features for global context modeling, thus affecting the model's understanding of the entire scene.

[0010] Liang et al. proposed TransCrowd, a crowd counting network based on Vision Transformer (ViT). (Science China)

[0011] The journal *Information Sciences*, 2022, Vol. 6, pp. 104-120, successfully introduced the Transformer into the field of crowd counting. This model uses ViT as the backbone network for global context modeling and achieves relatively ideal counting results with weak supervision. However, pure ViT-based counting models often have high computational costs, making them difficult to train and optimize, and hindering their deployment on mobile devices. Summary of the Invention

[0012] The present invention aims to overcome the problems of insufficient multi-scale feature fusion, low feature resolution, and difficulty in training networks in the aforementioned prior art, and provides a method that can maintain high resolution of crowd features and improve crowd prediction accuracy.

[0013] This invention employs an improved high-resolution feature extraction network HRNet as the front-end network, maintaining its feature map output size at 1 / 4 of the original input size to generate rich high-resolution representations. This helps to preserve the richness of receptive field information and further improves the accuracy of the predicted density map. A lightweight ViT Encoder is used to model complex global contexts, and multi-scale feature enhancement is used to strengthen feature extraction, effectively mitigating the impact of multi-scale variations, perspective effects, and other factors on the counting results.

[0014] To achieve the above objectives, this invention provides a method for dense crowd counting that combines a high-resolution CNN and a lightweight Transformer, comprising the following steps:

[0015] Step S1: Calculate the scale of human heads in the crowd image using the fixed Gaussian kernel method to generate a supervised density map for network training;

[0016] Step S2: Construct a crowd counting network based on the high-resolution feature extraction network HRNet and the lightweight Transformer;

[0017] Step S3: Augment the population dataset, train the counting network constructed in step S2 using the training set with multi-density graphs, and select and save the optimal model;

[0018] Step S4: Test the optimal network model obtained in step S3 using the test set, and obtain the final count result of the crowd in the image by summing the pixel values ​​of the density map predicted by the network.

[0019] Furthermore, the specific process of step S2 is as follows:

[0020] Step S2-1: Use the feature extraction modules Stage1-4 of the pre-trained high-resolution feature extraction network HRNet to extract primary features from the input image, resulting in four primary feature maps with different resolutions and numbers of channels. Further, use convolutional layers with a kernel size of 1×1 to extract features from the three primary feature maps with the highest resolution, resulting in three primary density maps of different resolutions: density map1, density map2, and density map3. These three primary density maps have 1 channel, and their vertical height and horizontal width are 1 / 4, 1 / 8, and 1 / 16 of the original input scale, respectively.

[0021] Step S2-2: Construct a multi-attention module. Based on four primary feature maps of different resolutions, perform various attention operations on each of them and fuse them according to the channel connection method to form a new attention feature map.

[0022] Step S2-3: Construct a parallel multi-scale enhancement module and a ViT Encoder feature extraction module, and use the parallel ViT encoder and multi-scale enhancement module to perform global context modeling and multi-scale feature enhancement on the fused attention feature map;

[0023] Step S2-4: Construct a decoder module. Input the feature map after feature enhancement and global context modeling into the decoder module for decoding, so as to restore the feature map size to the original input size and predict the final prediction density map (density map4).

[0024] Furthermore, the specific process of step S2-2 is as follows:

[0025] Step S2-2-1: Feed the four primary feature maps into the self-attention and channel attention sub-modules, and generate a self-attention feature sub-map and a channel attention feature sub-map for each primary feature map;

[0026] Step S2-2-2: Use a dynamic weight generation mechanism (a network consisting of convolutional layers and a sigmoid activation function) to calculate weights for the two attention feature subgraphs;

[0027] Step S2-2-3: Add the two output weights to obtain the total weight. The sum of the weights is used to normalize the two attention weight outputs.

[0028] Step S2-2-4: Multiply the pre-generated self-attention feature map and channel attention feature map by their normalized weights respectively, and then add them together to form a weighted attention feature map;

[0029] Step S2-2-5: Using the attention feature map with the highest resolution as a benchmark, the attention feature maps output by the other three branches are upsampled using the nearest neighbor interpolation method, and then fused into an attention feature map containing 480 channels by channel connection.

[0030] Furthermore, the specific process of steps S2-3 is as follows:

[0031] Step S2-3-1: Input the attention feature map obtained in step S2-2 into the dilated convolution submodule in the ViT encoder and the multi-scale enhancement module respectively, and output two intermediate feature maps;

[0032] Step S2-3-2: Use a convolutional layer to make the channels of the feature map output by the ViT encoder and the channels of the feature map output by the dilated convolutional module consistent, and add the two intermediate feature maps with the same channels together by element-wise addition;

[0033] Step S2-3-3: The feature map fused in step S2-3-2 is fed into the deformable convolutional submodule in the multi-scale enhancement module to obtain a further enhanced feature map;

[0034] Step S2-3-4: Add the feature map output by the ViT encoder and after channel conversion to the enhanced feature map obtained in step S2-3-3 by element-wise addition to obtain a further fused feature map.

[0035] Furthermore, step S3 augments the population dataset and performs multi-density map-supervised training, as detailed below:

[0036] Step S3-1: To enhance the training data, random cropping and horizontal flipping are used, where the cropping size is 256×256 and the flipping probability is 0.5;

[0037] Step S3-2: Perform multi-density map supervised training by calculating the weighted sum of the losses between the final predicted density map (density map4) and the predicted primary density maps (density map1, density map2, density map3) and their GT density maps, in order to enhance the robustness of the intermediate feature maps and thus improve the accuracy of the final density map regression.

[0038] Step S3-3: Filter and save the optimal model.

[0039] Compared with the prior art, the advantages of this invention are as follows: (1) This invention uses the improved high-resolution feature extraction network HRNet as the backbone network, which can not only maintain the high-resolution output of crowd features, but also integrate multi-scale information, thereby making the predicted feature map more accurate in space, especially for dense crowds or small-scale targets. (2) This invention uses a lightweight ViT Encoder to model complex global contexts and combines multiple attention operations (self-attention, channel attention) to balance global information and local details of feature maps, helping the model to better distinguish different crowd regions and complex backgrounds, reducing interference from occlusion, background and perspective problems, and further improving the robustness of crowd counting. (3) This invention constructs a simple and efficient multi-scale feature enhancement module in the CNN branch, which effectively compensates for the feature details that may be lost by the backbone network during the sampling operation, and solves the problem of low counting accuracy caused by multi-scale problems to a certain extent. (4) This invention adopts a multi-density graph supervised training strategy to optimize network parameters and fully gather feature information from different layers and different resolutions of the network for feature interaction. By leveraging the correlation between density maps at different resolutions, the model can better learn the distribution of people in a scene, significantly improving the convergence speed and generalization performance. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the process in this area;

[0041] Figure 2 Here is the flowchart for HRNet Stage 1-4 modules;

[0042] Figure 3 Flowchart of the Multiple Attention Module (MAM);

[0043] Figure 4 This is a flowchart of the multi-scale enhancement module. Detailed Implementation

[0044] like Figure 1 As shown in the figure, this embodiment proposes a dense crowd counting method that combines high-resolution CNN and lightweight Transformer, which includes the following steps:

[0045] Step S1: Using a fixed Gaussian kernel to determine the scale of human heads in a crowd image, a supervised density map for network training is generated. This involves convolving a Gaussian kernel with a size of 15 with the coordinates of the labeled human heads in the image. Through this process, the size of the Gaussian region corresponding to each human head is approximately equal to the actual size of the human head, and the sum of the pixel values ​​of a single human head region equals 1.

[0046] Step S2: Construct a crowd counting network based on the high-resolution feature extraction network HRNet and the lightweight Transformer;

[0047] Step S3: Augment the population counting dataset, and use the training set to train the counting network constructed in Step S2 under multi-density graph supervision, and select the optimal model.

[0048] Step S4: Test the optimal network model obtained in step S3 using the test set, and obtain the final count result of the crowd in the image by summing the pixel values ​​of the density map predicted by the network.

[0049] In this embodiment, the specific process of step S2 is as follows:

[0050] Step S2-1: Utilize the Stage 1-4 feature extraction module under the pre-trained high-resolution feature extraction network HRNet to extract primary features from the input image (H, W, 3), resulting in four primary feature maps with decreasing resolution and increasing channel number. The scales of these four primary feature maps are (H / 4, W / 4, 32), (H / 8, W / 8, 64), (H / 16, W / 16, 128), and (H / 32, W / 32, 256), respectively. Here, H and W represent the vertical height and horizontal width of the input image, respectively, and the values ​​3, 32, 64, 128, and 256 represent the number of channels. The operation flow of the Stage 1-4 feature extraction module is as follows: Figure 2 As shown. Further feature extraction is performed on the three primary feature maps with the highest resolution using convolutional layers of size 1×1, resulting in three primary density maps of different resolutions: density map1, density map2, and density map3. (See diagram below.) Figure 1 As shown in the dashed lines, these three primary density maps have 1 channel, and their vertical height and horizontal width are 1 / 4, 1 / 8, and 1 / 16 of the original input scale, respectively.

[0051] Step S2-2: Construct a multi-attention module. Based on four primary feature maps of different resolutions, perform various attention operations on each map and fuse them according to the channel connection method to form a new attention feature map. The operation process is as follows: Figure 3 As shown;

[0052] Step S2-3: Construct a parallel multi-scale enhancement module and a ViT Encoder feature extraction module, and use the parallel ViT encoder and multi-scale enhancement module to perform global context modeling and multi-scale feature enhancement on the fused attention feature map;

[0053] Step S2-4: Construct a decoder module. The feature map after feature enhancement and global context modeling is sent to the decoder module for decoding to restore the feature map size to the original input size and obtain the final prediction density map densitymap4(H, W, 1).

[0054] In this embodiment, the specific process of step S2-2 is as follows:

[0055] Step S2-2-1: Feed the four primary feature maps into the self-attention and channel attention submodules, each primary feature map F a Generate self-attention feature sub-maps F respectively s and channel attention feature subgraph F c ;

[0056] Step S2-2-2: Calculate weights M1 and M2 for these two attention feature subgraphs using a dynamic weight generation network. The dynamic weight generation network consists of a single 3×3 kernel convolutional layer and a sigmoid activation function.

[0057] Step S2-2-3: Add the two output weights to obtain the total weight M3. The sum of the weights is used to normalize the two attention weight outputs.

[0058] Step S2-2-4: Use the pre-generated self-attention feature submap F s and channel attention feature subgraph F c Multiplying each of these by their normalized weights yields the feature map F'. s =M1 / M3☉F s , F' c =M2 / M3☉F c Where ☉ represents element-wise multiplication. These are then summed to form a weighted attention feature map F. Y =F' s +F' c ;

[0059] Step S2-2-5: Using the attention feature map with the highest resolution as a benchmark, the attention feature maps output by the other three branches are upsampled using the nearest neighbor interpolation method, and then fused into an attention feature map containing 480 channels by channel connection.

[0060] In this embodiment, the specific process of steps S2-3 is as follows:

[0061] Step S2-3-1: Input the attention feature map obtained in step S2-2 into the ViT Encoder and the dilated convolutional submodule (ACM) in the multi-scale enhancement module, respectively, and output two intermediate feature maps. For example... Figure 4 As shown, the dilated convolutional submodule (ACM) consists of three parallel branches, each containing ordinary convolutional layers with dilation rates of 1, 2, and 4. The purpose is to further expand the receptive field and fuse features at different scales.

[0062] Step S2-3-2: Use a convolutional layer to make the channels of the feature map output by the ViT encoder and the channels of the feature map output by the dilated convolutional module consistent, and add the two intermediate feature maps with the same channels together by element-wise addition;

[0063] Step S2-3-3: The feature map fused in step S2-3-2 is fed into the deformable convolutional submodule DCM in the multi-scale enhancement module to obtain a further enhanced feature map. In this submodule, the parameter settings are similar to the ACM module, employing a three-branch deformable convolutional group design. These three sub-branches process the input feature map in parallel, enabling the model to capture richer geometric and structural information at different feature levels. Additionally, channel attention is performed after the last deformable convolutional layer of each branch to improve counting accuracy. Specifically, the DCM submodule uses feature F... a As input, features are captured by sequentially passing through three sets of deformable convolutional layers of different scales. Then, The data is fed into the AFS network to generate attention weights. AFS consists of an average pooling layer, a fully connected layer, and a sigmoid activation function. Following this, in a pre-generated... and weight Perform multiplication between them to obtain the characteristics Where ☉ represents element-wise multiplication. This is then aggregated into multi-scale features. in, Indicates a channel splicing operation;

[0064] Step S2-3-4: Add the feature map output by the ViT encoder and after channel conversion to the enhanced feature map obtained in step S2-3-3 by element-wise addition to obtain a further fused feature map.

[0065] In this embodiment, step S3 involves data augmentation of the crowd counting data and supervised training using multi-density maps. The specific process is as follows:

[0066] Step S3-1: To enhance the training data, random cropping and horizontal flipping are used, where the cropping size is 256×256 and the flipping probability is 0.5;

[0067] Step S3-2: Perform multi-density map supervised training by calculating the sum of weighted losses between the final predicted density map (density map4) and the predicted primary density maps (density map1, density map2, density map3) and their ground truth density maps. This enhances the robustness of intermediate feature maps and improves the accuracy of the final density regression. The weights are set to 0.3, 0.2, 0.15, and 0.1, respectively.

[0068] Step S3-3: Select and save the best model. Based on the lowest mean absolute error (MAE) during training, save the model with the best performance.

Claims

1. A dense crowd counting method combining high-resolution CNN and lightweight Transformer, characterized in that, Includes the following steps: Step S1: Calculate the scale of human heads in the crowd image using the fixed Gaussian kernel method to generate a supervised density map for network training; Step S2: Construct a crowd counting network based on the high-resolution feature extraction network HRNet and the lightweight Transformer; The specific process is as follows: Step S2-1: Use the feature extraction modules Stage1-4 of the pre-trained high-resolution feature extraction network HRNet to extract the primary features of the input image, resulting in four primary feature maps with different resolutions and numbers of channels. Use convolutional layers with a kernel size of 1×1 to extract features from the three primary feature maps with the highest resolution, resulting in three primary density maps of different resolution sizes: density map1, density map2, and density map3. These three primary density maps have 1 channel, and their vertical height and horizontal width are 1 / 4, 1 / 8, and 1 / 16 of the original input scale, respectively. Step S2-2: Construct a multi-attention module. Based on four primary feature maps of different resolutions, perform various attention operations on each of them and fuse them according to the channel connection method to form a new attention feature map. Step S2-3: Construct a parallel multi-scale enhancement module and a ViT Encoder feature extraction module, and use the parallel ViT encoder and multi-scale enhancement module to perform global context modeling and multi-scale feature enhancement on the fused attention feature map; Step S2-4: Construct a decoder module, and feed the feature map after feature enhancement and global context modeling into the decoder module for decoding, so as to restore the feature map size to the original input size and predict the final prediction density map densitymap4. Step S3: Augment the population dataset, train the counting network constructed in step S2 using the training set with multi-density graphs, and select and save the optimal model; Step S4: Test the optimal network model obtained in step S3 using the test set, and obtain the final count result of the crowd in the image by summing the pixel values ​​of the density map predicted by the network.

2. The dense crowd counting method combining high-resolution CNN and lightweight Transformer according to claim 1, characterized in that, The specific process of step S2-2 is as follows: Step S2-2-1: Feed the four primary feature maps into the self-attention and channel attention sub-modules, and generate a self-attention feature sub-map and a channel attention feature sub-map for each primary feature map; Step S2-2-2: Calculate weights for these two attention feature subgraphs using a dynamic weight generation mechanism; Step S2-2-3: Add the two output weights to obtain the total weight. The sum of the weights is used to normalize the two attention weight outputs. Step S2-2-4: Multiply the pre-generated self-attention feature map and channel attention feature map by their normalized weights respectively, and then add them together to form a weighted attention feature map; Step S2-2-5: Using the attention feature map with the highest resolution as a benchmark, the attention feature maps output by the other three branches are upsampled using the nearest neighbor interpolation method, and then fused into an attention feature map containing 480 channels by channel connection.

3. The dense crowd counting method combining high-resolution CNN and lightweight Transformer according to claim 2, characterized in that, The specific process of step S2-3: Step S2-3-1: Input the attention feature map obtained in step S2-2 into the ViT encoder and the dilated convolution submodule in the multi-scale enhancement module respectively, and output two intermediate feature maps; Step S2-3-2: Use a convolutional layer to make the channels of the feature map output by the ViT encoder and the channels of the feature map output by the dilated convolutional module consistent, and add the two intermediate feature maps with the same channels together by element-wise addition; Step S2-3-3: The feature map fused in step S2-3-2 is fed into the deformable convolutional submodule in the multi-scale enhancement module to obtain a further enhanced feature map; Step S2-3-4: Add the feature map output by the ViT encoder and after channel conversion to the enhanced feature map obtained in step S2-3-3 by element-wise addition to obtain a further fused feature map.

4. The dense crowd counting method combining high-resolution CNN and lightweight Transformer according to claim 3, characterized in that, Step S3 involves augmenting the population dataset and performing multi-density map-supervised training. The specific process is as follows: Step S3-1: To enhance the training data, random cropping and horizontal flipping are used, where the cropping size is 256×256 and the flipping probability is 0.5; Step S3-2: Perform multi-density map supervised training by calculating the weighted sum of the losses between the final predicted density map density map4 and the predicted primary density maps density map1, density map2, and density map3 and their GT density maps, in order to enhance the robustness of the intermediate feature maps and thus improve the accuracy of the final density map regression. Step S3-3: Filter and save the optimal model.

Images