Crowd counting method based on enhanced feature space alignment fusion network
By designing an enhanced feature space alignment fusion network in crowd counting technology, the problems of scale changes and non-uniform distribution are solved, and higher quality density map generation and counting accuracy are achieved.
Patent Information
- Application Number
- CN202411171031.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-26
AI Technical Summary
There are scale changes and non-uniform distribution problems in population counting technology, resulting in low counting accuracy and difficult to meet the actual application needs.
Design a population counting method based on enhanced feature spatially aligned fusion network. By constructing a feature extraction network, a spatially separable feature enhancement module, a spatially aligned feature fusion module and a density map generation module, the VGG-16 network is used to extract features and perform feature enhancement and aligned to generate high-quality population density maps.
The quality and counting accuracy of the population density map are improved, and the scale changes and non-uniform distribution problems in the population image can be better handled, which improves the counting accuracy.
Smart Images

Figure CN118675117B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a crowd counting method based on an enhanced feature space alignment fusion network. Background Art
[0002] As one of the important problems in the field of computer vision, crowd counting is of vital importance to modern society. Accurate crowd counting can not only provide important support for urban planning, security monitoring and traffic management, but also play a key role in the security and crowd management of various large-scale events. By real-time monitoring and accurate crowd counting, we can better understand the distribution and flow of urban population, and provide a scientific basis for urban development planning and resource allocation. The research and development of crowd counting technology is of great significance to improving the efficiency of urban management and social operation.
[0003] In practical application scenarios, crowd counting research remains challenging due to problems such as target occlusion, scale variation, complex background, and non-uniform distribution. Summary of the invention
[0004] Technical problem solved: In order to cope with the problems of scale change and non-uniform distribution in crowd images, the present invention discloses a crowd counting method based on an enhanced feature space alignment fusion network, which can make full use of the spatial information of feature maps in crowd counting tasks for feature enhancement and feature alignment, and has higher density map quality and higher counting accuracy.
[0005] Technical solution:
[0006] A crowd counting method based on an enhanced feature space alignment fusion network, the crowd counting method comprising the following steps:
[0007] S1: Obtain a crowd image dataset and perform data preprocessing, divide the crowd image dataset into a training set, a validation set, and a test set, and use the annotation map to generate a crowd density map;
[0008] S2: constructing an enhanced feature space alignment fusion network, wherein the enhanced feature space alignment fusion network includes a feature extraction network, three spatially separable feature enhancement modules, two spatially aligned feature fusion modules and a density map generation module;
[0009] The enhanced feature space alignment fusion network uses the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network as the feature extraction network to extract features from the crowd image; the three spatially separable feature enhancement modules respectively enhance the feature maps C1, C2 and C3 output by the third maximum pooling layer, the fourth maximum pooling layer and the 13th convolutional layer to obtain feature maps S1, S2 and S3; one of the spatial alignment feature fusion modules fuses the feature map S2 as a shallow feature and the feature map S3 as a deep feature to output a feature map A1; the other spatial alignment feature fusion module fuses the feature map S1 as a shallow feature and the feature map A1 as a deep feature to output a feature map A2; the density map generation module processes the feature map C2 and the feature map A2, outputs a crowd density map of the same size as the input image, and then integrates the crowd density map to obtain the number of people;
[0010] S3: Use the training set and validation set obtained in step S1 to train the enhanced feature space alignment fusion network to obtain the optimal network model;
[0011] S4: Input the test set obtained in step S1 into the optimal network model obtained in step S3 to obtain the crowd image counting result.
[0012] Step S1 further comprises the following steps:
[0013] S11: Crowd image dataset downloaded from open source dataset website or self-made;
[0014] S12: resizing the images and their annotation images in the crowd image dataset;
[0015] S13: Divide the scaled images and annotated images into a training set, a validation set, and a test set;
[0016] S14: Use Gaussian kernel function to generate the real crowd density map for the labeled image.
[0017] Furthermore, the spatially separable feature enhancement module sends the input feature map into three spatially separable convolution branches for convolution processing, and the convolution kernel sizes of the three spatially separable convolution branches are set to 5, 7, and 9 respectively; then the outputs of the three spatially separable convolution branches are directly added, and the addition results are sent to three channel branches respectively after convolution through a 3×3 convolution layer, and the first channel branch performs maximum pooling, 1×1 convolution, Relu activation, and 1×1 convolution operations in sequence, and the second channel branch performs average pooling, 1×1 convolution, Relu activation, and 1×1 convolution operations in sequence; then, after adding the feature maps output by the first channel branch and the second channel branch, Sigmoid activation is used to obtain the channel attention weight; and then the feature map of the third branch is multiplied by the channel attention weight to obtain the output feature map of the spatially separable feature enhancement module.
[0018] Furthermore, the spatial alignment feature fusion module upsamples the input deep features to make them the same size as the shallow features, and then splices the upsampled results with the shallow features. The spliced features are passed through a 1×1 convolution and a 3×3 convolution with an output channel number of 4, and then the output results are added to the shallow features and the upsampled deep features respectively. The output results are then input into a 3×3 convolution respectively, and then the output features are spatially aligned with feature maps of different depths by interpolation using the registration function R, and the aligned feature maps of different depths are added to obtain preliminary fused features; at the same time, the shallow features are average pooled and maximum pooled respectively, and the two pooled results are spliced and then subjected to a 7×7 convolution operation, and then the Sigmoid activation function is used for activation to generate spatial attention weights, and the preliminary fused features are multiplied by the spatial attention weights to obtain the fused features output by the spatial alignment feature fusion module.
[0019] Furthermore, the density map generation module concatenates the feature map C2 and the feature map A2, and then uses four 3×3 convolutions with gradually decreasing output channel numbers to reduce the channels of the fused feature map, and finally uses a 1×1 convolution with a channel number of 1 to generate a density map.
[0020] Step S3 further comprises the following steps:
[0021] S31: Setting initial parameters of the training model;
[0022] S32: Use L2 norm loss function to calculate the loss, calculate the gradient according to the loss function, and use stochastic gradient descent optimizer to update the network model parameters;
[0023] S33: The enhanced feature space alignment fusion network is evaluated using mean absolute error (MAE) and mean square error (MSE);
[0024] S34: Repeat the training process of steps S32-S33, and use the validation set to evaluate the enhanced feature space alignment fusion network after each training round until all training rounds are completed, and save the optimal network model.
[0025] Step S4 further comprises the following steps:
[0026] S41: Import the optimal network model obtained in step S3, and input the test set images and their real crowd density maps obtained in step S1;
[0027] S42: Calculate the mean absolute error (MAE) and mean square error (MSE) scores, and save the test results.
[0028] Beneficial effects:
[0029] The present invention discloses a crowd counting method based on an enhanced feature space alignment fusion network, designs a spatially separable feature enhancement module, uses a pyramid structure to extract richer multi-scale features, uses spatially separable convolution to increase the receptive field and control the number of parameters of the network model, and proposes a spatially aligned feature fusion module to calculate the offset of each pixel and perform registration during feature fusion to alleviate the misalignment phenomenon caused by the fusion of features at different levels, thereby generating a higher quality crowd density map and improving the counting accuracy of the overall image. The present invention can make full use of the spatial information of the feature map in the crowd counting task to perform feature enhancement and feature alignment, and the crowd density map has higher quality and higher counting accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a flow chart of a crowd counting method based on an enhanced feature space alignment fusion network of the present invention;
[0031] Figure 2 It is a schematic diagram of the enhanced feature space alignment fusion network structure;
[0032] Figure 3 It is a schematic diagram of the structure of the spatially separable feature enhancement module;
[0033] Figure 4 It is a schematic diagram of the structure of the spatial alignment feature fusion module;
[0034] Figure 5 It is a schematic diagram of the structure of the density map generation module;
[0035] Figure 6 It is a schematic diagram of the experimental data of the example. DETAILED DESCRIPTION
[0036] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.
[0037] like Figure 1 As shown, this embodiment provides a crowd counting method based on an enhanced feature space alignment fusion network, and the specific steps are as follows:
[0038] Step S1: Obtain a crowd image dataset and perform data preprocessing, divide the dataset into a training set, a validation set, and a test set, and generate a crowd density map using a labeled map;
[0039] Step S1 specifically includes the following steps:
[0040] S11: Download part A of the ShanghaiTech dataset from the open source dataset website https: / / www.kaggle.com / tthien / shanghaitech;
[0041] S12: Use the resize function in the torchvision.transforms toolkit to scale the 300 training images and 182 test images and their annotations in the downloaded dataset so that the longest side of the image does not exceed 2048 pixels;
[0042] S13: putting the scaled training images and the labeled images into the training set, and putting the scaled test images and the labeled images into the validation set and the test set at the same time;
[0043] S14: The parameter σ of the Gaussian kernel function has different values for different data sets. In the present invention, σ=10 is used to generate a real crowd density map for the labeled map.
[0044] Step S2: construct an enhanced feature space alignment fusion network;
[0045] Enhanced feature space alignment fusion network Figure 2 As shown, step S2 specifically includes the following steps:
[0046] S21: Use the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network as the feature extraction network, where the first 13 convolutional layers of VGG-16 are all 3×3 convolutions, and the number of channels of the convolutional layers is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, respectively; after the second convolutional layer, the fourth convolutional layer, the seventh convolutional layer and the tenth convolutional layer, there is a maximum pooling layer with a step size of 2 and a pooling area size of 2, which are the first maximum pooling layer, the second maximum pooling layer, the third maximum pooling layer and the fourth maximum pooling layer, respectively; C1 represents the feature map output by the third maximum pooling layer, C2 represents the feature map output by the fourth maximum pooling layer, and C3 represents the feature map output by the 13th convolutional layer;
[0047] S22: The feature maps C1, C2, and C3 obtained in step S21 are respectively input into three spatially separable feature enhancement modules SSFEM1, SSFEM2, and SSFEM3 for feature enhancement to obtain feature maps S1, S2, and S3; wherein the structure of the spatially separable feature enhancement module is as follows: Figure 3 As shown in the figure, the input feature map is first sent to three branches of spatially separable convolution (SSC), and the convolution kernel sizes of the spatially separable convolutions of the three branches are set to 5, 7, and 9 respectively. Then, the outputs of the three branches are directly added together, and then a 3×3 convolution is performed. Then, the output feature map is sent to the three branches respectively. The first branch performs maximum pooling, 1×1 convolution, Relu activation, and 1×1 convolution operations in sequence, and the second branch performs average pooling, 1×1 convolution, Relu activation, and 1×1 convolution operations in sequence. Then, after adding the feature maps output by the first branch and the second branch, Sigmoid activation is used to obtain the channel attention weight, and then the feature map of the third branch is multiplied by the channel attention weight to obtain the output feature map of the spatially separable feature enhancement module;
[0048] S23: Input the feature maps S2 (as shallow features) and S3 (as deep features) obtained in step S22 into the spatial alignment feature fusion module SAFFM1 for feature fusion, and output feature map A1; input the feature maps S1 (as shallow features) and A1 (as deep features) into the spatial alignment feature fusion module SAFFM2 for feature fusion, and output feature map A2; wherein the structure of the spatial alignment feature fusion module is as follows: Figure 4 As shown in the figure, the deep features are upsampled to make their size the same as the shallow features, and then the upsampled results are concatenated with the shallow features. The concatenated features are passed through a 1×1 convolution and a 3×3 convolution with an output channel number of 4. The output results are then added to the shallow features and the upsampled deep features respectively, and the output results are respectively input into a 3×3 convolution. The output features are then spatially aligned with the feature maps of different depths by interpolation using the registration function R, and the aligned feature maps of different depths are added to obtain the preliminary fused features. In addition, the shallow features are average pooled and maximum pooled respectively, and the two pooled results are concatenated and then subjected to a 7×7 convolution operation. The Sigmoid activation function is then used for activation to generate the spatial attention weights, and the preliminary fused features are multiplied by the spatial attention weights to obtain the fused features output by the spatial alignment feature fusion module.
[0049] S24: Input the feature map C2 obtained in step S21 and the feature map A2 obtained in step S23 into the density map generation module, output a crowd density map of the same size as the input image, and then perform an integral operation on the crowd density map to obtain the number of people; wherein the density map generation module structure is as follows: Figure 5As shown in the figure, the feature map C2 is first concatenated with the feature map A2, and then four 3×3 convolutions with gradually decreasing output channel numbers are used to reduce the channels of the fused feature map, and finally a 1×1 convolution with a channel number of 1 is used to generate the density map.
[0050] Step S3: Use the training set and validation set obtained in step S1 to train the enhanced feature space alignment fusion network to obtain the optimal network model;
[0051] Step S3 specifically includes the following steps:
[0052] S31: Set the initial parameters of the training model as follows;
[0053] Initial learning rate: 0.0000001
[0054] Optimizer: SGD optimizer;
[0055] Momentum: 0.95
[0056] Weight decay: 0.00001
[0057] Batch size: 4
[0058] Number of training rounds (epoch): 500;
[0059] Data augmentation: The input image is randomly cropped into 512×512 image patches, and horizontally flipped with a probability of 0.5, the RGB channels of the color image are swapped with a probability of 0.2, and Gaussian noise with a mean of 0 and a standard deviation of 5 is added with a probability of 0.1; for datasets containing gray images, the color image is randomly converted to a grayscale image with a probability of 0.1;
[0060] And the convolutional layers of the model before training all use normal distribution to initialize parameters, where: , ;
[0061] S32: Use the L2 norm loss function to calculate the loss during training. The loss function formula is:
[0062] ;
[0063] Where N represents the number of training images. is a set of learnable network weights, Display pictures The predicted density map of Display pictures The true density map of . The gradient is calculated according to the loss function, and the stochastic gradient descent (SGD) optimizer is used to update the network model parameters;
[0064] S33: The model is evaluated using mean absolute error (MAE) and mean square error (MSE). The calculation methods of MAE and MSE are as follows:
[0065]
[0066]
[0067] Where N is the number of input images, is the predicted count, is the true count. The lower the values of MAE and MSE, the better the counting performance of the model.
[0068] S34: Repeat the training process of steps S32-S33, use the validation set to evaluate the network model after each training round, save the network parameters according to the optimal MAE value, until 500 training rounds are completed, and save the optimal network model.
[0069] Step S4: Input the test set obtained in step S1 into the optimal network model obtained in step S3 to obtain the crowd image counting result.
[0070] Step S4 specifically includes the following steps:
[0071] S41: Import the optimal network model obtained in step S3, and input the test set images and their real crowd density maps obtained in step S1;
[0072] S42: Calculate MAE and MSE scores and save the test results.
[0073] Examples
[0074] This example uses the test set of Part A of the ShanghaiTech crowd counting dataset for experimental testing. Part A of the ShanghaiTech dataset includes 482 images collected from the Internet, including 300 training sets and 182 test sets. The average resolution is 589×868, and each image contains crowd counts ranging from 33 to 3139. Part A presents a crowded crowd scene, which is of great significance for studying dense crowd counting.
[0075] The experimental results are as follows Figure 6 As shown in the figure, the mean absolute error (MAE) and mean square error (MSE) obtained by the method of the present invention are better than those of the existing mainstream methods.
[0076] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.
Claims
1. A crowd counting method based on enhanced feature space alignment fusion network, characterized in that: The crowd counting method comprises the following steps: S1: Obtain a crowd image dataset and perform data preprocessing, divide the crowd image dataset into a training set, a validation set, and a test set, and use the annotation map to generate a crowd density map; S2: constructing an enhanced feature space alignment fusion network, wherein the enhanced feature space alignment fusion network includes a feature extraction network, three spatially separable feature enhancement modules, two spatially aligned feature fusion modules and a density map generation module; The enhanced feature space alignment fusion network uses the first 13 convolutional layers and the first 4 pooling layers of the VGG-16 network as the feature extraction network to extract features from the crowd image; the three spatially separable feature enhancement modules respectively enhance the feature maps C1, C2 and C3 output by the third maximum pooling layer, the fourth maximum pooling layer and the 13th convolutional layer to obtain feature maps S1, S2 and S3; one of the spatial alignment feature fusion modules fuses the feature map S2 as a shallow feature and the feature map S3 as a deep feature to output a feature map A1; the other spatial alignment feature fusion module fuses the feature map S1 as a shallow feature and the feature map A1 as a deep feature to output a feature map A2; the density map generation module processes the feature map C2 and the feature map A2, outputs a crowd density map of the same size as the input image, and then integrates the crowd density map to obtain the number of people; S3: Use the training set and validation set obtained in step S1 to train the enhanced feature space alignment fusion network to obtain the optimal network model; S4: input the test set obtained in step S1 into the optimal network model obtained in step S3 to obtain the crowd image counting result; The spatially separable feature enhancement module sends the input feature map to three spatially separable convolution branches for convolution processing, and the convolution kernel sizes of the three spatially separable convolution branches are set to 5, 7, and 9 respectively; then the outputs of the three spatially separable convolution branches are directly added, and the addition results are sent to three channel branches respectively after convolution through a 3×3 convolution layer, and the first channel branch performs maximum pooling, 1×1 convolution, Relu activation, and 1×1 convolution operations in sequence, and the second channel branch performs average pooling, 1×1 convolution, Relu activation, and 1×1 convolution operations in sequence; then the feature maps output by the first channel branch and the second channel branch are added, and the channel attention weight is obtained using Sigmoid activation; then the feature map of the third branch is multiplied by the channel attention weight to obtain the output feature map of the spatially separable feature enhancement module; The spatial alignment feature fusion module upsamples the input deep features to make them the same size as the shallow features, and then splices the upsampled results with the shallow features. The spliced features are passed through a 1×1 convolution and a 3×3 convolution with an output channel number of 4, and then the output results are added to the shallow features and the upsampled deep features respectively. The output results are then input into a 3×3 convolution respectively, and then the output features are spatially aligned with feature maps of different depths by interpolation using the registration function R, and the aligned feature maps of different depths are added to obtain preliminary fused features; at the same time, the shallow features are average pooled and maximum pooled respectively, and the two pooled results are spliced and then subjected to a 7×7 convolution operation, and then the Sigmoid activation function is used for activation to generate spatial attention weights, and the preliminary fused features are multiplied by the spatial attention weights to obtain the fused features output by the spatial alignment feature fusion module.
2. The crowd counting method based on enhanced feature space alignment fusion network according to claim 1 is characterized in that: Step S1 further comprises the following steps: S11: Crowd image dataset downloaded from open source dataset website or self-made; S12: resizing the images and their annotation images in the crowd image dataset; S13: Divide the scaled images and annotated images into a training set, a validation set, and a test set; S14: Use Gaussian kernel function to generate the real crowd density map for the labeled image.
3. The crowd counting method based on enhanced feature space alignment fusion network according to claim 1 is characterized in that: The density map generation module concatenates the feature map C2 and the feature map A2, and then uses four 3×3 convolutions with gradually decreasing output channel numbers to reduce the channels of the fused feature map, and finally uses a 1×1 convolution with a channel number of 1 to generate a density map.
4. The crowd counting method based on enhanced feature space alignment fusion network according to claim 1 is characterized in that: Step S3 further comprises the following steps: S31: Setting initial parameters of the training model; S32: Use L2 norm loss function to calculate the loss, calculate the gradient according to the loss function, and use stochastic gradient descent optimizer to update the network model parameters; S33: The enhanced feature space alignment fusion network is evaluated using mean absolute error (MAE) and mean square error (MSE); S34: Repeat the training process of steps S32-S33, and use the validation set to evaluate the enhanced feature space alignment fusion network after each training round until all training rounds are completed, and save the optimal network model.
5. The crowd counting method based on enhanced feature space alignment fusion network according to claim 1 is characterized in that: Step S4 further comprises the following steps: S41: import the optimal network model obtained in step S3, and input the test set images and their real crowd density maps obtained in step S1; S42: Calculate the mean absolute error (MAE) and mean square error (MSE) scores, and save the test results.
