A crowd counting method combining layer aggregation mechanism

Abstract

The present invention discloses a crowd counting method combining layer aggregation mechanism, belonging to the field of computer vision, mainly including the following steps: First, preprocess the pictures in the dataset to obtain image patches for network training; then use the VGG16 backbone network to extract the low-level features of the images and reduce the number of feature channels to reduce the network complexity; then use 5 cascaded LAMs to obtain rich multi-scale information and further optimize the features through the channel attention module; finally, use the refined features obtained previously by the backend network to generate a crowd density map. The total number of people in the image can be obtained by adding up the pixels in the density map. The method of the present invention solves the crowd counting difficulties such as crowd scale change to a certain extent. Experiments show that in various crowd density scenarios and environments, the counting accuracy of the present invention is better than some existing known methods and can be applied to crowd counting and density estimation in various scenarios such as crowded and sparse scenes.

Description

Technical Field

[0001] The present invention relates to the crowd counting technology in the field of computer vision, and particularly to a crowd counting method combining a layer aggregation mechanism. Background Art

[0002] Crowd counting aims to estimate the number of people and the density distribution in pictures. This research has wide applications in video surveillance, traffic control, public safety, urban planning, sports events, epidemic prevention and control, cell counting, etc. Therefore, in recent years, crowd counting has become a research hotspot in the field of computer vision. However, due to the drastic scale variations of people in pictures, accurately estimating the number of people and the density distribution remains a challenging problem.

[0003] Traditional methods attempt to directly estimate the number of people through detection and regression, but they are all easily interfered by the occlusion problem in the crowd, with poor counting performance and difficult to meet the requirements for counting accuracy in practical applications. The method based on convolutional neural network (CNN) learns the mapping between the features in the image and the crowd density distribution, and directly regresses and outputs a density map that can reflect the crowd spatial distribution information. The sum of the pixels in the density map is the total number of people in the picture.

[0004] Although encouraging counting performance has been achieved, these methods are still difficult to solve the problem of the scale variation of human heads in images. Many previous researchers used multi-column convolutional nerves or multi-branch structures with different convolutional kernel sizes to extract multi-scale information. There is literature proposing to use multi-column sub-networks with different convolutional kernel sizes in a multi-column convolutional neural network (MCNN) to extract features with different receptive fields. Similarly, there is literature proposing the Switch-CNN that uses a classifier to classify the image into different density levels according to the crowd density in the picture, and then sends it into different networks to regress the density map. In addition, the prior art also proposes a network (SANet) that fuses the features extracted by multi-branch convolutional kernels. This network extracts features with different receptive fields by designing branches with different numbers of convolutional kernels and convolutional kernel sizes, and finally splices them on the channel to obtain multi-scale features.

[0005] The multi-column or multi-branch network structure requires a large number of parameters and computing resources, and has limited ability to capture multi-scale information. Dilated convolution has fewer parameters when obtaining the same receptive field size as ordinary convolution. The prior art proposes a single-column network combining dilated convolution to focus on different receptive fields and capture multi-scale features, namely the CSRNet for high-density crowded crowd counting. There is also literature proposing a multi-level density map network (MLCNN), which combines certain specific layers in the network to obtain information focusing on different scales, and respectively outputs 3 intermediate density maps, and then fuses them into the final crowd density map.

[0006] The method based on the visual attention mechanism enables the neural network to selectively focus on more useful information to improve the model performance. Some literature has proposed an attention module (CBAM), which can generate attention weights in the channel and spatial dimensions to adjust the feature map according to the importance of information.

[0007] The present invention aims to solve the crowd counting problem by combining the layer aggregation mechanism and proposes a feasible implementation method. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a crowd counting method combining the layer aggregation mechanism to solve the problems of redundancy in multi-column and multi-branch crowd counting networks and insufficient multi-scale information extraction ability.

[0009] The following technical measures are adopted to solve the above technical problems: A crowd counting method combining the layer aggregation mechanism mainly includes the following steps:

[0010] (1) Image preprocessing: First, download the public dataset from the network, and adjust the maximum side of the image with too large resolution to 1024 pixels without changing the aspect ratio. Four non-overlapping image patches with a size of one-fourth of the original are obtained by cropping, and another four image patches are obtained by random cropping, and horizontal flipping is adopted to make full use of the limited image samples. Finally, histogram equalization is performed on all the obtained image patches to enhance the image. Figure 4 (2) Low-level feature extraction of the image: First, load the pre-trained parameters of the first 10 convolutional layers of VGG16 as the backbone network, and its specific structure is as

[0011] (3) Extraction of multi-scale features: Cascade 5 layer aggregation mechanisms (LAMs). The LAM aggregates layers with different receptive fields and semantic levels to obtain rich multi-scale information for density map regression. The specific structure of the LAM is as Figure 1 shown. Input the image patch into the backbone network to extract the low-level feature f v with 512 channels. After being processed by 3 max-pooling layers (MaxPooling) in VGG16, the feature map is downsampled to one-eighth of the original image size to reduce the computational complexity. Then, a 1×1 convolution with a ReLU activation function, an input channel number of 512, an output channel number of 128, a kernel size of 1, and a stride of 1 is used to reduce the channel number of the feature map, obtaining the feature map f l to further reduce the network parameters and computational amount, and its specific structure is as Figure 1 shown.

[0012] (3) Extraction of multi-scale features: Cascade 5 layer aggregation mechanisms (LAMs). The LAM aggregates layers with different receptive fields and semantic levels to obtain rich multi-scale information for density map regression. The specific structure of the LAM is as Figure 2As shown in the figure, it mainly includes the step-by-step aggregation of the convolutional layer, the iterative fusion of aggregated features, and the channel information attention module. The specific structure is as follows:

[0013] Step-by-step aggregation of the convolutional layer: There are 3 convolutional layers in LAM, the number of output channels is 128 for all, the kernel size is 3×3 for all, and the dilation rates are 1, 2, and 3 which are relatively prime to each other. First, the input feature map f l is sent into the first convolution c1. Then, the input of this convolution and the output activated by the ReLU function are concatenated on the channel dimension to obtain the aggregated feature map b1. This is one aggregation operation. Then, b1 is sent into the second convolution c2 for the second aggregation operation to obtain the aggregated feature map b2, and b2 is sent into the third convolution c3 for the third aggregation operation to obtain the aggregated feature map b3. This process can be expressed as:

[0014]

[0015] where σ represents the ReLU activation function, and cat represents the channel concatenation operation. To further reuse the inherent information in the network, iterative fusion is performed based on the aggregated feature maps b1, b2, and b3: First, b1 and b2 are input into the first fusion node. After the fusion node selects and maps, the feature map x1 is obtained. Then, x1 and b3 are input into the second fusion node to obtain the feature map x2. The fusion node is specifically a 1×1 convolution followed by the ReLU activation function, with the input channel number being 640 and the output channel number being 128. This process can be expressed as:

[0016]

[0017] Finally, x2 is sent into the channel information attention module to measure the importance of different channels. The weights w of each channel of the feature map are obtained through learning, and the weights are multiplied by the original feature map to selectively focus on more important channels and weaken relatively unimportant channels to obtain a better feature map A1. The channel attention module consists of global average pooling GAP and a 1×1 convolution followed by the ReLU activation function, with the input channel number being 128 and the output channel number being 128. This process can be expressed as:

[0018] w = σ(conv(GAP(x2))) (3)

[0019] A1 = x2 × w (4)

[0020] In addition, a residual connection is established between A1 and the input feature of LAM to enhance the conduction of features and gradients, obtaining the output feature map f l1 . This feature map will be sent into the next LAM to continue extracting multi-scale features to obtain more refined features. After stacking 5 LAMs, the input f of the first LAM lAdd it to the output f of the fifth LAM l5 to fuse the low-level features containing rich spatial information and the high-level features containing semantic information of crowd density estimation, and obtain the feature f containing rich multi-scale information for the regression of the backend network o , and its number of channels is 128

[0021] (4) Generation of density map: The backend network uses the feature map containing rich multi-scale information output by the LAM to predict the crowd density map, which consists of 3 convolutional layers with ReLU activation functions, kernel sizes of 3×3, and output channels of 128, 64, and 32 respectively, and 3 transposed convolutional layers with kernel sizes of 2×2, strides of 2, and output channels of 64, 32, and 32 respectively. Each transposed convolution upsamples the feature map by a factor of two, gradually upsampling the feature map to the same size as the original Figure 1 size. Finally, the final density map is obtained by regression using a 1×1 convolution with an input channel number of 32 and an output channel number of 1

[0022] In summary, due to the adoption of the above technical solutions, the advantages and beneficial technical effects of the present invention compared with the prior art are as follows:

[0023] (1) A crowd counting method combining a layer aggregation mechanism proposed by the present invention can obtain richer multi-scale information with fewer parameters and computational amounts by aggregating convolutional layers with different receptive fields and semantic levels in a single-column convolutional neural network compared with existing multi-column and multi-branch networks

[0024] (2) A crowd counting method combining a layer aggregation mechanism proposed by the present invention aggregates convolutional layer-level information and fuses low-level and high-level features, making fuller use of the information of each layer in the network compared with existing methods, thus achieving higher counting accuracy

[0025] (3) In the present invention, the dilation rates of the dilated convolutions in the layer aggregation mechanism are 1, 2, and 3 that are relatively prime to each other, which can effectively avoid the grid effect and prevent the loss of pixel information compared with the conventional structure of 2, 2, 2

[0026] (4) Based on the layer aggregation mechanism, the present invention generates attention weights through a channel attention module according to the degree of importance of each channel to adjust the feature map and obtain a better feature map for crowd density regression

[0027] (5) The present invention regresses the multi-scale features obtained by the layer aggregation mechanism through the backend network to obtain the crowd density map, and the designed transposed convolution with a kernel size of 2×2 and a stride of 2 can avoid the generation of visual checkerboard effects during the process of gradually upsampling the feature map BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the network structure diagram of the crowd counting model proposed by the present invention.

[0029] Figure 2 This is the schematic diagram of the layer aggregation mechanism (LAM) structure.

[0030] Figure 3 This is the effect diagram of the embodiment of the present invention.

[0031] Figure 4 This is the comparison diagram between the predicted value and the true value of the embodiment of the method of the present invention. Detailed implementation manners

[0032] The following further elaborates on the present invention in conjunction with the accompanying drawings. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the protection scope of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention according to the above-mentioned inventive content for specific implementation, which should still fall within the protection scope of the present invention.

[0033] (1) The method proposed by the present invention is carried out on the deep learning framework PyTorch. Among them, relatively important hyperparameters such as the network training learning rate is 0.00001, the optimizer uses RMSprop, the batchsize is set to 1, the epoch is set to 400, and the network parameter initialization method is a Gaussian distribution with a mean of 0 and a variance of 0.01;

[0034] (2) When verifying the counting performance of the method proposed by the present invention, the commonly used public dataset ShanghaiTech is selected. This dataset covers crowd pictures in various scenarios such as sparse, dense, streets, music festivals, etc.;

[0035] (3) When generating the label of the true crowd density map, the geometric adaptive Gaussian kernel is used to blur the label of each human head in the image. This process can be expressed as:

[0036]

[0037] where x is the pixel position in the image, N is the total number of people in the image, δ(x - xi) represents the person with a label at x i , σ i is the variance of the Gaussian kernel, which is calculated by the average distance between the human head to be blurred and the 3 nearest neighbors around it. Each human head is convolved with the corresponding Gaussian kernel G to generate the true crowd density map F(x);

[0038] (4) In the program of the algorithm proposed in the present invention, during the training of the network, the image preprocessing described in (2) is performed on the images, and the obtained image patches are fed into the network to regress the crowd density map. During the testing, the entire image is directly input into the trained model to obtain the crowd density estimation map;

[0039] (5) The existing commonly used crowd counting methods compared in the present invention are as follows:

[0040] Method 1: The method (MCNN) proposed by Zhang Y, Zhou D, Chen S, etc., reference "Single-image crowd counting via multi-column convolutional neural network[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:589-597";

[0041] Method 2: The method (Switch-CNN) proposed by Babu Sam D, Surya S, Venkatesh Babu R, reference "Switching convolutional neural network for crowd counting[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:5744-5752";

[0042] Method 3: The method (SANet) proposed by Cao X, Wang Z, Zhao Y, etc., reference "Scale aggregation network for accurate and efficient crowd counting[C] / / Proceedings of the European Conference on Computer Vision (ECCV). 2018:734-750";

[0043] Method 4: The method proposed by Li Y, Zhang X, and Chen D (CSRNet), reference "Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 1091-1100";

[0044] Method 5: The method proposed by Jiang X, Zhang L, Lv P, etc. (MLCNN), reference "Learning multi-level density maps for crowd counting [J]. IEEE transactions on neural networks and learning systems, 2019, 31(8): 2705-2715";

[0045] (6) The present invention uses the mean absolute error and the root mean square error RMSE to evaluate the accuracy and robustness of the proposed crowd counting method. As can be seen from Table 1, compared with the conventional multi-column (Methods 1 and 2) and multi-branch networks (Method 3), as well as the dilated convolutional network (Method 4) and the multi-level network (Method 5), the method proposed by the present invention obtains lower MAE and RMSE than the other 5 crowd counting networks, indicating that the method of the present invention can perform crowd counting more accurately. At the same time Figure 3 The partial result diagrams listed also show that the method of the present invention can predict high-quality crowd density distribution diagrams;

[0046] (7) As Figure 4 can be seen, whether it is the ShanghaiTech A dataset in a relatively crowded scene or the ShanghaiTech B dataset in a relatively sparse scene, the sample points are very close to the diagonal line, that is, it indicates that the predicted value of the total number of people by the method of the present invention is very close to the true value.

[0047] Table 1 Precision comparison of the above different methods on the ShanghaiTech dataset

[0048]

Claims

1. A crowd counting method combining layer aggregation mechanism, characterized in that It includes the following steps: (1) Image preprocessing: Four non-overlapping image patches with a size of one-fourth of the original image are obtained by cropping, and another four image patches are obtained by random cropping. Then, horizontal flipping is performed to make full use of the limited image samples. Finally, image enhancement is carried out; (2) Low-level feature extraction of the image: Use the first 10 convolutional layers of VGG16 as the backbone network to extract low-level features f containing rich spatial information v , reduce the number of channels of the feature map, so that the network parameters and computational amount are reduced, and the feature map f is obtained l ; (3) Extraction of multi-scale features: Five layer aggregation mechanisms (LAMs) are cascaded. LAM aggregates layers with different receptive fields and semantic levels to obtain rich multi-scale information for density map regression. It mainly includes the gradual aggregation of convolutional layers, the iterative fusion of aggregated features, and the channel information attention module. The specific structure is as follows: Step-by-step Aggregation of Convolutional Layers: First, the input feature map f l is fed into the first convolution c1. Then, the input of this convolution is concatenated with the output activated by the ReLU function along the channels to obtain the aggregated feature map b1, which is one aggregation operation. Then, b1 is fed into the second convolution c2 to perform the second aggregation operation to obtain the aggregated feature map b2, and b2 is fed into the third convolution c3 to perform the third aggregation operation to obtain the aggregated feature map b3. This process can be expressed as: where σ represents the ReLU activation function, cat represents the channel concatenation operation. To further reuse the inherent information in the network, iterative fusion is performed on the basis of the aggregated feature maps b1, b2, and b3: First, b1 and b2 are input into the first fusion node, and after being selected and mapped by the fusion node, the feature map x1 is obtained. Then, x1 and b3 are input into the second fusion node to obtain the feature map x2. The fusion node is specifically a 1×1 convolution followed by the ReLU activation function. This process can be expressed as: Finally, x2 is sent into the channel information attention module to measure the importance of different channels. The weights w of each channel of the feature map are obtained through learning, and the weights are multiplied by the original feature map to selectively focus on more important channels and weaken relatively unimportant channels to obtain a better feature map A1. The channel attention module consists of global average pooling GAP and a 1×1 convolution followed by the ReLU activation function. This process can be expressed as: w = σ(conv(GAP(x2))) (3) A1 = x2 × w (4) In addition, a residual connection is established between A1 and the input features of LAM to enhance the conduction of features and gradients; (4) Generation of density map: The backend network uses the feature map containing rich multi-scale information output by LAM to predict the crowd density map. It consists of 3 convolutional layers and 3 transposed convolutional layers. The transposed convolution upsamples the feature map to the same size as the original image, and finally, the final density map is obtained by 1×1 convolution regression; 2. The method for crowd counting combining a layer aggregation mechanism as claimed in claim 1, wherein The image with too large a resolution is processed to have a maximum side equal to 1024 pixels while keeping the aspect ratio unchanged; The pixel size of the cropped image patch is one-fourth of the original image; The image enhancement is to perform histogram equalization on all the obtained image patches; 3. The method for crowd counting combining a layer aggregation mechanism according to claim 1, wherein, The first 10 layers of VGG16 loaded with pre-trained parameters are used as the backbone network to extract low-level features f v , and its number of channels is 512; The number of channels is reduced to use a 1×1 convolution with a ReLU activation function, and its parameters are: the number of input channels is 512, the number of output channels is 128, the kernel size is 1, and the stride is 1, to obtain the compressed feature map f l , and its number of channels is 128.

4. The population counting method combining a layer aggregation mechanism according to claim 1, wherein Five LAMs are stacked, and the output of the previous LAM is sent to the next LAM to continue refining the features, and finally, the features for crowd density prediction are obtained; The 3 convolutional layers are all dilated convolutions with a kernel size of 3×3. While obtaining the same receptive field size as ordinary convolutions, the number of parameters is smaller. The dilation rates are 1, 2, and 3 that are relatively prime to each other, so as to avoid the grid effect caused by the conventional dilation rate of 2, 2, 2 when extracting features with different receptive fields; In the progressive aggregation of the convolutional layer, the aggregated features are fed as inputs into the next convolutional layer to better retain the information extracted by the previous convolutional layers. The number of output channels of the convolutional layers is 128, and the number of channels of the aggregated features b1, b2, and b3 are 256, 384, and 512 respectively; The first fusion node is a 1×1 convolution with a ReLU activation function, having an input channel number of 640 and an output channel number of 128. The number of channels of its output x1 is 128. The second fusion node is a 1×1 convolution with a ReLU activation function, having an input channel number of 640 and an output channel number of 128. The number of channels of its output x2 is 128; In the channel attention module, the input channel number of the convolution is 128, the output channel number is 128, and the ReLU activation function is adopted to ensure that all pixel values in the weight map are greater than or equal to 0.

5. The method for crowd counting combining a layer aggregation mechanism according to claim 1, characterized in that The kernel sizes of the 3 convolutions are 3×3. The input channel numbers are 128, 64, and 32 respectively, and the output channel numbers are 128, 64, and 32 respectively. All of them are equipped with ReLU activation functions to ensure that all pixel values in the feature map are greater than or equal to 0, that is, there are no pixels with negative values of the number of people; The 3 transposed convolutions perform upsampling by a factor of two each time, gradually restoring the feature map to the original image size. Their input channel numbers are 128, 64, and 32, and the output channel numbers are 64, 32, and 32. The kernel sizes are all 2×2, and the strides are all 2, such that the kernel size is divisible by the stride to avoid the checkerboard effect visually; The 1×1 convolution for regression has an input channel number of 32 and an output channel number of 1.

Images