A new people counting method

Abstract

The present application provides a new crowd counting method. First, the first 16 layers of VGG19 are used as the backbone network to extract shallow features, and then a double-branch structure is used in the feature extraction module. Branch 1 uses a pyramid structure with a fusion self-attention mechanism, and the feature map generated by the pyramid structure is sent to a transition residual block to generate the feature map of branch 1. Branch 2 uses a double-channel attention module, and the feature maps obtained by branches 1 and 2 are sent to a transition residual block for splicing and fusion. The fused feature map is sent to a transition module to generate the final feature map. Finally, the final feature map is sent to a 1x1 convolution to generate a density map. During model training, the present application uses a joint loss function to minimize the influence of outliers on the entire model. Branch 1 of the present application can accurately locate targets of different scales and depict the spatial dependency between any two positions in the feature map. Branch 2 of the present application can focus on important features in the crowd, thus achieving excellent crowd counting performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a crowd counting method based on double-branch intermediate feature extraction and joint loss function. BACKGROUND

[0002] In recent years, with the vigorous development of computer vision field, great progress has been made in the field of crowd counting in deep learning, and many crowd counting methods with higher precision have also emerged. Traditional crowd counting is mainly divided into two categories: detection-based method and regression-based method. There are three kinds of existing pedestrian detection methods, which are based on global features, human body parts and stereo vision. The detection-based crowd counting method is effective only when there are few pedestrians and the pedestrians in the image are obvious. However, in daily life, especially during holidays, people often gather together to form high-density crowds, and the detection-based crowd counting method is not applicable in this case.

[0003] The main idea of the regression-based method is to learn a mapping from a feature to the number of people. This method mainly consists of two steps: the first step is to extract low-level features such as foreground features, edge features, texture and gradient features; the second step is to learn a regression model such as linear regression, piecewise linear regression, ridge regression and Gaussian process regression to learn a mapping relationship from low-level features to the number of people. Regression-based methods are divided into single-feature regression-based methods and multi-feature regression-based methods. However, single-feature regression-based methods cannot provide reliable crowd counting results in high-density crowd scenes and have low robustness. Although multi-feature regression-based crowd counting methods obtain more accurate and efficient crowd counting values to some extent, they still cannot solve the problem of crowd counting in high-density crowd scenes.

[0004] Most of the traditional crowd counting methods are no longer applicable now, and the reason is that the detection accuracy is low and the robustness is not high. Now many scholars prefer to study deep learning-driven crowd counting methods. The input picture is processed through a convolutional neural network to extract the head features of the people, generate a crowd density map, and integrate the density map to obtain the total number of people in the picture. This is a brief process of crowd counting based on deep learning. At present, deep learning-driven crowd counting methods have made a lot of research results, and many researchers have been constantly improving the network model, loss function and data set related to crowd counting, which has continuously improved the accuracy of the crowd counting model. Target occlusion, large head size difference, complex background environment and other problems have always been the difficulties faced by crowd counting research. In order to solve these problems, researchers have been optimizing the network model of crowd counting, so the research of multi-scale feature extraction and rich context connection has become the research focus of crowd counting. SUMMARY

[0005] The present application aims to solve the problem of different head scales and complex background environment of crowd counting, use different scene, different light, different density of crowd data set, directly predict the density map through the deep learning network, and integrate the density map to obtain the final crowd number. Through the above method, the density map in the real scene can be obtained and excellent crowd counting result can be obtained.

[0006] In order to achieve the above purpose, the present application provides a crowd counting method based on double-branch intermediate feature extraction and joint loss function, which mainly includes five parts: the first part is to preprocess the data set; the second part is to extract and fuse the features of the data set image; the third part is to generate a density map for the fused feature tensor; the fourth part is to integrate the density map to obtain the crowd number; the fifth part is network training and testing to obtain the final density map.

[0007] The first part includes two steps:

[0008] Step 1, first download crowd counting ShanghaiTech PartB, UCF-QNRF, UCF-CC50 data set, wherein the label format is.mat, the mat file records the two-dimensional coordinates of each annotated head and the total number of heads, the true value density map is generated according to the annotation in the data set, first construct a matrix with the same size as the original picture, and set all to 0, then set the position corresponding to each marked head to 1, so that a matrix with only 0 and 1 is obtained, and finally a continuous density map is obtained by convolution through a Gaussian kernel function;

[0009] Step 2, randomly crop 1 / 4 of the data set image as training image, and enhance the training sample set through rotation and translation basic operation to form the final training set sample;

[0010] The second part includes two steps:

[0011] Step 3, input the training sample in step 2 into the backbone network composed of the first 16 layers of VGG19, and obtain the feature tensor A through channel number transformation from the RGB space

[0012] Step 3, input the training sample in step 2 into the backbone network composed of the first 16 layers of VGG19, and obtain the feature tensor A through channel number transformation from the RGB space

[0013] Step 4, send the feature tensor A obtained in step 3 into branch 1 and branch 2 respectively; in branch 1, the feature tensor A obtains feature tensors B1, B2, B3, B4 and B5 through a pyramid structure, wherein B1 is a feature tensor obtained through 1x1 convolution, B2, B3 and B4 are feature tensors obtained through three different dilated convolutions, and B5 is a feature tensor obtained through global pooling; the feature tensors B2, B3,

[0014] B4 sends into the self-attention module to obtain feature tensors C1, C2, C3, and then splices the feature tensors C1, C2, C3, B1 and B5 to obtain a feature tensor D, and finally sends the feature tensor D into a residual transition block with two convolution cascades to obtain the final feature tensor E of branch 1;

[0015] Step 5, in branch 2, the feature tensor A passes through a double-channel attention module to obtain a feature tensor F; the feature tensor E obtained in branch 1 is spliced with the feature tensor F obtained in branch 2 to obtain a feature tensor G, and G is sent into a residual transition block to obtain a fused feature tensor H;

[0016]

[0017] The third part includes one step:

[0018] Step 6, the feature tensor H obtained in step 5 is processed through a transition module with three convolution blocks in cascade to extract deep two-dimensional information and transform channels to obtain a final feature map I, and finally the feature map I is processed through a 1x1 convolution to obtain a predicted density map Figure 7 ;

[0019] The fourth part includes one step:

[0020] Step 7, the predicted density map obtained in step 6 is summed to obtain the final crowd number;

[0021] The fifth part includes three steps:

[0022] Step 8, in training, the Euclidean distance loss function L2 and L1 are jointly used;

[0023] Step 9, adjust the network structure hyperparameters from step 3 to step 6, and obtain the final training model;

[0024] Step 10, input the test set in step 1 into the training model in step 9 to obtain the final density map.

[0025] ​The present application proposes a new crowd counting method. First, the first 16 layers of VGG19 are used as the backbone network to extract shallow features, and then deep features are extracted using a double-branch structure. In branch 1, a pyramid structure is used in the feature extraction module, forming five parts of three dilated convolutions with different expansion rates, 1x1 convolution and global pooling. After the dilated convolution, a self-attention module is connected, the attention map formed by the self-attention module is spliced with the 1x1 convolution and the global pooling to form an intermediate feature map, and then the intermediate feature map is sent to the residual transition block to form the feature extraction tensor of branch 1. In branch 2, a double-channel attention module is directly used to obtain the feature extraction tensor of branch 2. The feature tensors of the double branches are fused by the residual transition block, and finally three convolution kernels of the same size are used to form a transition module and a ReLU function is connected after each convolution. Then the final feature map is sent to a 1x1 convolution to generate a density map. The feature extraction module of the present application can accurately locate targets of different scales and depict the spatial dependency between any two positions in the feature map. The joint double-channel attention module can focus on the most important part of the crowd and suppress background interference. During the training process, the improved absolute error loss function is used in combination with the Euclidean distance to minimize the influence of outliers on the entire model, reduce the sensitivity of L2 to outliers, and improve the detection accuracy when the crowd distribution is uneven. The crowd counting method proposed by the present application achieves excellent crowd counting performance and achieves excellent results on different data sets. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 It is the network overall framework diagram of the present application;

[0027] Figure 2 It is the pyramid structure diagram of the present application;

[0028] Figure 3 It is the self-attention mechanism diagram of the present application;

[0029] Figure 4 It is the double-channel attention mechanism diagram of the present application;

[0030] Figure 5 It is the residual transition block structure diagram of the present application;

[0031] Figure 6 It is the original image;

[0032] Figure 7 It is the density map processed by the present application Figure 6 after. DETAILED DESCRIPTION

[0033] For a better understanding of the present application, the crowd counting method based on multi-scale feature extraction of the present application based on a double-branch structure is described in more detail below in conjunction with the specific embodiments. In the following description, detailed descriptions of the prior art that may obscure the subject matter of the present application will be omitted.

[0034] Step 1, download the crowd counting public data set, select different scenes, different crowd density data sets, and then generate the true value density map according to the annotation in the data set. Different ways are used to generate the true value density map for different data sets. The present application uses three data sets for testing. In order to obtain the true value density map, the three data sets are generated in different ways. For ShanghaiTech Part B, UCF-QNRF, and UCF-CC50 data sets, a fixed Gaussian kernel is used to generate a density map, while ShanghaiTech part A uses a geometric adaptive Gaussian kernel to generate a density map. As a label for prediction learning, the specific formula (1) and formula (2) are as follows:

[0035]

[0036]

[0037] wherein formula (2) is the label of the crowd counting data set, x is a pixel point in the image, x i represents the position of the i-th head in the picture, G(x) is a Gaussian kernel filter, d is the average distance between each person's head and its adjacent k persons, and β is set to 0.3;

[0038] Step 2, randomly crop 1 / 4 of the data set picture as a training image, and perform rotation and translation basic operations to enhance the training sample set to form the final training set sample;

[0039] Figure 1 is the specific network model diagram of the present application based on a double-branch intermediate feature extraction module, in the present embodiment, the following steps are performed:

[0040] Step 3, the image feature tensor A is preliminarily extracted by a convolutional network, wherein the convolutional network is the first 16 layers of the VGG19 network;

[0041] Step 4, a double-branch structure is used to extract deep layer features, which is implemented as follows:

[0042] Step 4-1, branch 1 uses a pyramid structure to extract multi-scale features, and the pyramid structure is as follows: Figure 2It consists of three sets of improved dilated convolution blocks, one set of 1×1 convolution blocks, and one set of global pooling blocks. The dilation rates of the dilated convolutions are set to 2, 4, and 6, respectively. The feature tensor A is fed into the pyramid feature extraction module to obtain feature tensors B1, B2, B3, B4, and B5.

[0043] Step 4-2, self-attention mechanism as follows Figure 3 The three sets of dilated convolutional blocks are fed into the attention module to obtain attention maps C1, C2, and C3. Then, the attention maps are concatenated with 1×1 convolutional blocks and global pooling blocks to obtain intermediate feature maps D.

[0044] Step 4-3: The feature tensor D is fed into a residual transition block with two convolutional cascades to obtain the final feature tensor E of branch 1. The residual transition block is as follows: Figure 4 ;

[0045] Step 4-4, Branch 2 uses a dual-channel attention module as follows: Figure 5 The feature tensor A is used to obtain the feature tensor F through a dual-channel attention module;

[0046] Steps 4-5: Concatenate the feature tensor E obtained from branch 1 with the feature tensor F obtained from branch 2 to obtain feature tensor G, and feed G into the residual transition block to obtain the fused feature tensor H.

[0047] Step 5, the calculation formula (3) for attention Q obtained from step 4-2, where F in ∈R C×H×W F represents the input to the attention module. out ∈R C×H×W Representing the output of the attention module, the Softmax function converts the output values ​​into a probability distribution ranging from [0,1] to 1, and R1() reshapes the input features into R... C×N N = H × W is the number of features. Then, we combine the obtained attention weights Q with the module input F. in Combined to generate attention-enhanced feature maps, plus the input F in The final output F of the attention module is obtained. out The expression is as shown in formula (4), where R2() reshapes the feature into R. C×H×W Note that the 1×1 convolution and global average pooling in the pyramid structure utilize the minimum and maximum receptive fields, respectively, to preserve the inherent properties of the original feature map. Therefore, no attention module is added to these two branches.

[0048] Q = softmax(R1(Conv(F)) in )) T ×R1(Conv(F in (3)

[0049] F out =F in +R2(R1(Conv(F in ))×Q T ) (4)

[0050] Step 6, the feature tensor H is formed into the final feature map I through a transition block of three convolution blocks, and the final feature map is subjected to 1*1 convolution to obtain the last prediction density map, each convolution block is composed of 3*3 convolution with different input and output channels, Group Normalization and ReLU function;

[0051] Step 7, when training the crowd counting model, the Euclidean distance is usually used as the loss function for network training, and the Euclidean distance loss function is used in combination with the improved absolute value error loss function to constrain the difference between the predicted density map and the real density map, and the Euclidean distance loss function is as formula (5):

[0052]

[0053] In the formula, B is the batch size, is the prediction density map of the i-th picture, is the ground truth density map of the i-th picture;

[0054] The mean square error (MSE) is much larger than the mean absolute error (MAE) at the point of large error, and it gives greater weight to the outlier, and the model will strive to reduce the error caused by the outlier, so that the overall performance of the model is reduced, therefore, the improved absolute value error loss function is used in combination with the Euclidean distance to minimize the influence of the outlier on the whole model, reduce the sensitivity of L2 to the outlier, improve the detection accuracy when the crowd is unevenly distributed, and the improved absolute value error loss function is as shown in formula (6):

[0055]

[0056] In the formula, B is the batch size, is the prediction density map of the i-th picture, is the ground truth density map of the i-th picture, and the final joint loss function used is as formula (7):

[0057] Loss=L2+αL1 (7)

[0058] Wherein, alpha is a hyperparameter, used to balance the weight between L1 and L2, and the specific value is obtained through experiment;

[0059] Step 8, debug the network framework of steps 3 to 7, set the network model parameters, wherein the initial learning rate is set to 0.0001, Epochs is set to 2000, Bachsize is set to 16, the parameters are optimized using Adam optimizer, the parameter decay is set to 0.0005, the joint loss function hyperparameter alpha is adjusted, and the optimal training model is obtained by training;

[0060] Step 9, input the test picture Figure 6 into the pre-trained model, and output as Figure 7 a medium density map.

[0061] The present application aims at the scale change and background interference problem in static images, adopts a double-branch structure to extract deep features, uses a multi-scale feature extraction module based on self-attention mechanism fusion to locate targets of different scales and depict the spatial dependency between any two positions in the feature map, adds a double-channel attention module to make the model focus on important features and suppress background interference, introduces the first 16 layers of the convolutional neural network-VGG19 network as a forward network to extract features, designs a residual transition block for feature fusion regression, and finally generates a high-quality density map through 1x1 convolution. The method proposed in the present application is optimized from two aspects of enhancing the scale adaptability of features and reducing background noise interference, and the algorithm is simple and has strong operability.

[0062] Although the above describes the specific embodiments of the present application, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are included in the protection.

Claims

1. A novel method for counting people, characterized in that... The network model is designed with two branches for deep feature extraction. Branch 1 incorporates a pyramid structure with a fusion attention mechanism, while Branch 2 incorporates a dual-channel attention module. It also features a residual transition block for feature fusion, a transition module for extracting deep two-dimensional information, and a joint loss function during model training. These features are further elaborated in five parts: dataset preprocessing, image feature extraction and fusion, generating a density map from the fused feature tensor, integrating the density map to obtain the number of people, and network training and testing. The first part consists of two steps. Step 1: First, download the crowd counting datasets ShanghaiTech PartB, UCF-QNRF, and UCF-CC-50. The label format is .mat. The .mat file records the two-dimensional coordinates of each annotated head and the total number of heads. Generate a ground truth density map based on the annotations in the dataset. First, construct a matrix of the same size as the original image and set all of it to 0. Then, set the position corresponding to each labeled head to 1. This results in a matrix with only 0s and 1s. Finally, convolve the matrix using a Gaussian kernel function to obtain a continuous density map. Step 2: Randomly crop 1 / 4 of the images in the dataset as training images, and perform basic operations such as rotation and translation on the images to enhance the training sample set and form the final training set samples; Part Two consists of three steps: Step 3: Input the training samples from Step 2 into the backbone network consisting of the first 16 layers of VGG19, and obtain the feature tensor A from the RGB space through channel number transformation. Step 4: The feature tensor A obtained in Step 3 is fed into Branch 1 and Branch 2 respectively. In Branch 1, feature tensor A is processed through a pyramid structure to obtain feature tensors B1, B2, B3, B4, and B5. B1 is a feature tensor obtained through 1×1 convolution, B2, B3, and B4 are feature tensors obtained through three different dilated convolutions, and B5 is a feature tensor obtained through global pooling. Feature tensors B2, B3, and B4 are fed into the self-attention module to obtain feature tensors C1, C2, and C3. Feature tensors C1, C2, C3, B1, and B5 are then concatenated to obtain feature tensor D. Finally, feature tensor D is fed into a residual transition block with two concatenated convolutions to obtain the final feature tensor E of Branch 1. Step 5, in branch 2, feature tensor A is passed through the dual-channel attention module to obtain feature tensor F; feature tensor E obtained from branch 1 is concatenated with feature tensor F obtained from branch 2 to obtain feature tensor G, and G is sent into the residual transition block to obtain the fused feature tensor H. Part Three includes a step: Step 6: The feature tensor H obtained in step 5 is processed by a transition module consisting of three convolutional blocks to extract deep two-dimensional information and transform the channels to obtain the final feature map I. Finally, the feature map I is convolved by 1×1 to obtain the prediction density map. Part Four includes one step: Step 7: Sum the predicted density maps obtained in Step 6 to obtain the final population size; Part Five consists of three steps: Step 8: During training, use the Euclidean distance loss functions L2 and L1 together; Step 9: Debug the network structure hyperparameters from Step 3 to Step 6 and obtain the final trained model; Step 10: Input the test set from Step 1 into the training model in Step 9 to obtain the final density map.

2. The novel crowd counting method according to claim 1, characterized in that, In step 4, the multi-scale feature extraction module that integrates the attention mechanism and the dual-branch structure that incorporates the dual-channel attention model accurately locates targets at different scales, depicts the spatial dependency between any two locations, and enables the model to learn to pay attention to important regions.

3. A novel crowd counting method according to claim 1, characterized in that, The residual transition block designed in step 4 uses a residual structure to dynamically adjust the model complexity and change the number of channels, and performs fusion regression on the extracted features.

Images