A crowd counting method combined with a learnable attention mechanism

Abstract

The application discloses a crowd counting method combined with a learnable attention mechanism, and comprises the following steps: 1, acquiring an image of a target region and a real point array image; 2, pre-processing the image of the target region; 3, generating a real density distribution image; 4, initializing a parameter of a learnable attention mechanism network model, taking an image in a training set as input, taking a predicted density distribution image of the target region as output, training the learnable attention mechanism network model by comparing an error between the predicted density distribution image of the target region and the real density distribution image, and testing the model precision through a test set to obtain the trained learnable attention mechanism network model. The application can solve the problems of scale change and head position deviation of a crowd counting network model, and is more suitable for scale change in a scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a crowd counting method combined with a learnable attention mechanism. BACKGROUND

[0002] Crowd counting plays a vital role in crowd estimation, video surveillance and crowd management. Real-time crowd detection and counting are increasingly attracting attention. Therefore, dense crowd counting has become a research hotspot in the field of computer vision. This research has a wide range of applications in the real world, such as security monitoring, traffic control and intelligent transportation. However, due to the influence of factors such as scale variation, background noise and occlusion in the monitoring scene, accurately and effectively predicting the number of crowds is still a serious challenge. In the problem of crowd counting and density estimation, the research on crowd counting technology based on CNN has achieved remarkable performance. Many researchers have devoted themselves to improving the prediction accuracy through various advanced deep learning methods, such as attention mechanism module, multi-scale module, feature extraction and feature fusion. Thus, the performance of the network model is significantly improved. However, due to the large changes in perspective, density and scale, crowd counting is still a challenging task.

[0003] In recent years, typical counting methods use convolutional neural networks (CNN) as the backbone and regress density maps to predict the total crowd count. However, due to the wide angle of the camera, there are often large-scale changes in the crowd image in 2D perspective projection. Traditional neural networks with fixed-size convolution kernels have difficulty dealing with these changes, and the counting performance is severely limited. In order to alleviate this problem, multi-scale mechanisms such as multi-scale blocks, pyramid networks and multi-column networks are used. These methods introduce an intuitive local structure-sensitive bias, indicating that the corresponding field should adapt to the size of the object. SUMMARY

[0004] The purpose of the present application is to provide a crowd counting method combined with a learnable attention mechanism to solve the problems of scale variation and head position deviation of the crowd counting network model.

[0005] Technical scheme: The crowd counting method combined with a learnable attention mechanism of the present application collects the image of the target area and the real dot array map, trains a learnable attention mechanism network model by performing the following steps, and obtains the predicted density distribution map of the target area based on the model, and then obtains the final crowd counting map of the target area;

[0006] Step 1: Obtain the image of the target area and the real dot array map of the target area;

[0007] Step 2: After pre-processing the image of the target area, a data set is constructed, and the data set is divided into a training set and a test set in proportion;

[0008] Step 3: After the convolution of the real point array of the target area through the Gaussian kernel, input the self-attention module to generate the real density distribution map of the target area;

[0009] Step 4: Initialize the parameters of the learnable attention mechanism network model, input the image in the training set as input, and the predicted density distribution map of the target area as output. By comparing the error between the predicted density distribution map of the target area and the real density distribution map, the learnable attention mechanism network model is trained, and the model accuracy is tested through the test set to obtain the trained learnable attention mechanism network model;

[0010] Step 5: Based on the learnable attention mechanism network model, the predicted density distribution map of the target area is obtained, and the loss function is used to jointly train the predicted density distribution map and the real density distribution map to obtain the final crowd counting map.

[0011] Further, in step 2, the image in the target area is preprocessed to have a uniform image size, and the edge is filled to convert it to a uniform size format. The data is sequentially cropped, horizontally flipped, and brightness adjusted.

[0012] Further, in step 4, the learnable attention mechanism network model uses VGG-19 as the convolutional neural network backbone network, which is pre-trained on ImageNet. The LRA replaces the attention module, and the regression decoder consists of one upsampling layer and three convolutional layers with ReLU activation function. The kernel size of the first two layers is 3x3, and the kernel size of the last layer is 1x1. The important hyperparameters such as the training learning rate lr are set to 5e-7, and the weight decay weight_decay is set to 1e-4.

[0013] Further, in step 4, the learnable attention mechanism network model includes the following steps:

[0014] Step 41: Extract local features of the target area image;

[0015] Step 42: Determine the local area of interest for each feature position based on the learnable attention module;

[0016] Step 43: Use the density map refinement framework to train and refine the density map, and output the feature and learnable region attention to the transformer encoder;

[0017] Step 44: After decoding through the decoder, the predicted density map of the target area is obtained.

[0018] Further, in step 41, the feature F is extracted, where C, W, and H are the channel, width, and height, respectively, and then the feature map is flattened.

[0019] Further, in step 42, the structure of the learnable attention module is as follows:

[0020] First, determine the position p = (X p , Y p ), where 0 ≤ X p < W, 0 ≤ Y p < H. The position of the labeled point is predicted as follows:

[0021]

[0022]

[0023] Given two predicted vertices, bottom left (bl) and upper right (ur): for a particular feature, b = (x b , y b ), u = (x u , y u ), the filtering region is calculated by the following formula:

[0024]

[0025] The product of the two filtering regions is represented as:

[0026]

[0027] When global attention is used, R is represented as an all-1 matrix.

[0028] Further, in step 43, the density map refinement framework refines the density map and trains the counter from the refined density map, letting (X i , Y i ) be the i-th image and the traditional density map pair, f(X i ) represent the predicted density map of image X i , g(Y i ) be the refined density map of Y i , train using density map refinement, then use LRA to transfer it to the transformer encoder to learn features F' of different scales. Then, use the regression decoder to predict the final density map D ∈ RW' × H' from F'.

[0029] Further, the number of encoder layers T is set to 4.

[0030] Further, in step 5, the loss function L is as follows

[0031]

[0032] where, is the counter predicted density map, M iYes, the real density map, and M i ′ is the vectorized version of and M i , is the cosine similarity between two vectors;

[0033]

[0034] where epsilon = 10 -8 , spatially regularize the density map using cosine similarity, normalize the vector and M i ′ / ||M i ′||2 represents the spatial distribution of the crowd, independent of the count, and the cosine similarity in the above formula encourages similar spatial distribution of the crowd and non-crowd between the GT and the predicted density map.

[0035] Advantages: compared with the prior art, the present application has the following remarkable advantages:

[0036] The algorithm makes each feature learn the range that should be paid attention to, captures multi-scale context information in the local attention module, improves the counting accuracy, relieves the counting inaccuracy problem caused by scale change in the crowd counting image, and generates a high-quality density estimation map. Through the combination of the crowd density prediction map and the real crowd density map, the adaptive density map generation framework is adopted to solve the problem that the occlusion between people, complex background and the like will cause the performance of the crowd counting to decline. The algorithm reduces the influence of the chaotic distribution of the crowd on the quality of the density map, and further improves the counting performance and generalization ability of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is the overall flowchart of the present application;

[0038] Figure 2 is the network structure diagram of the present application;

[0039] Figure 3 is the determination of the attention range of feature learning in the learnable attention module;

[0040] Figure 4 the learnable attention module;

[0041] Figure 5 is the result comparison of the real density map and the predicted density map of the method of the present application. DETAILED DESCRIPTION

[0042] The technical solutions of the present application will be further described below in combination with the drawings.

[0043] A crowd counting method combined with a learnable attention mechanism according to the present application is as follows:Figure 1 As shown in the figure, it mainly includes the following steps:

[0044] (1) Image preprocessing: Since it is necessary to unify the image size for the training and test part of the dataset, edge padding processing is adopted to convert it into a unified size format for the next step. Then, the data is cropped, horizontally flipped, and brightness adjusted. Then, the corresponding Python program processes the annotation file, that is, the.mat file and the corresponding image information into the.h5py format.

[0045] (2) Local feature extraction of images: Based on VGG19, a learnable attention mechanism is adopted for the regions that should be noted for each feature learning, mainly to address the scale difference problem that the head size may vary greatly due to the distance relationship in the same image.

[0046] (3) Adaptive density map generation: As[[ID=ll]] Figure 2 shown, it mainly generates a density map from the point annotation map. Using this density map generation framework, the entire system can be trained end-to-end without any intermediate density maps. This method adaptively fuses the density maps constructed from different kernels and then refines them to generate a new density map.

[0047] (4) Learnable region attention: As Figure 3 shown, the learnable attention mechanism can flexibly determine the local region that each feature position should focus on. The local attention module provides an effective way to extract the most relevant local information for scale changes. Through this mechanism, each feature can learn the most suitable local region. Since a rectangular region can be recognized by two vertices, we start from the region filtering mechanism to obtain the exclusive region for each position. The specific structure of the learnable attention module is as follows:

[0048] First, determine the position p=(X p , Y p ), where 0≤X p <W, 0≤Y p <H. The prediction of the annotation point position is as follows:

[0049]

[0050] Given two predicted vertices, the lower left (bl) and the upper right (ur): For a specific feature, b=(x b , y b ), u=(x u , y u ), and its filtered region is calculated by the following formula:

[0051]

[0052] The product of the two filtering regions is denoted as:

[0053]

[0054] When global attention is adopted, R is represented as a full 1 matrix.

[0055] The present application uses VGG-19 as the backbone to extract features F e RCxWxH, where C, W and H are the channel, width and height respectively. Then the feature map is flattened and transmitted to the transformer encoder using the proposed LRA to learn features F' of different scales. Then, the final density map D e RW'xH' is predicted from F' using the regression decoder.

[0056] Embodiment

[0057] The method proposed in the present application uses VGG-19 as the convolutional neural network backbone network, which is pre-trained on ImageNet. LRA is used to replace the attention module. The regression decoder is composed of one up-sampling layer and three convolutional layers with activation ReLU function. The kernel size of the first two layers is 3x3, and the kernel size of the last layer is 1x1. Among them, the important hyperparameters such as the training learning rate lr are set to 5e-7, and the weight decay weight_decay is set to 1e-4;

[0058] When verifying the prediction performance of the method proposed in the present application, the ShanghaiTech A, ShanghaiTech B and UCF-QNRF public data sets are selected. Covering common sparse, dense, street, square and other types of crowd gathering places.

[0059] In the training stage, first, random scaling and horizontal flipping are used for each training image, and then we randomly crop image blocks with a size of 512x512. Since some images in ShanghaiTech A contain smaller resolution, the cropping size of this dataset is changed to 256x256. We also limit the short side of each image in all datasets to within 2048 pixels. The Adam algorithm with a learning rate of 10-5 is used to optimize the parameters. We set the number of encoder layers T to 4 and the loss balancing parameter lambda to 100.

[0060] The mean absolute error (MAE) and root mean square error (RMSE) are used to evaluate the prediction accuracy and robustness of the method proposed in the present application.

[0061] The commonly used lightweight network for comparison of the method of the present application is:

[0062] Method 1: Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, etc. proposed a simple but effective multi-column convolutional neural network (MCNN) architecture, which fused three columns of different scale features to generate a density map. It can be applied to any shooting angle and any scene density, and can accurately estimate the crowd number from a single image with any crowd density and any perspective. But it is difficult to solve the problem of serious background occlusion in the actual scene.

[0063] Method 2: Vishwanath A. Sindagi and Vishal M. Patel, etc. proposed a network architecture that mainly solves the problem of large scene changes and large scale and appearance changes of people in the scene in crowd density estimation. The network mainly learns a model that meets various density levels in the data set by combining a high-level prior with the network. The high-level prior first divides the number of people in the picture into different labeled groups. Using the label, this high-level prior can roughly estimate the number of people in the entire picture without being affected by the scale change, so that the network can learn more global features. The high-level prior and CNN network are used together to estimate the density map.

[0064] Method 3: Deepak Babu Sam, Shiv Surya, etc. proposed Switch-CNN (CVPR 2017), which sends different patches to the corresponding convolutional neural network regressor through the Switch Layer, so that each convolutional neural network regressor only processes a specific scale of crowd. This model has the same problem as MCNN, that is, how many sub-networks should be selected. But this method determines the path of patch input through network learning, which is relatively novel for patch classification.

[0065] As can be seen from Table 1, compared with other light networks, the method proposed in the present application has obtained lower MAE and MSE on most data sets, indicating that the method of the present application is more suitable for scale changes in the scene.

[0066] Table 1

[0067]

Claims

1. A crowd counting method incorporating learnable attention mechanisms, characterized in that, Images of the target area and real raster maps are collected. By performing the following steps, a network model with a learnable attention mechanism is trained, and a predicted density distribution map of the target area is obtained based on the model, thereby obtaining the final crowd counting map of the target area. Step 1: Acquire the image of the target area and the real bitmap of the target area; Step 2: After preprocessing the images of the target region, construct a dataset and divide the dataset into training and test sets proportionally; Step 3: After convolving the real bitmap of the target region with a Gaussian kernel, input it into the self-attention module to generate the real density distribution map of the target region; Step 4: Initialize the parameters of the learnable attention mechanism network model. Take the images in the training set as input and the predicted density distribution map of the target region as output. Train the learnable attention mechanism network model by comparing the error between the predicted density distribution map and the real density distribution map of the target region. Test the model accuracy through the test set to obtain the trained learnable attention mechanism network model. Step 4, specifically training the learnable attention mechanism network model, includes the following steps: Step 41: Extract local features of the target region image; In step 41, extract local features F∈RC×W×H of the target region image, where C, W and H are the channel, width and height, respectively, and then flatten the local feature map; Step 42: Determine the local region of interest for each feature location based on the learnable attention module; the specific structure of the learnable attention module in step 42 is as follows: First, determine the position p = (X p , Y p ), where 0 ≤ X p < W, 0 ≤ Y p < H. The prediction of the marked point position is as follows: Given two prediction vertices, bottom left (bl) and top right (ur): for a specific feature, b = (x b ,y b ), u=(x u ,y u The filtering area is calculated using the following formula: The product of the two filtering regions is expressed as: When global attention is used, R is represented as an all-one matrix; Step 43: The density map is trained and refined using the density map refinement framework, and the output features and learnable region attention are transmitted to the transformer encoder. Step 44: After decoding by the decoder, the predicted density map of the target region is obtained; Step 5: Obtain the predicted density distribution map of the target region based on the learnable attention mechanism network model, and use the loss function to jointly train the predicted density distribution map and the real density distribution map to obtain the final crowd counting map.

2. The crowd counting method incorporating a learnable attention mechanism according to claim 1, characterized in that, In step 2, the preprocessing of the image of the target area involves using edge filling to convert it into a format of uniform size, and then cropping, horizontally flipping, and adjusting the brightness of the image in sequence.

3. The crowd counting method incorporating a learnable attention mechanism according to claim 1, characterized in that, In step 4, the learnable attention mechanism network model uses VGG-19 as the backbone network of the convolutional neural network. This network is pre-trained on ImageNet, and LRA is used to replace the attention module. The regression decoder consists of an upsampling layer and three convolutional layers with ReLU activation function. The kernel size of the first two layers is 3×3, and the kernel size of the last layer is 1×1. The training learning rate lr is set to 5e-7 and the weight decay weight_decay is set to 1e-4.

4. The crowd counting method incorporating a learnable attention mechanism according to claim 1, characterized in that, In step 43, the density map thinning framework jointly thins the density map and trains a counter from the thinned density map, allowing (X) to be thinned. i ,Y i As the i-th image and the traditional density map pair, f(X) i ) is represented as image X i The predicted density map, g(Y) i As Y i The refined density map for f(X) i ) and g(Y i Joint training is performed using combined loss, where α represents a decimal number close to 0; the training loss function is as follows: The attention module is used to transmit it to the transformer encoder to learn features F′ at different scales; the regression decoder is used to predict the final density map D∈RW′×H′ from F′.

5. The crowd counting method incorporating a learnable attention mechanism according to claim 4, characterized in that, The number of encoder layers T is set to 4.

6. The crowd counting method incorporating a learnable attention mechanism according to claim 1, characterized in that, In step 5, the loss function L is as follows: in, It is a counter prediction density map, M i Yes, it's a true density map. and M i ′ is the vectorized version and M i , It is the cosine similarity between two vectors; Where ε = 10 -8 Cosine similarity is used to perform spatial regularization on the density map, and the vectors are standardized. and M i ′ / ||M i ′||2 represents the spatial distribution of the population.

Images