Real-time crowd counting method combined with scale perception

Abstract

The application discloses a real-time crowd counting method combined with scale perception, mainly comprising the following steps: firstly, local features are extracted from an image, and multi-scale information is learned by using a scale module (SFEM) according to the extracted results to adapt to the scale change of the crowd in the scene. Then, a scale fusion module (SFFM) is used to fuse the feature maps with different scale information together to enhance the scale expression ability of the network, and the final output feature map is sent to a regression module (DMR) to predict the output crowd density map. In addition, the application also proposes a multi-level intersection-over-union joint loss function (MIoU) to further reduce the background misrecognition rate and prediction error of the network model. Experimental results show that the method can effectively reduce the parameter quantity and reasoning time of the network model, meet the real-time demand of edge devices, and achieve a better balance in accuracy and complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the problem of dense crowd counting in the field of computer vision, in particular to a real-time crowd counting method combining scale perception. BACKGROUND

[0002] In recent years, dense crowd counting has become a research hotspot in the field of computer vision. This research has a wide range of applications in the real world, such as security monitoring, traffic control and intelligent transportation. However, due to the influence of factors such as scale variation, background noise and occlusion in the monitoring scene, it is still a serious challenge to accurately and effectively predict the number of crowds. Many researchers have devoted to improving the prediction accuracy through various advanced deep learning methods, such as attention mechanism and multi-scale feature extractor, thereby significantly improving the performance of network models.

[0003] At the same time, in the application of edge devices, the designed network model needs to obtain accurate prediction and real-time inference speed under limited computing resources. However, the existing models are difficult to meet the requirements of speed and accuracy at the same time. At present, these models with superior performance have complex network structure and cumbersome training operation. Using these methods requires a large amount of computing resources, which brings a huge burden to edge devices. For computer vision tasks such as target detection and image classification, various lightweight models have been proposed to meet the application needs of edge devices. Based on these successful explorations, some researchers have begun to study lightweight network models dedicated to crowd counting, realizing the rapid inference and prediction of the number of crowds.

[0004] It can be seen that lightweight network models make real-time crowd prediction tasks possible. However, we observe that recent lightweight network work still has some shortcomings. First, in order to obtain more rich scale information in dense scenes, the existing lightweight models are mostly multi-column or branch structures, which still require a large amount of computing resources and are not friendly to edge devices. Second, the extraction and fusion of scale information are crucial to enhance the feature map representation ability for accurate prediction of the model. However, the current lightweight model only focuses on the extraction of scale information, resulting in a certain loss of prediction accuracy. In addition, the widely used L2 loss function considers that all pixels in an image have the possibility of existing crowds. Therefore, when using this loss function, the network model is difficult to learn the difference between background and foreground in the crowd scene, thereby affecting the prediction accuracy. In other words, under the premise of meeting the real-time demand, there is still the possibility of further improving the prediction accuracy. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a real-time crowd counting method combining scale perception to solve the balance problem between accuracy and complexity of the crowd counting network model.

[0006] The technical problems are solved by the following technical solutions: A real-time crowd counting method combined with scale perception mainly comprises the following steps:

[0007] (1) Local feature extraction of image: Many advanced methods use the pre-trained VGG16 first 10 layers network as a local feature extractor. However, due to the large number of parameters and large amount of calculation, it is difficult to be directly applied to a lightweight network. On the basis of VGG16, a local feature extractor (LFE) specially used for lightweight network is designed, and the specific structure is shown in Table 1. Compared with the previous extractor, the number of network layers and the number of channels are limited, and the last feature output is down-sampled to 1 / 8 of the original input size through three MaxPooling layers to further reduce the calculation complexity.

[0008] (2) Multi-scale information extraction: The multi-scale feature extractor (SFEM) is composed of 6 cascaded hybrid dilated convolution modules (HDC), wherein each adjacent two HDC modules share the same weight, so that the network model can obtain more rich scale information under the condition of controlling the same amount of parameters. It is worth noting that the multi-scale feature extractor has only 0.024M parameters and requires very little computing resources. The specific structure of the HDC module is as follows:

[0009] As the basic unit of the multi-scale feature extractor, the parameter amount of a single HDC module is less than 0.008M, and the specific structure is as shown in Figure 2 . First, the input feature I∈R N×H×W×C is sent to a 1×1 convolution layer for channel compression to obtain the output feature I0∈R N ×H×W×(C / 4) , wherein 4 is the channel compression ratio, N, H, W and C represent the batch size, height, width and channel number. Then I0 is sent to three cascaded and dilated convolution layers D={1,2,3} with coprime dilation rates to extract scale information and enlarge the receptive field. Assuming that the output features of the three dilated convolution layers are I1∈R N×H×W×(C / 4) , I2∈R N×H×W×(C / 4) , I3∈R N×H×W×(C / 4) , then the final output of the HDC module is:

[0010] F HDC = C{I0,I1,I2,I3}+I (1)

[0011] , wherein C represents a channel splicing operation. In addition, a shortcut connection is established between the spliced features and the HDC input features I to enhance the gradient backpropagation in the HDC block.

[0012] (3) Multi-scale feature information fusion: The multi-scale feature fusion module (SFFM) contains three efficient fusion modules (EAF) based on attention mechanism, and each EAF module has only 0.004M parameters, aiming to effectively fuse multiple feature maps with different scale information. The multi-scale fusion module proposed in the application mainly fuses the feature maps from four different modules, including the output F LEF ∈R N×H×W×C of the LFE, and the outputs F HDC2 ∈R N×H×W×C and F HDC4 ∈R N×H×W×C of each two HDC modules. HDC6 ∈R N ×H×W×C At the same time, in order to enhance the gradient backpropagation, a long skip connection is established to directly add the fused features and the features generated by the LFE, and the final output of the multi-scale feature fusion module is as follows:

[0013] F SFFM =E(E(E(F LEF ,F HDC2 ),F HDC4 ),F HDC6 )+F LEF (2)

[0014] Where E represents the operation of the EAF module, and the specific implementation details of the module are as follows:

[0015] The EAF module aims to enhance the feature expression ability by integrating the information of two different features. As shown in Figure 3 , suppose two features are χ1∈R N×H×W×C and χ2∈R N×H×W×C . By global average pooling (GAP), the features χ1, χ2 can be compressed to one-dimensional corresponding outputs ω1∈R N×1×1×C , ω2∈R N×1×1×C . Then we transform and splice the two outputs together to form a new feature combination vector ω∈R N×2×C , and feed it into two parallel one-dimensional convolution layers with a kernel size of 9 to learn the channel weights corresponding to the two features χ1, χ2 when fused, i.e. ω'1∈R N×1×1×C , ω'2∈R N×1×1×C . In summary, the weight learning of the channel is as follows:

[0016] ω1=G(χ1) (3)

[0017] ω2=G(χ2) (4)

[0018] ω'1=R{σ[f1[C(R(ω1),R(ω2));θ2]]} (5)

[0019] ω'2 = R{σ[f2[C(R(ω1), R(ω2)); θ2]]} (6)

[0020] where G denotes global average pooling, C denotes concatenation, f(; θ) is one-dimensional convolution operation, σ is sigmoid function, and R denotes vector transformation. Then we apply the channel weights to two input features χ1, χ2 and fuse the corresponding weighted features χ'1∈R N×H×W×C , χ'2∈R N×H×W×C together, and a residual connection is adopted here to improve the information flow. The above feature map fusion process can be summarized as:

[0021] F' EAF = ω'1 ⊙ χ1 + ω'2 ⊙ χ2 + χ1 + χ2 (7)

[0022] In addition, we use a normalization layer to process the fused feature F'EAF∈R N×H×W×C , and then send it to a 1x1 convolution to generate the final feature map F EAF ∈R N×H×W×C of the EAF block.

[0023] (4) Density map prediction: The density map prediction module (DMR) mainly uses the feature map output by the SFFM to predict the crowd density distribution map, and the specific structure is shown in Table 2. It is a simple and effective fully convolutional structure, mainly composed of a series of convolutional and deconvolutional layers. Among them, the three deconvolutional layers are mainly responsible for restoring the feature map to the same size as the original image. Finally, a ReLU function is added to the output of the network to ensure that all pixel values in the density map are not less than 0.

[0024] (5) The method of the present application adopts a normalized Euclidean loss function (NEL) and a multi-level intersection-over-union loss function (MIoU) to jointly train the network model. NEL is used to measure the error between the predicted value and the true value at each pixel, and can also alleviate the problem of sample imbalance during training. The loss function is defined as follows:

[0025]

[0026]

[0027] where (i,j) is the pixel position in the density map, X represents the input image, Y is the corresponding label, θ represents the network weight parameter, F(X; θ) is the density map predicted by the network, and ||.||2 represents the Euclidean distance.

[0028] The classic IoU is usually used in the target detection task to calculate the coincidence degree between the predicted region and the real region, so as to distinguish the background from the target. Therefore, the present application proposes a multi-level intersection over union (MIOU) loss function to measure the accurate crowd prediction rate of the output density map, so as to guide the network to learn accurate crowd recognition and density value estimation. The visual calculation process of the MIoU loss is shown in Figure 4 The whole calculation process mainly includes three steps. First, the predicted crowd number C est and the real crowd number C gt are obtained by using the following formula:

[0029]

[0030]

[0031] Where D gt is the real density map, C est is the predicted density map, and (i, j) represents the pixel position in the density map. Therefore, the IoU loss function defined in the method of the present application is as follows:

[0032]

[0033] Where ∩ represents the intersection operation, and ∪ represents the union operation. Finally, as shown in Figure 4 , due to the possible pixel deviation between the predicted position and the real position, a down-sampling operation is performed to reduce the measurement error caused by the pixel deviation, and the final MIoU calculation formula is as follows:

[0034]

[0035] Where l represents the down-sampling size of the original picture, which is 1 / (2^(l-1)). In the method disclosed in the present application, l is set to 4 to obtain the best performance. The target loss function Lc of the final network training is defined as follows:

[0036] L c =NL e +MIoU (14) BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The structure diagram of the real-time crowd counting network proposed in the present application.

[0038] Figure 2 The structure diagram of the hybrid dilated convolution module (HDC).

[0039] Figure 3 The structure diagram of the efficient attention fusion module (EAF) based on the attention mechanism.

[0040] Figure 4The schematic diagram for visualizing the solving process of the MIoU loss function.

[0041] Figure 5 The visualization result comparison of the predicted density map and the real density map of the method of the application. DETAILED DESCRIPTION

[0042] The application will be further described below with reference to the drawings. It is necessary to point out here that the following examples are only used to further illustrate the application and cannot be understood as limiting the protection scope of the application. Those skilled in the art can make some non-essential improvements and adjustments to the application according to the above content of the application, which still belongs to the protection scope of the application.

[0043] (1) The method of the application is developed on the depth learning standard framework 1.6.0. The important hyperparameters are as follows: the training learning rate lr is set to 2e-4, and the weight decay is set to 1e-3.

[0044] (2) When verifying the prediction performance of the method of the application, four commonly used public data sets, ShanghaiTec, UCF_CC_50, WorldExpo'10 and UCF_QNRF, are selected, which cover common sparse, dense, street, square and other crowd gathering scenes.

[0045] (3) When generating the density map label, a fixed Gaussian kernel normalized to 1 is used to blur each head mark in the image. The whole generation process can be defined as:

[0046]

[0047] Where x is the pixel position of the marked object, and N is the total number of head marks in the image. Each head mark can be represented by a delta function δ(x-x i ). Each delta function is convolved with a fixed-size Gaussian kernel G σ to generate the real density map F(x);

[0048] (4) The mean absolute error (MAE) and the root mean square error (RMSE) are used to evaluate the prediction accuracy and robustness of the method of the application; the frame rate (FPS) of the image with a size of 1280*720 per second is used to evaluate the processing speed of the method; the total number of parameters (unit: M) of the network model and the inference calculation amount (unit: GFLOPs) of the image are used to evaluate the complexity of the method of the application;

[0049] (5) The commonly used lightweight network for comparison of the method of the application is as follows:

[0050] Method 1: Method proposed by Zhang Y, Zhou D, Chen S et al., reference literature "Single-image crowd counting via multi-column convolutional neural network[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:589-597";

[0051] Method 2: Method proposed by Cao X, Wang Z, Zhao Y et al., reference literature "Scale aggregation network for accurate and efficient crowd counting[C] / / Proceedings of the European Conference on Computer Vision (ECCV).2018:734-750";

[0052] Method 3: Method proposed by Wang, P et al., reference literature "MobileCount: An efficient encoder-decoder framework for real-time crowd counting. Neurocomputing, 2020. 407: p. 292-299";

[0053] (6) As can be seen from Table 3 and Table 4, compared with other lightweight networks, the method proposed in the application has obtained lower MAE and MSE on most data sets, indicating that the method of the application can better adapt to the scale change in the scene, and at the same time Figure 5 The part of the result listed in the schematic diagram also shows that the method of the application can generate high-quality density map.

[0054] (7) The method of the application and the other three lightweight network methods are all tested in speed on three GPU platforms (GTX 1050Ti, RTX2080, RTX 2080Ti). The experimental results are shown in Table 5. Due to the lower inference complexity of the method of the application, the inference speed can reach up to 70.9FPS, which is much higher than that of other lightweight network methods, and the prediction accuracy is also better. The experimental results show that the method of the application achieves a better balance between crowd counting accuracy and inference complexity.

[0055] Table 1 Specific structure of local feature extractor (LFE)

[0056]

[0057] Table 2 Specific structure of density map prediction module (DMR)

[0058]

[0059] Table 3 Accuracy comparison of different methods on ShanghaiTech, UCF-QNRF, Mall data

[0060]

[0061] Table 4 Accuracy comparison of different methods on WorldExpo'10 data

[0062] Method S1 S2 S3 S4 S5 Avg. Parameter amount (M) Method 1 3.4 20.6 12.9 13.0 8.1 11.6 0.13 Method 2 2.6 13.2 9.0 13.3 3.0 8.2 1.39 Method 3 - - - - - - 3.4 The method of the invention 1.6 15.6 11.4 9.4 2.5 8.1 0.2

[0063] Table 5 Speed comparison of different methods

[0064]

Claims

1. A real-time crowd counting method incorporating scale perception, characterized in that The method comprises the following steps: (1) Local feature extraction of image: on the basis of VGG16, a local feature extractor (LFE) specially used for lightweight network is designed, mainly to reduce the number of network layers and channels to reduce the computational complexity; (2) Multi-scale information extraction: the multi-scale feature extractor (SFEM) is composed of 6 cascaded hybrid hollow convolution modules (HDC), each adjacent two HDC modules share the same weight, thereby controlling the model parameters while obtaining more rich scale information, and the specific structure of the HDC module is as follows: First, input features I ∈ R N×H×W×C are sent into a 1 × 1 convolutional layer for channel compression to obtain output features I0 ∈ R N ×H×W×(C / 4) , where N, H, W and C represent batch size, height, width and channel number respectively, and then I0 is sent into three cascaded and dilated convolutional layers D = {1, 2, 3} with coprime dilation rates, and if the output features of the three dilated convolutional layers are I1 ∈ R N ×H×W×(C / 4) , I2 ∈ R N×H×W×(C / 4) , I3 ∈ R N×H×W×(C / 4) , then the final output of the HDC module is: F HDC = C{I0, I1, I2, I3} + I (1) Wherein C represents a channel splicing operation, in addition, a residual connection is established between the spliced features and the HDC input features I to enhance the gradient back propagation; (3) Multi-scale feature information fusion: The multi-scale feature fusion module (SFFM) contains three efficient fusion modules (EAF) based on attention mechanism, aiming to effectively fuse multiple feature maps with different scale information. The multi-scale fusion module proposed in the application mainly fuses the feature maps from four different modules, including the output F LEF ∈R N×H×W×C of the LFE, and the outputs F HDC2 ∈R N×H×W×C , F HDC4 ∈R N×H×W×C , F HDC6 ∈R N×H×W×C of each two HDC modules. At the same time, in order to enhance the gradient back propagation, a long skip connection is established, and the final output of the multi-scale feature fusion module is as follows: F SFFM = E(E(E(F LEF ,F HDC2 ),F HDC4 ),F HDC6 )+F LEF (2) Wherein E represents the operation of the EAF module, and the specific implementation details are as follows: If two features are χ1∈R N×H×W×C and χ2∈R N×H×W×C , learn a weight for two inputs to fuse features more effectively, the weight learning process is as follows: ω1=G(χ1) (3) ω2=G(χ2) (4) ω'1=R{σ[f1[C(R(ω1),R(ω2));θ2]]} (5) ω'2=R{σ[f2[C(R(ω1),R(ω2));θ2]]} (6) Wherein G represents global average pooling, C represents splicing, f(;θ) is one-dimensional convolution operation, σ is a sigmoid function, R represents vector modification, and then the channel weight is applied to the two input features χ1 and χ2 for feature fusion, and the process is as follows: In addition, we adopt a normalization layer to process the fused feature F EAF and then send it to a 1x1 convolution to generate the final feature map F EAF ∈R N×H×W×C ; (4) Density map prediction: the density map prediction module (DMR) mainly uses the feature map output by the SFFM to predict the crowd density distribution map, which is composed of a series of convolution layers and deconvolution layers, wherein the three deconvolution layers are responsible for restoring the feature map to the same size as the original image, and finally a ReLU function is added at the output of the network to ensure that all pixel values in the density map are non-negative; (5) The method adopts a normalized Euclidean loss function (NEL) and a multi-level intersection-over-union loss function (MIoU) to jointly train the network model, the NEL is used to measure the error between the predicted value and the true value at each pixel, and meanwhile, the sample imbalance problem during training can be relieved, when calculating the multi-level intersection-over-union loss, firstly, the following formula is used to obtain the predicted crowd quantity C est and the real crowd quantity C gt : where D gt is the ground truth density map, D est is the predicted density map, (i, j) denotes the pixel position in the density map, and is defined by the IoU loss function as follows: Wherein ∩ represents intersection operation, and ∪ represents set operation, since there may be pixel deviation between the predicted position and the real position, a downsampling operation is performed to reduce the measurement error caused by the pixel deviation, and finally the MIoU calculation formula is as follows: Wherein, l represents that the downsampling is 1 / (2^(l-1)) of the original image, in the method disclosed by the application, l is set to 4 to obtain the best performance, and the target loss function Lc of the network training is defined as follows: L c = NL e + MIoU (12) Experiments show that the loss function can greatly improve the network performance.

2. A real-time crowd counting method incorporating scale perception as claimed in claim 1, wherein Improved local feature extractor LFE.

3. A real-time crowd counting method incorporating scale perception as claimed in claim 1, wherein Multi-scale information is extracted by the designed scale feature extractor SFEM.

4. The method for real-time crowd counting with scale perception as claimed in claim 1, wherein When the extracted multi-scale information is fused, the designed multi-scale information fusion module SFFM is used.

5. The method for real-time crowd counting with scale perception as claimed in claim 1, wherein The density map regression module is used to predict the output density map.

6. A real-time crowd counting method incorporating scale perception as claimed in claim 1, wherein A multi-level intersection-over-union loss function is used to guide the network to learn accurate crowd recognition and density value estimation.

7. A system for performing the real-time crowd counting method combining scale perception according to any one of claims 1 to 6.

Images