Crowd counting method for construction scenes based on cross-layer connection of dilated convolution

Abstract

The present invention discloses a construction scene crowd counting method based on cross-layer connection of dilated convolution, which comprises the following steps: (1) obtaining a public crowd counting dataset and generating a true density map based on manual annotation; (2) establishing a construction scene crowd counting network CL-DCNN based on cross-layer connection of dilated convolution; (3) inputting a crowd image in the dataset into the crowd counting network CL-DCNN, which outputs a predicted density map of the image; (4) integrating and summing the output predicted density map to obtain the total number of people in the image, performing loss calculation between the output predicted density map and the true density map, and continuously iterating and updating the parameters in the crowd counting network; (5) inputting a test image in a construction scene into the trained crowd counting network, generating a crowd density map, and integrating and summing the map to obtain a crowd counting result. The present invention has good adaptability and high prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision technology, and particularly relates to a construction scene crowd counting method based on cross-layer connection of void convolution. Background Art

[0002] Construction sites are full of safety hazards. Too many people on site are more likely to cause major accidents. If the number and location of people on site can be grasped in time, safety hazards can be discovered earlier, timely and reasonable solutions can be made, and casualties can be reduced. It can be seen that crowd counting in construction scenarios is very meaningful.

[0003] At present, the methods for crowd counting can be roughly divided into three categories: (1) Detection-based methods, which use a moving window detector to identify people in the image and use this information for counting tasks. There are two ways of detection: global-based and local-based. After extracting global or local features with the detector, a classifier is trained, such as support vector machine, boosting and random forest. However, no matter which detection-based method is used, it is difficult to deal with the serious occlusion problem between people. (2) Regression-based methods, which are divided into two steps: the first step is to extract effective features from the crowd image. In the early stage, manual features such as edge features and texture features (Histogram of Directed Gradients HOG, Scale-Invariant Features SIFT, Local Binary Patterns LBP, Gray Level Co-occurrence Matrix GLCM, etc.) were used. The second step is to use various regression functions to estimate the number of people. Regression methods include linear regression, ridge regression and Gaussian process regression. This method directly regresses the crowd counting results from the features and cannot reflect the crowd density distribution. (3) Based on the convolutional neural network method, an end-to-end regression method is constructed using a convolutional neural network. The entire image is taken as input, and the density map of the crowd image is directly predicted. The density map is integrated and summed to obtain the crowd counting result.

[0004] Construction sites are mostly complex and changeable, and the crowds are often unevenly distributed. In addition, due to the different distances of the crowds from the camera equipment, the size of the people will also cause distortion of the perspective information. There is also serious occlusion between people and between people and the background. These problems have brought considerable difficulties to the task of crowd counting in construction scenes. Based on this, the present invention proposes a construction scene crowd counting method based on cross-layer connection of void convolution. Summary of the Invention

[0005] The purpose of the present invention is to provide a construction scene crowd counting method based on cross-layer connection of void convolution to solve the problems of perspective distortion in crowd images and severe occlusion between people.

[0006] The technical solution adopted by the present invention is: a construction scene crowd counting method based on cross-layer connection of void convolution, which specifically includes the following steps:

[0007] Step 1: Obtain a public crowd counting dataset and generate a real density map based on the annotation information corresponding to the crowd image;

[0008] Step 2: Establish a construction scene crowd counting network CL-DCNN based on cross-layer connections of void convolution;

[0009] Step 3: Input the crowd image in the dataset into the predicted density map of the CL-DCNN output image in step 2;

[0010] Step 4: Integrate and sum the output predicted density map to obtain the total number of people in the image, and calculate the loss between the output predicted density map and the true density map. Continuously update the parameters in the crowd counting network to obtain the optimal crowd counting network CL-DCNN.

[0011] In step 5, the test image of the construction scene is input into the optimal crowd counting network CL-DCNN trained in step 4, and the corresponding crowd distribution density map is output. The density map is integrated and summed to obtain the counting result of the crowd image of the construction scene.

[0012] The present invention is also characterized in that:

[0013] The dataset in step 1 includes crowd images and corresponding manually annotated files. The format of the manually annotated files is the coordinates of the position of each head in the image. Each dataset is sorted and classified separately, and the required number of images are randomly selected as the training set, and the rest are used as the test set.

[0014] According to the annotation information contained in the dataset, the original image is binarized according to the annotation information and then Gaussian smoothing is used to generate the true density map of the crowd image, as follows.

[0015] In a crowd image, first create a full 0 matrix with the same size as the original image, set pixel x i There is a head target at the position, which is expressed as δ(xx i ), and set the value of this point in the matrix to 1. Then an image with N head labels is defined as H(x), which is expressed by the following formula:

[0016]

[0017] Among them, x represents the two-dimensional coordinate, i represents the pixel x in the crowd image i There is a head at , N represents the total number of heads in the crowd image, and the crowd density map is defined as F(x), which is obtained by convolving H(x) with a two-dimensional Gaussian kernel.

[0018] At this time, the calculation formula of the density map F(x) is as follows.

[0019]

[0020]

[0021] Different datasets have different sparseness of population distribution, and the standard deviation of the Gaussian kernel is also different. When the population distribution in the image is relatively uniform, a Gaussian kernel with a fixed standard deviation is used, σ i =15;

[0022] When the crowd in the image is dense, the heads will appear in different sizes due to severe perspective distortion. In this case, the average of the distances to the k nearest neighbor heads is used as the standard deviation of the Gaussian kernel. That is, Gaussian kernels with different standard deviations are used for heads at different positions in the same crowd image, = 3:

[0023] σ i =βd i (4)

[0024]

[0025] Among them, * represents the convolution operation, β is the coefficient factor, d i Represents pixel x i The average distance of the k nearest neighbor heads at the head, Represents pixel x i Head and pixel x j The distance between people's heads.

[0026] Step 2 is as follows:

[0027] Step 2.1: Construct a crowd counting network for construction scenes based on cross-layer connections of dilated convolutions and determine the network structure, depth, and training method.

[0028] Step 2.2: The CL-DCNN described in step 2.1 consists of two parts: a front-end network and a back-end network. The front-end network uses the first 10 convolutional layers of VGG16 as feature extractors to generate deep feature maps. The back-end network performs two-dimensional dilated convolution on the deep feature maps to expand the receptive field, extract deeper semantic features, and splice them with the shallow feature maps in the channel dimension to achieve multi-layer feature fusion. The fused feature maps are then upsampled to obtain a density map of the same size as the original image.

[0029] Step 3 is as follows:

[0030] Input the training set images obtained from the dataset in step 1 into the crowd counting network CL-DCNN in step 2.1 to output the predicted density map.

[0031] Step 4 is as follows:

[0032] Step 4.1: Integrate and sum the predicted density map output in step 3 to obtain the total number of people. Calculate the mean square error between the predicted density map generated in step 3 and the true density map generated in step 1 as the counting loss. In addition to calculating the crowd counting loss, the structural similarity loss function between the predicted density map and the true density map is also calculated to measure the quality of the predicted density map.

[0033] The loss function of the crowd counting network CL-DCNN described in step 2.1 Loss by counting and SSIM loss function The calculation formula is as follows.

[0034]

[0035]

[0036]

[0037] in, Represents the population counting loss, calculates the sum of squared errors between the predicted density map and the true density map, M represents the total number of samples, represents the SSIM loss, which is used to constrain the quality of the predicted density map; α represents the weight factor, which is a hyperparameter. i represents the i-th input sample, θ represents the model parameter set, F(X i ,θ) represents the predicted density map of the i-th sample, D i Represents the true density map of the i-th sample;

[0038] Step 4.2: According to the loss function designed in step 4.1 The Adam optimizer is used to train the crowd counting network CL-DCNN in step 2.1. The first-order moment estimation and second-order moment estimation of the gradient are used to dynamically adjust the learning rate of each parameter. After bias correction, the learning rate of each iteration has a certain range, making the parameters relatively stable.

[0039] Step 4.3: After the training is completed, the optimal crowd counting network CL-DCNN is obtained.

[0040] Given the predicted density map p and the corresponding true density map q, their SSIM calculation formula is as follows:

[0041]

[0042] c1=(k1L) 2 ,c2=(k2L) 2

[0043] Among them, μp is the pixel mean of the predicted density map p, μ q is the pixel average of the true density map q, is the pixel variance of the predicted density map p, is the pixel variance of the true density map q, σ pq is the pixel covariance between the predicted density map p and the true density map q; c1 and c2 are constants used to maintain stability, L is the dynamic range of the input image pixel value, where L = 255, k1 = 0.01, k2 = 0.03; the value range of SSIM is [-1, 1]. The higher the similarity between the predicted density map p and the true density map q, the higher the SSIM measurement index value between them.

[0044] The beneficial effects of the present invention are that the construction scene crowd counting method based on cross-layer connection of dilated convolution is based on CNN features, can not only process images of any scale, but also has high detection accuracy. The dilated convolution is used to expand the receptive field and can adapt to crowds of varying sizes within the image. By using cross-layer connection of shallow feature maps and deep feature maps, the effect of multi-layer feature fusion can be achieved, and the features of each layer can be fully utilized. Finally, SSIM (Structural Simlarity) loss is used to further constrain the quality of the generated density map. This method has good adaptability and high prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of the overall implementation of the construction scene crowd counting method based on cross-layer connection of void convolution of the present invention;

[0046] Figure 2 Schematic diagram of a training model for a construction scene crowd counting method based on cross-layer connection of dilated convolutions of the present invention;

[0047] FIG3( a ) is a diagram showing the detection effect of the construction scene crowd counting method based on cross-layer connection of dilated convolution of the present invention applied to an actual scene;

[0048] Figure 3(b) is the density map predicted by the crowd counting network in Figure 3(a). DETAILED DESCRIPTION

[0049] The present invention is further illustrated below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims appended to this application.

[0050] The overall embodiment flow chart of the present invention is shown in FIG. Figure 1-Figure 2, a construction scene crowd counting method based on cross-layer connection of void convolution, includes the following steps:

[0051] S1. Obtain a public crowd counting dataset and generate a real density map based on the annotation information corresponding to the crowd image;

[0052] S2. Establish a construction scene crowd counting network CL-DCNN (Cross-Layer Dilated Convolution Neural Network) based on cross-layer connection of void convolution;

[0053] S3, input the crowd image in the dataset into the predicted density map of the CL-DCNN output image in step S2;

[0054] S4. Integrate and sum the output predicted density map to obtain the total number of people in the image, and perform loss calculation between the output predicted density map and the true density map, and continuously update the parameters in the crowd counting network;

[0055] S5. Input the test image of the construction scene into the crowd counting network trained in step S4, generate a crowd density map, and integrate and sum it to obtain the crowd counting result.

[0056] Furthermore, the specific steps of step S1 are as follows:

[0057] S11. Obtain multiple public crowd counting datasets from the internet, which include crowd images and corresponding manually annotated files. The format of the annotation files is the coordinates of each head position in the image. Organize and categorize each dataset separately, randomly select the required number of images as the training set, and the remaining number as the test set.

[0058] S12. Based on the manual annotation information contained in the data set, the original image is binarized according to the annotation information and then Gaussian smoothing is used to generate a true density map of the crowd image.

[0059] In a crowd image, first create a full 0 matrix with the same size as the original image, set pixel x i There is a head target at the position, which is expressed as δ(xx i ), and set the value of this point in the matrix to 1, then an image with N head labels is defined as H(x), which is expressed by the following formula.

[0060]

[0061] Where x represents the two-dimensional coordinate, i represents the pixel x in the crowd image i There is a head at , and N represents the total number of heads in the crowd image. The crowd density map is defined as F(x), which is composed of H(x) and the two-dimensional Gaussian kernel Gσ () convolution is obtained,

[0062] At this time, the calculation formula of the density map F(x) is as follows.

[0063]

[0064]

[0065] Different datasets have different sparseness of population distribution, and the standard deviation of the Gaussian kernel is also different. When the population distribution in the image is relatively uniform, a Gaussian kernel with a fixed standard deviation is used, σ i =15;

[0066] When the crowd in the image is dense, the heads will appear in different sizes due to severe perspective distortion. In this case, the average of the distances to the k nearest neighbor heads is used as the standard deviation of the Gaussian kernel. That is, Gaussian kernels with different standard deviations are used for heads at different positions in the same crowd image, = 3:

[0067] σ i =βd i (4)

[0068]

[0069] Among them, * represents the convolution operation, β is the coefficient factor, d i Represents pixel x i The average distance of the k nearest neighbor heads at the head, Represents pixel x i Head and pixel x j The distance between people's heads.

[0070] Furthermore, the specific steps of step S2 are as follows:

[0071] S21. Build a construction scene crowd counting network CL-DCNN based on cross-layer connections of void convolution, and determine the network structure, depth, and training method;

[0072] S22. The CL-DCNN described in step S21 consists of a front-end network and a back-end network. The front-end network uses the first 10 convolutional layers of VGG16 as feature extractors to generate deep feature maps. The back-end network performs dilated convolution on the deep feature maps to expand the receptive field, extract deeper semantic features, and splice them with the shallow feature maps in the channel dimension to achieve multi-layer feature fusion. The fused feature maps are then upsampled to obtain a density map with the same size as the original image.

[0073] Atrous convolution, also known as dilated convolution or expanded convolution, adds holes to the standard feature map to increase the receptive field. Compared to normal convolution, atrous convolution has an additional hole ratio, while the hole ratio of normal convolution is 1. With the same number of parameters, atrous convolution can increase the receptive field. A large receptive field helps identify targets in areas with different crowd densities and avoids missing pedestrians with large heads in sparse areas.

[0074] The specific parameters of the convolutional layer of CL-DCNN are shown in Table 1. The front-end Conv1-2 indicates that the original image is convolved with the first set of two identical convolution kernels, with a kernel size of 3*3, a number of 64 kernels, and a dilation ratio of 1. The dilated convolution with a dilation ratio of 1 is the same as the ordinary convolution, and 64 feature maps are obtained. A maximum pooling layer is added after Conv1-2 for downsampling, and the size of the feature map becomes 1 / 2 of the original image; Conv2-2 means that the obtained feature map is convolved with a second set of two identical convolution kernels, the convolution kernel size is 3*3, the number of convolution kernels is 128, and 128 feature maps are obtained. A maximum pooling layer is added after Conv2-2 for downsampling. At this time, the feature map size becomes 1 / 4 of the original image; the obtained feature map is input into Conv3-1 to Conv3-3 to obtain 256 feature maps, and then a maximum pooling layer is added. At this time, the feature map size becomes 1 / 8 of the original image; the obtained feature map is then input into Conv4-1 to Conv4-3 to obtain 512 feature maps. The obtained feature map is then input into the dilated convolution with 256 convolution kernels. The size of the feature map remains unchanged at 1 / 8 of the original image. The feature map is fused with the feature map obtained by Conv3-3 in the channel dimension, and the fused feature map is upsampled. The size of the feature map becomes 1 / 4 of the original image. The obtained feature map is fused with the feature map obtained by the front-end Conv2-2, and upsampled to obtain a feature map of 1 / 2 of the original image. The obtained feature map is fused with the feature map obtained by the front-end Conv1-2, and then upsampled to obtain a feature map of the same size as the original image. Finally, the number of channels is changed through a 1*1 convolution kernel to become a single-channel density map.

[0075] Table 1 Specific parameters of each convolution layer of CL-DCNN

[0076]

[0077]

[0078] Furthermore, the specific steps of step S3 are as follows:

[0079] S31, input the training set images obtained from the data set of step S1 into the crowd counting network CL-DCNN of step S21 to output the predicted density map.

[0080] Furthermore, the specific steps of step S4 are as follows:

[0081] In step S41, the predicted density map output from step S31 is integrated and summed to obtain the total number of people. The mean squared error (MSE) between the predicted density map generated in step S31 and the true density map generated in step S1 is calculated as the counting loss. To overcome the severe background interference in crowd images, in addition to calculating the crowd counting loss, the SSIM loss function is also used to measure the quality of the predicted density map and improve the counting accuracy of the model.

[0082] The loss function of the crowd counting network CL-DCNN described in step S21 Loss by counting and SSIM loss function The composition, specifically, the calculation formula is as follows.

[0083]

[0084]

[0085]

[0086] in, Represents the population counting loss, calculates the sum of squared errors between the predicted density map and the true density map, M represents the total number of samples, represents the SSIM loss, which is used to constrain the quality of the predicted density map. α represents the weight factor, which is a hyperparameter; X i represents the i-th input sample, θ represents the model parameter set, F(X i ,) represents the predicted density map of the i-th sample, D i Represents the true density map of the i-th sample.

[0087] Given the predicted density map p and the corresponding true density map q, their SSIM calculation formula is as follows:

[0088]

[0089] c1=(k1L) 2 ,c2=(k2L) 2

[0090] Among them, μ p is the pixel mean of the predicted density map p, μ q is the pixel average of the true density map q, is the pixel variance of the predicted density map p, is the pixel variance of the true density map q, σ pq is the pixel covariance between the predicted density map p and the true density map q. c1 and c2 are constants used to maintain stability. L is the dynamic range of the input image pixel values. Here, L = 255, k1 = 0.01, and k2 = 0.03. The SSIM value range is [-1, 1]. The higher the similarity between the predicted density map p and the true density map q, the higher the SSIM metric index value between them.

[0091] S42. Use the Adam optimizer to train the model based on the loss function designed in S41. Use the first-order moment estimation and second-order moment estimation of the gradient to dynamically adjust the learning rate of each parameter. After bias correction, the learning rate of each iteration has a certain range, making the parameters relatively stable.

[0092] S43. After the training is completed, the optimal crowd counting network CL-DCNN is obtained.

[0093] Furthermore, the specific steps of step S5 are as follows:

[0094] S51. Input the crowd image of the construction scene collected in advance as a test image into the optimal crowd counting network CL-DCNN described in step S43, output the corresponding crowd distribution density map, and obtain the counting result of the construction scene crowd image by integrating and summing the density map.

[0095] As shown in Figure 3(a)-(b), we input the construction scene pictures taken in the actual scene into the optimal crowd counting network CL-DCNN described in step S43, detect the crowd counting results in the pictures, and achieve the ideal effect.

[0096] In summary, the present invention discloses a method for counting crowds in construction scenes based on cross-layer connections of dilated convolutions, and mainly describes the training method. First, a public crowd counting dataset is obtained, and a true density map is generated based on manual annotations; secondly, a construction scene crowd counting network CL-DCNN (Cross-Layer Dilated Convolution Neural Network) with cross-layer connections based on dilated convolutions is established; then, the crowd image in the dataset is input into the crowd counting network CL-DCNN to output the predicted density map of the image; the predicted density map is integrated and summed to obtain the total number of people in the image, and the output predicted density map is compared with the true density map to perform loss calculation, and the parameters in the crowd counting network are continuously iterated and updated; finally, the test image in the construction scene is input into the trained crowd counting network to generate a crowd density map and integrate and sum to obtain the crowd counting result.

[0097] The construction scene crowd counting method provided by the present invention, based on cross-layer connections with dilated convolutions, is based on CNN features and can process images of any scale while achieving high detection accuracy. Dilated convolutions are used to expand the receptive field, adapting to crowds of varying sizes within the image. Cross-layer connections between shallow and deep feature maps achieve multi-layer feature fusion, fully utilizing features from each layer. Finally, the SSIM loss is used to further constrain the quality of the generated density map. This method exhibits excellent adaptability and high prediction accuracy.

Claims

1. A construction scene crowd counting method based on cross-layer connection of void convolution, characterized by: The specific steps include: Step 1: Obtain a public crowd counting dataset and generate a real density map based on the annotation information corresponding to the crowd image; According to the annotation information contained in the dataset, the original image is binarized according to the annotation information and then Gaussian smoothing is used to generate the real density map of the crowd image, as follows: In a crowd image, first create a full 0 matrix with the same size as the original image, and set the pixel There is a head target at , which is represented as , and set the value of the point to 1 in the matrix, then a An individual head-labeled image is defined as , expressed as follows: in, represents the two-dimensional coordinates, Indicates the pixel in the crowd image There are heads everywhere, Representing the total number of heads in the crowd image, the crowd density map is defined as ,Depend on Convolving with a two-dimensional Gaussian kernel yields, At this time, the density map The calculation formula is as follows: （2） Different datasets have different sparseness of population distribution, and the standard deviation of the Gaussian kernel is also different. When the population distribution in the image is relatively uniform, a Gaussian kernel with a fixed standard deviation is used. ; When the crowd in the image is dense, the heads of people will appear in different sizes due to severe perspective distortion. The average value of the distances between the nearest neighbor heads is used as the standard deviation of the Gaussian kernel, that is, Gaussian kernels with different standard deviations are used for heads at different positions in the same crowd image. : （5） in, represents the convolution operation, is the coefficient factor, Representative pixels Head The average distance between the nearest neighbor heads, Represents pixels Heads and pixels The distance between the heads of people; Step 2: Establish a construction scene crowd counting network CL-DCNN based on cross-layer connections of void convolution; the details are as follows: Step 2.1: Construct a crowd counting network for construction scenes based on cross-layer connections of dilated convolutions and determine the network structure, depth, and training method. Step 2.2: The CL-DCNN described in step 2.1 consists of a front-end network and a back-end network. The front-end network uses the first 10 convolutional layers of VGG16 as feature extractors to generate deep feature maps. The back-end network performs two-dimensional dilated convolution on the deep feature maps to expand the receptive field, extract deeper semantic features, and concatenate them with the shallow feature maps in the channel dimension to achieve multi-layer feature fusion. The fused feature maps are then upsampled to obtain a density map with the same size as the original image. Step 3: Input the crowd image in the dataset into the predicted density map of the CL-DCNN output image in step 2; Step 4: Integrate and sum the output predicted density map to obtain the total number of people in the image, and perform loss calculation between the output predicted density map and the true density map. Continuously update the parameters in the crowd counting network to obtain the optimal crowd counting network CL-DCNN. The details are as follows: Step 4.1: Integrate and sum the predicted density map output in step 3 to obtain the total number of people. Calculate the mean square error between the predicted density map generated in step 3 and the true density map generated in step 1 as the counting loss. In addition to calculating the crowd counting loss, the structural similarity loss function between the predicted density map and the true density map is also calculated to measure the quality of the predicted density map. The loss function of the crowd counting network CL-DCNN described in step 2.1 Loss by counting and SSIM loss function The calculation formula is as follows: in, Represents the population counting loss, calculates the sum of squared errors between the predicted density map and the true density map, M represents the total number of samples, Represents SSIM loss, which is used to constrain the quality of the predicted density map; Represents the weight factor, which is a hyperparameter. Indicates the input samples, represents the set of model parameters, Indicates the The predicted density plot of samples, Indicates the The true density map of samples; Step 4.2: According to the loss function designed in step 4.1 The Adam optimizer is used to train the crowd counting network CL-DCNN in step 2.

1. The first-order moment estimation and second-order moment estimation of the gradient are used to dynamically adjust the learning rate of each parameter. After bias correction, the learning rate of each iteration has a certain range, making the parameters relatively stable. Step 4.3: After training, the optimal crowd counting network CL-DCNN is obtained. In step 5, the test image of the construction scene is input into the optimal crowd counting network CL-DCNN trained in step 4, and the corresponding crowd distribution density map is output. The density map is integrated and summed to obtain the counting result of the crowd image of the construction scene.

2. The construction scene crowd counting method based on cross-layer connection of void convolution according to claim 1 is characterized in that The dataset in step 1 includes crowd images and corresponding manually annotated files. The format of the manually annotated files is the coordinates of the position of each head in the image. Each dataset is sorted and classified separately, and the required number of images are randomly selected as the training set, and the rest are used as the test set.

3. The construction scene crowd counting method based on cross-layer connection of void convolution according to claim 1 is characterized in that Step 3 is as follows: Input the training set images obtained from the dataset in step 1 into the crowd counting network CL-DCNN in step 2.1 to output the predicted density map.

4. The method for counting people in construction scenes based on cross-layer connection of dilated convolution according to claim 1 is characterized in that: Given a predicted density map and the corresponding true density map , their SSIM calculation formula is as follows: in, is the predicted density map The average pixel value, is the true density map The average pixel value, is the predicted density map The pixel variance of is the true density map The pixel variance of is the predicted density map and the true density map Pixel covariance of ; is a constant used to maintain stability. is the dynamic range of the input image pixel values, where , ; The value range of SSIM is [-1,1], and the predicted density map and the true density map The higher the similarity, the higher the SSIM index value between them.

Images