Snowfield passenger flow statistical method based on deep learning

Abstract

A snowfield passenger flow statistical method based on deep learning relates to the technical field of crowd flow statistics in dense areas such as snowfields, and comprises the following steps: segmenting an image into different areas according to the difference of crowd density in the image, adopting a multi-branch structure, and carrying out crowd counting on the corresponding density area by each branch; for an area with relatively low density, accurate counting of the area can be realized if multi-scale information is extracted; a branch model aiming at low density is designed, the core of the branch model is that the receptive field range of neurons is expanded by using expansion convolution, and meanwhile, multi-scale information is obtained through aggregation by using Dense connection, so that accurate estimation of a multi-scale head is realized; for a high-density area, a branch model which is based on Transform and aims at high density is provided. According to the invention, the final density map is obtained by selecting the most suitable branch result for each area according to the local density information of the density map generated by double branches, so that high-accuracy counting is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of passenger flow statistics in densely populated areas such as ski resorts, and in particular to a deep learning-based passenger flow statistics method for ski resorts. Background Art

[0002] The core areas of cities and many public places have high population densities, which are prone to large-scale crowds. If an abnormal event occurs in a crowded crowd, the crowd will scatter, which is very likely to cause a stampede. At the very least, it will cause property losses, and at the worst, it may cause casualties, which is regrettable. Therefore, it is very necessary to monitor these places and analyze them to promptly warn of events that have occurred or may occur. In addition, compared with the method of using manpower to monitor and analyze these places, the crowd analysis system based on video, which reasonably uses image processing, deep learning and other technologies to automatically monitor and analyze, is obviously more efficient.

[0003] The traditional crowd counting method based on detection or regression is completely correct in its idea, but its performance is mainly limited by the limitation of manually extracting features, and the extracted features have weak representation ability. Therefore, it was quickly replaced by crowd counting methods based on CNN (convolutional neural network) that automatically select features.

[0004] Wang et al. first introduced CNN into the field of crowd counting. The model they designed is essentially AlexNet. They changed the number of neurons in the last layer of the original model, so that AlexNet was used for image classification and used to regress the number of people. In the field of crowd counting, the most important problem is the multi-scale problem. In order to better solve this problem, Boominathan et al. proposed a crowd counting network with a dual-branch structure. The principle is to use the large receptive field of neurons in the deep network of CNN to obtain high-level semantic information, while the shallow layer can capture more underlying information. Through this combination of information at different levels, it can effectively adapt to changes in head scale and perspective, and improve the counting effect. Crowd counting based on CNN can basically count sparse areas and relatively not very dense scenes accurately. However, in highly crowded scenes, its performance is still limited due to background interference, occlusion between individuals, and uneven distribution. Summary of the invention

[0005] In order to solve the problem that the method of using CNN and fusing multi-scale information can effectively count sparse areas, but cannot effectively and accurately estimate dense areas; at the same time, the Transformer using the self-attention mechanism can obtain the connection between inputs from a global perspective, so as to effectively estimate dense scenes; however, the technical problem that its counting effect in sparse areas is limited due to the lack of position information, the present invention provides a ski resort passenger flow counting method based on deep learning.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] A method for counting passenger flow at a ski resort based on deep learning is characterized in that the method is as follows: according to the difference in crowd density in an image, different regions are divided, a multi-branch structure is adopted, and each branch counts the crowd in the corresponding density region respectively; for areas with low density, there are heads of various scales, and the occlusion between them is not serious. If multi-scale information is extracted, a more accurate count of the area can be achieved; a branch model for low density is designed, the core of which is to use dilated convolution to expand the receptive field range of neurons, and use Denseconnection to aggregate and obtain multi-scale information, so as to achieve accurate estimation of multi-scale heads; for high-density areas, a branch model for high density based on Transformer is proposed. Transformer can obtain global information due to its internal self-attention mechanism, so as to effectively count high-density areas; a method of selecting the most suitable branch result for each area according to the local density information of the density map generated by dual branches is proposed to obtain the final density map, so as to obtain a high-accuracy count.

[0008] The beneficial effects of the present invention are as follows: the method of using CNN and fusing multi-scale information can effectively count sparse areas, but cannot effectively and accurately estimate dense areas; at the same time, the Transformer using the self-attention mechanism can obtain the connection between inputs from a global perspective, thereby effectively estimating dense scenes; however, due to the lack of position information, its counting effect in sparse areas is limited. The present invention proposes a crowd counting method based on CNN and Transformer, and adopts a dual-branch form of CNN and Transformer in the model structure. The two branches each generate a density estimation map, and an adaptive selection module is used to select a suitable branch result for each area according to the local density information, so as to obtain a higher quality density estimation map and thus improve the counting effect. The crowd counting algorithm proposed in the present invention is trained and tested on the ShanghaiTech and UCF-QNRF datasets, respectively. Compared with the classic algorithm CSRNet, the model of the present invention has obvious reduction in MAE and MSE; it decreases by 19.94% and 22.52% on the ShanghaiTech dataset part A, 34.91% and 31.25% on the ShanghaiTech dataset part B, and 21.95% and 42.30% on the UCF-QNRF dataset. The experimental results show the improvement effect of Transformer on counting performance and the effectiveness and excellence of the crowd counting method proposed in the present invention. DETAILED DESCRIPTION

[0009] The invention discloses a method for counting passenger flow at a ski resort based on deep learning, which is characterized in that the method comprises the following steps: dividing the image into different regions according to the difference in crowd density, adopting a multi-branch structure, and each branch counts the crowd in the corresponding density region; in the region with lower density, there are heads of various scales, and the mutual occlusion between the heads is not serious. If the multi-scale information is extracted, a relatively accurate count of the region can be achieved; a branch model for low density is designed, the core of which is to use dilated convolution to expand the receptive field range of neurons, and to use Denseconnection to aggregate and obtain multi-scale information, so as to achieve accurate estimation of multi-scale heads; for high-density regions, a branch model for high density based on Transformer is proposed, and Transformer can obtain global information due to its internal self-attention mechanism, so as to effectively count the high-density regions; a method for selecting the most suitable branch result for each region according to the local density information of the density map generated by double branches is proposed to obtain the final density map, so as to obtain a high-accuracy count.

[0010] The overall structure of the model of the method of the present invention uses VGG16 as the basic feature extractor, which is then divided into two branches, and the features obtained from VGG16 are respectively sent to Dense ASPP and Transformer. The two branches respectively send the features further obtained by Transformer or Dense ASPP to their respective regression headers to generate corresponding initial density maps. After that, the two initial density maps are divided into regions through the Adaptive select module. In this part, the initial density map is divided into blocks, and the number of local people is extracted to generate the corresponding attention value. After that, the initial density map of each branch is segmented according to the attention value of each region, and finally the density maps after attention weighting are added to obtain the final density map. It should be emphasized that the present invention takes into account that too many pooling layers will lead to the loss of underlying and local information, which is not conducive to the prediction of highly dense areas. Therefore, the underlying feature extractor used in the present invention only uses the first 10 layers of VGG16, that is, the fourth pooling layer and the subsequent convolutional layer and fully connected layer in its original structure are removed, so the feature size of its output is 1 / 8 of the input image size. At the same time, the network structure of the two regression headers is the same, both consisting of three convolutional layers. The convolution kernel size of the first two layers is 3x3, and the convolution kernel size of the last layer is 1x1. At the same time, each convolutional layer is followed by a BatchNorm layer and a nonlinear activation function Relu. The number of output channels of the three convolutional layers is 256, 64, and 1 respectively.

[0011] Transformer is composed of multiple Transformer blocks. Let the input of Transformer block be Z, the normalized output of the first layer be Z′, and the final output of Transformer block be Z″. The input and output calculation process of each Transformer block can be written as Equation (3-1) and Equation (3-2).

[0012] Z′=LN(MHA(Z)+Z) (3-1)

[0013] Z″=LN(Z′+FFN(Z′)) (3-2)

[0014] In formulas (3-1) and (3-2), MHA is a multi-head attention module, FFN is a feed forward network, and LN is a layer normalization module. The specific principles of these modules have been explained in detail in Chapter 2 and will not be repeated here. There are two short-circuit operations that should be noted. Before each layer normalization, the input and output of the previous module need to be short-circuited to utilize features at multiple levels.

[0015] The calculation process in the multi-head attention module (Multi head attention MHA) is shown in formulas (3-3) and (3-4).

[0016]

[0017] Q=ZW q K=ZW K V=ZW V (3-4)

[0018] In formula (3-4), Z is the input of multi-head attention, Q, K, and V are query, key, and value, respectively, which are respectively generated by input Z and W. Q , W K , W V It is obtained by matrix multiplication. From formula (3-2), it can be seen that the output of multi-head attention is the dot product of Q and K and the weighted addition of V normalized by softmax. is the scale factor, which ensures that the variance of the dot product of Q and K is always 1 regardless of how the lengths of the Q, K, and V vectors change.

[0019] Assume that the layer normalized input is Z, the mean is μ, and the variance is σ, then the calculation process is shown in equations (3-5) and (3-6).

[0020]

[0021]

[0022] In formula (3-5) and (3-6), z i is the element in Z, n is the number of elements in Z, after obtaining the mean of input Z as μ and variance σ, the output result of layer normalization is The calculation process is shown in formula (3-7).

[0023]

[0024] In formula (3-7), γ and β are the parameter vectors of scaling and translation that need to be learned, respectively, and have the same dimension as Z. ε is a small number to prevent division by zero errors. The calculation formula of the feedforward neural network (FFN) refers to formulas (2-4) and (2-5), which will not be expanded here.

[0025] In addition, as mentioned earlier, Transformer can only process sequence data. Therefore, before the 2D image features obtained from the VGG16 underlying feature extractor are sent to the Transformer block, they should be converted into a sequence form. This conversion process is patch embedding, and the conversion process in the present invention is similar to the form in DETR. It does not divide the patch in the image, but performs it on the acquired image features. Since the image feature size obtained from the underlying feature extractor is 1 / 8 of the original image size, if the patch size is directly set to 1, it will result in too many patches, too much calculation, and difficult training. The present invention uses the patch embed method to set the patch size to 2x2, so that the number of patches is reduced to 1 / 4 of the original, and the calculation amount is sharply reduced. The actual implementation process is to use a set of convolution kernels with K=2 and S=2 to convert the underlying features into sequences, while the number of channels remains unchanged and is still set to 512.

[0026] D represents the dilation rate of the convolution. The convolution kernels used in all convolutional layers are 3x3 in size except for the last layer which is 1x1. "c" represents the addition operation of features in the channel dimension. Like Transformer, Dense aspp is also composed of multiple blocks in cascade. Each block is composed of multiple dilated convolutions with different dilation rates. At the same time, these dilated convolutions are connected using dense connections to obtain relatively dense multi-scale information, which helps to predict multi-scale human heads in sparse scenes.

[0027] The working principle of the adaptive selection module used in the present invention is as follows: first, the initial density map is divided into blocks, and a fixed block size is used in the block division of the density map, and different input images will obtain different numbers of blocks. The present invention has experimentally determined that the optimal block size is 32x32. After that, the average pooling operation is used to extract the number of local people, and nonlinear mapping is performed through multi-layer convolution to obtain the attention value of each block. The initial density map of each branch is segmented according to the attention value of each area. Finally, the density maps after attention weighting are added to obtain the final density map.

[0028] "S" represents the Sigmoid function, "x" represents matrix element multiplication, "+" represents matrix element addition, and "1-" represents the operation of taking and adding 1. The basic calculation process of input and output is shown in (3-8), (3-9), and (3-10).

[0029] d bi =F i1 (P ave (D i , k(i)), θ i1 ) (3-8)

[0030] d ati =F i2 (d bi ,θ i2 ) (3-9)

[0031] D final =D c *(1-d atc )+D t *d att (3-10)

[0032] Where D i The density estimation maps generated for the two branches, i∈c,t, c,t represents CNN and Transformer respectively. ,D final where d bi is the generated upsampled block density map, and d ati is the resulting normalized attention map. Function F i1 Contains a 1×1 convolutional network. Function F i2 It consists of a 1×1 convolutional network with bilinear interpolation and sigmoid function. ii (j=1, 2) is the function F ij Learning parameters in the 1×1 convolution of (j=1,2). ave The formula is the average pooling operation, which is used to obtain the local average value of density estimation, and k(i) is the size of the pooling area. For the result of the Dense aspp branch, since it focuses on the low-density area, the output result of the Sigmoid function is negated and added with 1 to obtain its correct attention map. finalTo finally obtain a density map. The pooling method adopted by the present invention is a fixed pooling area method, and the area size is set to 32x32. Some papers use an adaptive average pooling method to generate a fixed number of blocks. The problem with this method is that the resolution of the graphics used in the training stage is usually much smaller than the image resolution during testing, which results in a larger area and more number of pooled areas due to its count during testing. That is to say, the adaptive average pooling method causes deviations during training and testing, which may cause the Adaptive select module to select non-optimal branches during testing, thereby affecting performance to a certain extent. The extent of this impact is mainly related to the difference in image size during training and testing.

[0033] In order to explain in detail the core advantages of the objective function used in the present invention, it is necessary to analyze the principles and existing problems of the existing different forms of objective functions. One of the most commonly used methods is to generate a true density map, calculate the error between the predicted density map and the true value density map, and guide the learning of the model by minimizing its overall error on the data set. This form of model learning method is very dependent on the quality of the generated density map. Therefore, in order to analyze the problems of this method, its implementation principle is described in detail below.

[0034] The most basic annotation information provided by the existing crowd counting dataset is the coordinates of the center point of the head. According to the coordinates of the center point of the head, the corresponding discrete point map can be generated, and the discrete point map can be expressed by formula (3-11).

[0035]

[0036] In formula (3-11), x i The coordinates of the center point of the head in the image, δ(xx i ) is located at x i The unit impulse response function at , N represents the number of heads in the image.

[0037] The discrete point map is obtained based on the labeled point information, but the discrete point map cannot be used directly for training the model. Therefore, each labeled point in the discrete point map needs to be blurred using a Gaussian kernel to obtain a smooth density map. Since the discrete points in the two-dimensional image are blurred, a two-dimensional Gaussian kernel function is required. The two-dimensional Gaussian kernel function is shown in formula (3-12).

[0038]

[0039] In formula (3-12), x and y are the two-dimensional coordinates in the image, and the parameter σ is used to control the radial range of the Gaussian kernel function. The larger the σ, the larger the smoothing range of the points with a value of 1 in the discrete point map. Suppose the final discrete point map after Gaussian blur is F(x), and its calculation formula is shown in the following formula (3-13).

[0040]

[0041] When generating a density map, it is necessary to consider the effect of perspective distortion on the size of the head. The heads that are close to the lens are larger and occupy more spatial pixels, while the heads that are far from the lens are smaller and occupy fewer spatial pixels. Therefore, when using the Gaussian kernel function, it is necessary to set different blur radii according to different head sizes, that is, to determine the parameter σ according to the size of each head in the image. The literature proposes to adaptively determine the parameter σ based on the average distance between each individual and the nearest neighbor. Suppose the average distance between each individual and the nearest neighbor is The density map generated using the adaptive Gaussian kernel method can be expressed as follows (3-14).

[0042]

[0043] In formula (3-14), β is the scaling factor, which is generally set to 0.3. One problem with using the adaptive Gaussian kernel is that for sparse scenes, the distance between heads is much larger than the head size, which results in the Gaussian kernel used being too large. To address this problem, reference 34 proposed a fixed Gaussian kernel method, which sets the Gaussian kernel corresponding to each head in the image to a fixed value to solve the problem of the Gaussian kernel being too large for nearby heads. However, there is still overlap of the Gaussian kernels corresponding to distant heads, resulting in the problem of low density map quality. At the same time, different fixed Gaussian kernel sizes should be used for different data sets. Generate a density map using the fixed Gaussian kernel method

[0044] From the above analysis, we can know that based on the method of generating density maps, whether it is the adaptive Gaussian kernel method or the fixed Gaussian kernel method, there are certain limitations. To address this problem, Ma et al. proposed to directly use the Bayes Loss of point annotation information to guide the learning of the model. In the literature, the discrete point map generated according to the point annotation information is converted into N "smoothed true value density maps", where N is the number of discrete points. The pixel value of each smoothed true value density map is the posterior probability of the corresponding annotation point at a given pixel position. However, this function also requires a Gaussian kernel to construct a likelihood function for each annotation point, and this loss is equivalent to an underdetermined set of equations with infinite solutions. Using this loss training may result in a predicted density map that is different from the true value density map.

[0045] In view of the problems of using the method of generating density maps and the Bayesian Loss of point annotation information, an objective function as shown in formula (3-15) is proposed. This objective function also does not require any Gaussian method to preprocess the point annotation information. It directly uses the discrete point map and guides the learning of the model by minimizing the optimal transmission loss from the density estimation map to the discrete point map distribution. As shown in formula (3-15), the objective function consists of three items, namely The total loss of Optimal transmission loss, Overall bias loss.

[0046]

[0047] In formula (3-15), z is a vectorized discrete point graph, as shown in formula (3-11). It is a vectorized density estimation map, which is the final output of the model. ||z||1 represents the 1-norm of z.

[0048] By calculation The 1-norm of the image is taken and the L1 loss between the two is calculated to measure the deviation between the actual number of heads and the predicted number of heads in each image, as shown in formula (3-16).

[0049]

[0050] The optimal transmission loss measures the difference between the two distributions, which is Vectorized true value point map, density prediction map, and the gap between the two distributions.

[0051]

[0052] In formula (3-17), C is the distribution from z to The transfer cost matrix, T is the transition cost matrix from distribution z to The transfer matrix U is the transfer matrix from distribution z to The collection of all possible ways. Literature research found that using This allows the model to fit well in dense areas, but performs poorly in sparse areas. To alleviate this problem, a Overall deviation loss. The overall deviation loss is essentially the difference between each pixel in the calculated point map and the density prediction map. The calculation formula is shown in formula (3-18). The overall deviation loss not only makes the model perform better in sparse areas, but also improves the stability of the training process.

[0053]

Claims

1. A method for counting the number of visitors to a ski resort based on deep learning, characterized in that: The method is as follows: according to the difference in crowd density in the image, it is divided into different areas, and a multi-branch structure is adopted, in which each branch counts the crowd in the corresponding density area; for the area with low density, there are heads of various scales, and the occlusion between them is not serious. If the multi-scale information is extracted, a more accurate count of the area can be achieved; a branch model for low density is designed, the core of which is to use dilated convolution to expand the receptive field of neurons, and use Denseconnection to aggregate and obtain multi-scale information, so as to achieve accurate estimation of multi-scale heads; for high-density areas, a branch model for high density based on Transformer is proposed. Transformer can obtain global information due to its internal self-attention mechanism, so as to effectively count high-density areas; a method of selecting the most appropriate branch result for each area according to the local density information of the density map generated by dual branches is proposed to obtain the final density map, so as to obtain a high-accuracy count.

Images