Crowd counting method and system based on depth information and saliency information

Abstract

The application discloses a kind of crowd counting method and system based on depth information and saliency information, comprising: collecting crowd sample image of specified area;The crowd sample image collected is input into the density map prediction model based on saliency information and depth information trained;Output total number in crowd sample image.The crowd saliency information is introduced into the field of crowd counting, and the head annotation point is used as the eye focus point, the visual saliency label of crowd counting is generated using Gaussian blur, and the visual saliency information of crowd counting is obtained by training and testing using deep learning network, to assist the training of crowd counting;Visual saliency information and depth information are combined to assist crowd counting, and the saliency information can be used to correct the depth information, reduce the interference caused by the area without crowd information, and improve the counting effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and more particularly, relates to a crowd counting method based on depth information and saliency information. BACKGROUND

[0002] The task of dense crowd counting is to estimate the number of people contained in an image or video. With the increase of global population and human social activities, large crowds often gather in public places, such as transportation hubs and entertainment venues, which poses a huge hidden danger to public safety. The task of dense crowd counting is widely used in video surveillance, traffic control and metropolitan safety, and researchers in various countries have carried out a large number of researches. The method of crowd counting can also be extended to similar tasks in other fields, such as cell number estimation in microscopic images in medicine, vehicle estimation in traffic congestion and extensive biological sample investigation, etc.

[0003] Traditional crowd counting methods can mainly be divided into detection and regression-based methods, and as the crowd density increases, these two methods are difficult to handle the serious occlusion problem between the crowds. Since deep learning models have strong feature extraction capabilities, research on crowd counting methods based on deep learning has achieved many excellent results. The current mainstream method is to use a convolutional neural network to predict the density map of the original image, and to calculate the number of people using the density map.

[0004] Wang et al. first introduced Convolution Neural Networks (CNN) into the field of crowd counting, and proposed an end-to-end CNN regression model suitable for dense crowd scenes. The model improves the AlexNet network, and replaces the last fully connected layer with a single neuron layer to directly predict the crowd number. The disadvantage is that it cannot count the personnel distribution in the scene, and the effect is not good in the case of dense crowd or complex scene. Zhang et al. proposed a multi-column convolutional neural network MCNN for crowd counting based on the multi-branch deep convolutional neural network. Each branch network uses a convolution kernel of different size to extract feature information of targets of different scales, and reduces the counting error caused by the different sizes of targets due to the change of viewing angle. Although the multi-branch structure counting network achieves good counting effect, the complexity of the multi-branch structure network model is high, which also brings some new problems. For example, the network model has too many parameters, the training is difficult, and the structure is redundant. Therefore, Li et al. proposed a dilated convolutional neural network model CSRNet suitable for dense crowd counting. CSRNet does not use the multi-branch network structure widely used in the past, but uses the VGG16 network without a fully connected layer as the front-end part of the network, and uses a 6-layer dilated convolutional neural network at the back end to form a single-channel counting network, which greatly reduces the parameter amount and reduces the training difficulty. At the same time, by means of the advantage of expanding the receptive field while maintaining the resolution of the input image, more image detail information is retained, so that the generated crowd density map has higher quality.

[0005] In order to solve the problem of large target size change caused by different distances between the camera and the crowd, attention has been paid to introducing auxiliary information to assist crowd counting. Shi et al. combined perspective information with crowd counting to improve counting accuracy. Perspective information presents the depth difference of the entire image, which has certain similarity with depth image. Xu et al. used the depth information of the image to segment the scene into long-range and near-range areas, and then applied different mechanisms (based on density map and based on detection) to estimate the counting results of the two areas to count the total number of people. Yang et al. used a pre-trained depth branch to provide depth information for crowd counting. Depth information reflects the density of the crowd to some extent, and implicitly contains scale change information, but ignores the problem that depth information outside the crowd area will affect the counting result. SUMMARY

[0006] The purpose of the present application is to solve the above problems, and provide a crowd counting method and system based on depth information and saliency information. The method combines visual saliency information and depth information to assist crowd counting. The saliency information can correct the depth information, reduce the interference caused by the area without crowd information, and improve the counting effect.

[0007] To achieve the above object, the present application is realized by the following technical scheme:

[0008] In the first aspect, the present application provides a crowd counting method based on depth information and saliency information, comprising:

[0009] Collecting crowd sample images of a specified area;

[0010] Inputting the collected crowd sample images into a trained density map prediction model based on saliency information and depth information;

[0011] Outputting the total number of people in the crowd sample images.

[0012] Further, the density map prediction model is constructed by the following method:

[0013] Using an image depth information prediction network to perform depth prediction on the input crowd sample images to obtain image depth information;

[0014] Inputting the input crowd sample images, their corresponding predicted saliency information and depth information into a crowd density map prediction network, using saliency information to correct depth information, using the corrected depth information to guide the training of the density map prediction network, and generating a density map prediction model.

[0015] Further, the saliency information is generated by predicting the saliency map of the input crowd sample image using a saliency prediction model, and the saliency prediction model is constructed by the following method:

[0016] Gaussian blurring the head label data corresponding to the input crowd sample image to generate a true value saliency map;

[0017] Using a visual saliency prediction network to predict the saliency information of the input crowd sample image to generate a predicted saliency map;

[0018] According to the predicted saliency map and the true value saliency map, a loss function is calculated, the network parameters are adjusted through gradient back propagation, and after iteration, a saliency prediction model is generated.

[0019] Further, the Gaussian blurring of the head label data corresponding to the input crowd sample image is Gaussian blurring using a Gaussian kernel function with a standard deviation of 19.

[0020] Further, the density map prediction network training comprises:

[0021] For the input crowd sample image R, its corresponding depth map D and saliency map S, at the lth layer of the encoder, let R l , D land S l respectively, are the output feature maps of the previous convolutional layers of the encoder, the depth information is corrected by the saliency features of the corresponding layers, and the correction method is:

[0022] V l = sigmoid(Φ s (S l ))

[0023] D l = V l ⊙D l

[0024] wherein, Φ s represents a 1×1 convolutional layer, V l is the weight of the lth layer of the encoder, which is calculated by the sigmoid function, and ⊙ represents element-level multiplication, which uses the weight V l to act on Dl, so as to highlight the crowd area and reduce the influence of depth information of non-crowd areas;

[0025] The corrected depth information D l is weighted to R l as

[0026] R l = R l ⊙D l

[0027] wherein, ⊙ is element-level multiplication;

[0028] After that, R l , D l and S l are input to the subsequent network.

[0029] Further, the density prediction network comprises an encoding module, a depth correction and embedding module, an enhanced multi-scale module and a decoding module.

[0030] The encoding module is used to extract multi-level features of the input image.

[0031] The depth correction and embedding module is used for correction and fusion of depth information.

[0032] The enhanced multi-scale module is used for multi-scale comprehensive feature extraction and fusion.

[0033] The decoding module is used to output a predicted density map with the same size as the input image.

[0034] Further, the encoder module is a pre-trained VGG16 network; the enhanced multi-scale module includes multi-branch 3x3 convolution with different expansion rates, and the dilated convolution provides a larger receptive field than the ordinary convolution operation; and the decoding module is a 7-layer dilated convolution network for outputting a predicted density map with the same size as the input image.

[0035] Further, the total number of people in the output crowd sample image includes

[0036] The total number of people in the crowd sample image is obtained by summing all pixel points in the predicted density map generated by the density prediction model.

[0037] In a second aspect, the present application further provides a crowd counting system based on depth information and saliency information, comprising a processor and a storage medium.

[0038] The storage medium is used to store instructions.

[0039] The processor is used to operate according to the instructions to perform the steps of the method of the first aspect.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] (1) The present method introduces crowd saliency information into the field of crowd counting, uses the Gaussian blur to generate the visual saliency label for crowd counting, and uses the deep learning network for training and testing to obtain the visual saliency information for crowd counting, which assists the training of crowd counting.

[0042] (2) The present method uses the combination of visual saliency information and depth information to assist crowd counting, uses the saliency information to correct the depth information, reduces the interference caused by the area without crowd information, and improves the counting effect. BRIEF DESCRIPTION OF DRAWINGS

[0043] Fig. 1 is a flowchart of a crowd counting method based on depth information and saliency information provided by the embodiment of the present application;

[0044] Fig. 2 is a schematic diagram of the overall network structure of crowd counting according to the embodiment;

[0045] Fig. 3 is a schematic diagram of the network structure of the enhanced multi-scale module according to the embodiment. DETAILED DESCRIPTION

[0046] The present application will be further described below in conjunction with the drawings and specific embodiments, and the following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0047] Embodiment 1

[0048] As Figs. 1-3 shown, a crowd counting method based on depth information and saliency information includes: collecting a crowd sample image of a specified area; inputting the collected crowd sample image into a trained density map prediction model based on saliency information and depth information; and outputting the total number of people in the crowd sample image.

[0049] In this embodiment, for the collected crowd sample image, a crowd counting method based on depth information and saliency information is adopted, and the application process is as Fig. 1 shown, and specifically involves the following steps:

[0050] Step 1) Gaussian blur is performed on the head label data corresponding to the input crowd sample image to generate a true value saliency map.

[0051] In this embodiment, the SALICON dataset is referred to, which contains 20,000 images selected from the Microsoft COCO dataset and is the largest dataset in the field of image eye focus point detection so far. However, the dataset does not use an eye tracker to record eye movement data, but uses an Amazon crowd sourcing marking platform to let the annotators click the positions they focus on with a mouse. For the crowd counting dataset, this is similar to the process of head labeling. Then, Gaussian blur is performed on all the preprocessed mouse click samples of the same image to generate a true value saliency map. For the head label data corresponding to the input sample image, Gaussian blur is performed using a Gaussian kernel function with a standard deviation of 19 to generate a true value saliency map.

[0052] Step 2) The saliency information of the input crowd sample image is predicted using a visual saliency prediction network to generate a predicted saliency map, and a loss function is calculated according to the predicted saliency map and the true value saliency map. The network parameters are adjusted through gradient back propagation, and after iteration, a saliency prediction model is generated. The model is used to predict the saliency map of the input sample image to generate predicted saliency information.

[0053] Step 3) The image depth information is predicted using an image depth information prediction network to obtain the image depth information.

[0054] In the embodiment of the application, the pre-trained depth information prediction network model is used to predict the depth of the input sample image, and the predicted image can well adapt to various scene layouts and show the distance change from different positions to the camera.

[0055] Step 4) inputting the input sample image, its corresponding predicted saliency information and depth information into the crowd density map prediction network, correcting the depth information by using the saliency information, guiding the training of the density map prediction network by using the corrected depth information, and generating a density map prediction model.

[0056] In this embodiment, the overall network structure of the crowd density map prediction network is as shown in the figure. Fig. 2 The input sample image and its corresponding predicted saliency information and depth information are inputted into the crowd density map prediction network. For the input sample image R, its corresponding depth map D and saliency map S, at the lth layer of the encoder, let R l , D l and S l be the output feature maps of the previous convolution layers of the encoder, respectively. The depth feature is corrected by using the saliency feature of the corresponding layer, and the correction method is as follows:

[0057] V l = sigmoid(Φ s (S l ))

[0058] D l = V l ⊙D l

[0059] wherein Φ s represents a 1x1 convolution layer, V l is the weight of the lth layer of the encoder, calculated by the sigmoid function, and ⊙ represents element-level multiplication. The weight V l is used to act on D l , so as to highlight the crowd area and reduce the influence of the depth information of the non-crowd area.

[0060] The corrected depth information D l is weighted to R l as follows:

[0061] R l = R l ⊙D l

[0062] wherein ⊙ is element-level multiplication.

[0063] Then, R l , D l and S l are inputted into the subsequent network.

[0064] To deal with the scale variation problem of the crowd, most previous works adopt a multi-column network architecture, such as MCNN uses three columns of subnetworks to extract features of different scales. However, the scale diversity of features is limited by the number of network columns. To solve this problem, the architecture idea of Inception is borrowed, and further use of dilated convolution is proposed to enhance the multi-scale module, which enhances the scale of the input feature map, as shown in Fig. 3 .

[0065] Dilated convolution provides a larger receptive field than ordinary convolution operation, which can capture the region around the boundary and more rich context information than ordinary convolution. In the crowd scene, the head region of the person in the image always changes a lot. A single receptive field cannot adapt to the scale variation of the human head, and 3x3 convolution with dilation rate d of 1, 2, and 4 is used to capture features to better adapt to the diversified crowd distribution in the crowd scene.

[0066] Finally, the feature map is input into the decoder module, which is a 7-layer dilated convolution network that uses a larger receptive field to extract deeper important information for outputting a predicted density map with the same size as the input image. The Euclidean distance is used to measure the gap between the predicted result density map and the label, as shown in equation (1)

[0067]

[0068] In equation (1), X i is the input image, F(X i ) is the estimated density map, D(X i ) is the true density map, and N is the number of training samples. Adjust the network parameters through gradient backpropagation, and iteratively train the density prediction network.

[0069] Step 5) When counting the number of people in a single image, use the density prediction model to generate a predicted density map for the image, and obtain the total number of people in the image by summing the pixel points of the predicted density map.

[0070] After training a stable density prediction model, use the model to generate a predicted density map for the input image. After obtaining the density map, the total number of people in the image is obtained by summing the pixel points.

[0071] Example 2

[0072] Based on the crowd counting method based on depth information and saliency information described in embodiment 1, the present embodiment provides a crowd counting system based on depth information and saliency information, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method described in embodiment 1.

[0073] Having described various embodiments of the application, it is to be understood that the above description is meant not to be exhaustive or limited by the various embodiments disclosed. Many modifications and variations are possible in light of the above teachings without departing from the scope and spirit of the disclosed embodiments. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A crowd counting method based on depth information and saliency information, characterized in that, The method comprises the following steps: collecting a crowd sample image of a specified area; inputting the collected crowd sample image into a trained density map prediction model based on saliency information and depth information; outputting the total number of people in the crowd sample image; the density map prediction model is constructed by the following method: using an image depth information prediction network to perform depth prediction on the input crowd sample image to obtain image depth information; inputting the input crowd sample image, its corresponding predicted saliency information and depth information into a crowd density map prediction network, using the saliency information to correct the depth information, using the corrected depth information to guide the training of the density map prediction network, and generating a density map prediction model; the training of the density map prediction network comprises: For an input human crowd sample image , a corresponding depth map , and a saliency map , at the first layer of the encoder, let , and be the output feature maps of the previous convolutional layers of the encoder, correct the depth features with the saliency features of the corresponding layers by the following method: ； ； wherein, represents a 1x1 convolutional layer, is the weight of the first layer of the encoder, denotes element-wise multiplication; corrected depth information to weighted as: ； wherein, is an element-wise multiplication; The output of the network is then , and input to a subsequent network.

2. The crowd counting method based on depth information and saliency information according to claim 1, characterized in that, the saliency information is generated by predicting a saliency map of the input crowd sample image using a saliency prediction model, and the saliency prediction model is constructed by the following method: performing Gaussian blur on the head label data corresponding to the input crowd sample image to generate a true value saliency map; using a visual saliency prediction network to predict the saliency information of the input crowd sample image to generate a predicted saliency map; calculating a loss function according to the predicted saliency map and the true value saliency map, adjusting the network parameters through gradient back propagation, and generating a saliency prediction model through iteration.

3. The crowd counting method based on depth information and saliency information according to claim 2, characterized in that, The Gaussian blur on the head label data corresponding to the input crowd sample image is performed using a Gaussian kernel function with a standard deviation of 19. 4.The crowd counting method based on depth information and saliency information according to claim 1, characterized in that, The density map prediction network comprises an encoding module, a depth correction and embedding module, an enhanced multi-scale module and a decoding module; the encoding module is used to extract multi-level features of the input image; the depth correction and embedding module is used for depth information correction and fusion; the enhanced multi-scale module is used for multi-scale comprehensive feature extraction and fusion; the decoding module is used to output a predicted density map with the same size as the input image.

5. The crowd counting method based on depth information and saliency information according to claim 4, characterized in that, The encoder module is a pre-trained VGG16 network; the enhanced multi-scale module comprises a 3x3 convolution with multiple branches and different expansion rates; and the decoding module is a 7-layer dilated convolution network used to output a predicted density map with the same size as the input image.

6. The crowd counting method based on depth information and saliency information according to claim 1, characterized in that, The output of the total number of people in the crowd sample image comprises using the density prediction model to generate a predicted density map of the crowd sample image, and obtaining the total number of people in the image by summing each pixel point in the predicted density map.

7. A crowd counting system based on depth information and saliency information, characterized in that: The system comprises a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to perform the steps of the method of any one of claims 1-6.

Images