A crowd counting method based on scale interaction and efficient dense residual connection

Abstract

The application discloses a crowd counting method based on scale interaction and efficient dense residual connection, and comprises the following steps: step 1, shooting a crowd scene image, and dividing the image into a training set and a test set; step 2, constructing the training set; step 3, pre-processing the training set; step 4, constructing a crowd counting model based on scale interaction and efficient dense connection; step 5, inputting the training set into the crowd counting model to obtain a training model; and step 6, using the training model to perform density map prediction on a test image to obtain the number of people and crowd distribution of the image. The application adopts a scale interaction scheme to solve the problem of violent scale change in a monitoring scene, and further adopts an efficient dense connection scheme to improve the representation ability of the counting model to solve the problem of uneven crowd distribution in the monitoring scene; and the application constructs a fast and efficient crowd counting model, which can effectively solve the problems of crowd scale change and uneven distribution in the monitoring scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent traffic and intelligent monitoring, and particularly relates to a crowd counting method based on scale interaction and efficient dense residual connection. BACKGROUND

[0002] With the growth of population and the progress of technology, today's society puts forward more demands for intelligent monitoring and intelligent traffic. Intelligent monitoring needs to obtain the number and distribution of crowds in the monitoring scene in real time, so as to timely adjust security measures and prevention and control tasks. Intelligent traffic can provide reasonable travel suggestions and road planning for people's travel by monitoring the number of vehicles and crowds on the road in real time.

[0003] The purpose of crowd counting technology is to count the number and distribution of crowds in a scene, and accurate crowd number and distribution can be perfectly applied to the fields of intelligent monitoring and intelligent traffic. Early crowd counting methods mainly use detection methods, which use some artificially designed features combined with sliding window methods to detect moving individuals, and then sum to obtain the total number of people. This detection-based method is only suitable for scenes with a small number of crowd distribution. When the number of crowds increases, the occlusion phenomenon and scale change intensify, and this method will fail.

[0004] In order to overcome the defects of the above method, researchers propose a crowd counting method based on regression. This method uses various regressors such as Gaussian process regression and neural networks to learn the mapping from input images to crowd number. Although this method can effectively solve the occlusion problem, it can only output the number of crowds in the scene, and cannot output the spatial distribution information, which limits the practical application range of this method.

[0005] In view of this problem, researchers propose a crowd counting method based on density estimation. This method uses the powerful fitting ability of neural networks to learn the mapping between input images and crowd distribution density maps. The density map can not only reflect the number of crowds in the monitoring scene, but also reflect the distribution of crowds in the monitoring scene. SUMMARY

[0006] In order to solve the problem of severe crowd scale change and uneven crowd distribution in the monitoring scene, the present application provides a crowd counting method based on scale interaction and efficient dense residual connection, comprising the following steps:

[0007] Step 1: shoot crowd scene images, and divide the images into training set and test set;

[0008] Step 2: construct the training set;

[0009] Step 3: pre-process the training set;

[0010] Step 4: constructing a crowd counting model based on scale interaction and efficient dense connection;

[0011] Step 5: inputting the training set into the crowd counting model to obtain a training model;

[0012] Step 6: using the training model to perform density map prediction on a test image to obtain the number of people and crowd distribution of the image.

[0013] The specific content of each step is described in the embodiments.

[0014] The advantages and beneficial effects of the present application are as follows: a crowd counting method based on scale interaction and efficient dense residual connection is provided, the scale interaction scheme is used to solve the problem of severe scale change in the monitoring scene, and the efficient dense connection scheme is used to improve the representation ability of the counting model to solve the problem of uneven crowd distribution in the monitoring scene; a fast and efficient crowd counting model is constructed, which can effectively solve the problems of crowd scale change and uneven distribution in the monitoring scene.

[0015] At present, the main challenges of crowd counting are uneven crowd distribution and severe scale change. In view of the difficulties faced by the current technology, the present application proposes a crowd counting method based on scale interaction and efficient dense connection, which can quickly and accurately predict the crowd distribution in the monitoring scene. The main innovations of the present application are as follows: (1) a scale interaction scheme is proposed to solve the problem of severe crowd scale change in the scene; the scheme includes two groups of three rows of convolution, each row of convolution uses different size convolution kernels to obtain different receptive fields, and the outputs of the three rows of convolution are spliced to obtain the effect of scale interaction, then the spliced results are respectively input into the second group of three rows of different size convolution kernels to further obtain the interaction between different scale features; finally, the interaction results are adaptively fused to obtain multi-scale representation. (2) an efficient dense connection scheme is proposed to enhance the representation ability of the crowd counting model to solve the problem of uneven crowd distribution in the monitoring scene; the scheme uses the scale interaction module as the basic convolution module, connects multiple basic convolution modules in a dense connection manner, and adds a 1x1 convolution in the skip connection to improve the adaptive representation ability of the dense connection. By using the above two innovative schemes, the present application constructs a fast and efficient crowd counting model, which can effectively solve the problems of crowd scale change and uneven distribution in the monitoring scene. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a structural schematic diagram of the present application;

[0017] Figure 2 is a structural schematic diagram of EDCN in the present application;

[0018] Figure 3 is a structure diagram of SIB in the application. DETAILED DESCRIPTION

[0019] The specific embodiments of the application are further described below in conjunction with the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0020] The application provides a crowd counting method based on scale interaction and efficient dense residual connection, comprising the following steps:

[0021] Step 1: shoot a crowd scene image, and label the position of the crowd in the image, generally the head position of the crowd can be labeled, and then divide the image into a training set and a test set;

[0022] Step 2: construct the training set, generate a density map as a label according to the label, construct a training set pair, and the density map formula is as follows:

[0023]

[0024] Wherein, G(x) is a density map, x is an input image, x i is the position of the ith person, G σ is a Gaussian kernel, m is the mth person, and M is the number of persons contained in the input image;

[0025] Step 3: pre-process the training set, scale the image pairs of the training set so that the length of the short side of the input image pair is between 640-1024; then randomly crop 4 512x512 image blocks from each input image pair, and perform data enhancement by horizontal flipping;

[0026] Step 4: build a crowd counting model F Model based on scale interaction and efficient dense connection, as shown in Figure 1 ; the counting model comprises three sub-networks, a feature extraction network F FEN (Feature Extraction Network, FEN), an efficient dense connection network F EDCN (Efficient Dense Connection Network, EDCN), and a density map generation network F DGN (Density Generation Network, DGN), and the crowd counting model is as follows:

[0027] y=F DGN (F EDCN (F FEN (x))), (2)

[0028] where x denotes the input image, and y denotes the predicted density map;

[0029] Further, the feature extraction network F FEN The main purpose is to extract general features of the image, which is conducive to the further integration and utilization of the subsequent modules to obtain crowd-related features and generate a density map; the present application extracts general features of the crowd image by using the first ten layers of VGG16, and then fuses the general features by using a 1x1 convolution to reduce the channel number of the output features to one fourth of the original, which can be expressed by the following formula:

[0030] X FEN = Conv 1×1 (VGG(x)), (3)

[0031] where X FEN denotes the output of the feature extraction network F FEN ;

[0032] Further, the efficient dense connection network F EDCN The main purpose is to integrate general features to obtain crowd-related features, as shown in F Figure 2 ; F EDCN contains three efficient dense connection blocks (EDCB), which can be expressed by the following formula:

[0033] X EDCN = F EDCB (F EDCB (F EDCB (X FEN ))), (4)

[0034] where X EDCN denotes the output of the feature extraction network F EDCN ;

[0035] Each efficient dense connection block (EDCB) adopts an efficient dense residual connection method, and first a 1x1 convolution is used to fuse the input features, which can be expressed by the following formula:

[0036]

[0037] wherein, denotes the output of the 1x1 convolution;

[0038] Then, it sequentially passes through two scale interaction blocks (SIB), which can be expressed by the following formula:

[0039]

[0040]

[0041] wherein, and denote the output of the first SIB and the second SIB respectively;

[0042] In order to effectively integrate the outputs of the two SIB modules, the application adopts an efficient dense connection scheme, which uses multiple 1x1 convolutions to splice the outputs of the two SIB modules together, and the 1x1 convolution can adaptively select the effective features of the layer features and suppress the invalid features, thereby improving the representation ability of the model; this scheme fully integrates the representation ability of multiple modules to solve the problem of uneven crowd distribution, and can be expressed by the following formula:

[0043]

[0044]

[0045]

[0046]

[0047] wherein, X1, X2, X3 and X4 are respectively: module input, 1x1 convolution output, first SIB output and second SIB output dense connection; then they are spliced together and subjected to a 1x1 convolution for channel fusion, which can be expressed by the following formula:

[0048] X5=Conv 1×1 (Concat(X1,X2,X3,X4)), (12)

[0049] wherein, Concat() represents a splicing operation;

[0050] Finally, in order to increase the transmission of information in the training process and reduce the training difficulty, a long skip connection is added, which can be expressed by the following formula:

[0051] X Out =X5+X EDCN ; (13)

[0052] Further, the density map generation network F DGN is used to improve the resolution of the feature map and generate a density map, mainly including four convolution layers and three bilinear interpolation up-sampling, which can be expressed as a whole:

[0053] y=F DGN (X EDCN ); (14)

[0054] Further, asFigure 3 As shown in the above, the main purpose of the scale interaction module SIB is to interact multi-scale features and improve the model's perception ability to the scale change of the crowd in the monitoring scene, and obtain multi-scale features. SIB contains two groups of three rows of convolution with different kernel sizes, and performs feature interaction between different rows to improve the scale perception ability of the model. Specifically, the input of SIB is transmitted into three convolutions with different kernel sizes respectively, which can be expressed by the following formula:

[0055]

[0056]

[0057]

[0058] wherein, and represent the 1x1 convolution, 3x3 convolution and 5x5 convolution of the first group respectively, represent the outputs of the three convolutions of the first group respectively; next, the outputs of the three convolutions are spliced to make the features of different scales interact and improve the scale perception ability of the features, which can be expressed by the following formula:

[0059]

[0060] wherein, represents the output of splicing three columns of different convolutions; then, is input into the next three rows of convolutions to improve the scale combination under different receptive fields and obtain more rich multi-scale representations, which can be expressed by the following formula:

[0061]

[0062]

[0063]

[0064] wherein, and represent the 1x1 convolution, 3x3 convolution and 5x5 convolution of the second group respectively, represent the outputs of the three convolutions of the second group respectively, and then they are spliced together and input into a 1x1 convolution for fusing features of different scales, which can be expressed by the following formula:

[0065]

[0066] Further, the density map generation network F DGN is to improve the resolution of the feature map and generate a density map, mainly including four convolution layers and three bilinear interpolation up-sampling, which can be expressed as a whole:

[0067] y = F DGN (X EDCN ); (23)

[0068] Step 5: input the training set into the crowd counting model, train the counting model using the MSE loss, obtain a trained model, and the MSE loss formula is as follows:

[0069]

[0070] wherein, x i is the i-th input image, θ is the F Model learned parameter, and n is the number of training set samples;

[0071] Step 6: use the trained model obtained in step 5 to predict the density map of the test image, and obtain the number of people and the crowd distribution of the image.

[0072] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A crowd counting method based on scale interaction and efficient dense residual connections, characterized in that, Includes the following steps: Step 1: Capture images of the crowd scene and label the positions of the crowd in the images, including the positions of the heads of the crowd. Then, divide the images into training and testing sets. Step 2: Construct a training set. Generate density maps as labels based on the annotations, and construct training set pairs. The formula for generating density maps is as follows: Where G(x) is the density map, x is the input image, and x i G is the position of the i-th person. σ It is a Gaussian kernel, m is the m-th person, and M is the number of people in the input image; Step 3: Preprocess the training set by scaling the image pairs in the training set so that the shorter side length of the input image pairs is between 640 and 1024; then randomly crop four 512×512 image blocks from each pair of input images and horizontally flip them for data augmentation. Step 4: Construct a crowd counting model F based on scale interaction and efficient dense connections Model The counting model consists of three sub-networks: the feature extraction network F FEN Highly efficient and densely connected network F EDCN Density map generation network F DGN The formula for the crowd counting model is as follows: y=F DGN (F EDCN (F FEN (x))), Where x represents the input image and y represents the predicted density map; The feature extraction network F FEN This is used to extract general features from the image, which is beneficial for subsequent modules to further integrate and utilize the features, obtain crowd-related features, and generate density maps. The first ten layers of VGG16 are used to extract general features from the crowd image, and then a 1×1 convolution is used to fuse these general features, reducing the number of output feature channels to one-quarter of the original. The formula is as follows: X FEN =Conv 1×1 (VGG(x)), Among them, X FEN The feature extraction network F FEN The output; The high-efficiency dense connection network F EDCN Used to integrate common features and obtain population-related features; F EDCN It contains three high-efficiency densely connected blocks (EDCBs), as shown in the following formula: X EDCN =F EDCB (F EDCB (F EDCB (X FEN ))), Among them, X EDCN The feature extraction network F EDCN The output; Each high-efficiency dense connection block (EDCB) employs a high-efficiency dense residual connection method; First, a 1×1 convolution is used to fuse the input features, as shown in the following formula: in, This represents the output of a 1×1 convolution; Then it passes through two scale interaction modules (SIB) in sequence, as shown in the following formula: in, and These represent the outputs of the first SIB and the second SIB, respectively. To effectively integrate the outputs of the two SIB modules, an efficient dense connection scheme is adopted. This scheme uses multiple 1×1 convolutions to concatenate the outputs of the two SIB modules together. The 1×1 convolution can adaptively select effective features of the layer and suppress ineffective features, thereby improving the model's representational ability. This scheme fully utilizes the representational capabilities of multiple modules to address the problem of uneven population distribution. The formula is as follows: Where X1, X2, X3, and X4 are: module input, 1×1 convolution output, and dense connections of the first and second SIB outputs, respectively; they are then concatenated together and subjected to a 1×1 convolution for channel fusion, as shown in the following formula: X5=Conv 1×1 (Concat(X1,X2,X3,X4)), Concat() represents the concatenation operation; Finally, to increase information transfer during training and reduce training difficulty, a long jump connection was added, as shown in the following formula: X Out =X5+X EDCN ; The density map generation network F DGN To improve the resolution of the feature map and generate a density map, four convolutional layers and three bilinear interpolation upsampling layers are used, as shown in the following formula: y=F DGN (X EDCN ); The Scale Interaction Module (SIB) is used to perform multi-scale feature interaction, improve the model's ability to perceive changes in the scale of the crowd in the monitoring scene, and obtain multi-scale features. The SIB contains two sets of three-column convolutions with different kernel sizes, and performs feature interaction between different columns, thereby improving the model's scale perception ability. The input to SIB is fed into three convolutions with different kernel sizes, as shown in the following formula: in, and These represent the 1×1 convolution, 3×3 convolution, and 5×5 convolution of the first group, respectively. These represent the outputs of the first group of three convolutions; Next, the outputs of the three convolutions are concatenated to allow features at different scales to interact, thereby improving the scale-awareness of the features. The formula is as follows: in, This represents concatenating the outputs of three different convolutions; Then, The next three columns of convolutions are input to enhance the scale combinations under different receptive fields, obtaining richer multi-scale representations, as shown in the following formula: in, and These represent the 1×1 convolution, 3×3 convolution, and 5×5 convolution of the second group, respectively. These represent the outputs of the three convolutions in the second group. The outputs of the three convolutions are then concatenated and fed into a 1×1 convolution to fuse features at different scales, as shown in the following formula: Step 5: Input the training set into the crowd counting model, and train the counting model using MSE loss to obtain the trained model; the MSE loss formula is as follows: Among them, y i Let θ be the i-th input image, and F be the input image. Model The learned parameters, where n is the number of samples in the training set; Step 6: Use the trained model obtained in Step 5 to predict the density map of the test image and obtain the number of people and the distribution of the crowd in the image.

Images