A crowd density estimation method based on deep hierarchical aggregation

Abstract

The present invention belongs to the field of artificial intelligence and relates to a crowd density estimation method based on deep aggregation, comprising the following steps: Step 1: Inputting an original image and preprocessing the original image; Step 2: Inputting the preprocessed image into a model backbone to extract an image feature map; Step 3: Passing the feature map obtained in Step 2 through two parallel three-layer convolutional layers, outputting two prediction branches: a head center prediction branch and a head classification branch; Step 4: Using the head classification branch to output head confidence, filtering out predictions from the head center prediction branch with a confidence level less than 0.5, and completing the final crowd target center point location; Step 5: Outputting the crowd head center location result and counting the number of people. Compared with existing technologies, the present invention can directly predict crowd centers, thereby achieving end-to-end closed-loop training for crowd density estimation datasets. As a result, model training converges faster and the model detection effect is better.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a crowd density estimation method based on deep aggregation. Background Art

[0002] Crowd counting is a core task in computer vision, aiming to estimate the number of pedestrians in a still image or video frame. It has important application value in smart cities and smart communities. Existing crowd density estimation methods generally use intermediate representations (density maps or detection boxes) to estimate crowd density. However, both methods have their shortcomings. The density map regression-based method generates a crowd distribution density map based on the center point of the head as the model training target, but the density map generated by this method is not accurate. In some crowded scenes, the detection method based on the estimated box will cause the NMS method of target detection to fail due to overlapping heads, ultimately leading to the failure of the crowd counting task.

[0003] Visual recognition tasks require a wealth of information, ranging from low to high levels of richness and from small to large scales. As the depth of convolutional neural networks increases, a single layer cannot provide sufficient information. Only by aggregating this information can the accuracy of acquiring deep semantic information about the target be improved. The Deep Layer Aggregation (DLA) architecture iteratively integrates the feature information of the network structure, resulting in higher accuracy and fewer parameters.

[0004] Therefore, it is urgent to design a crowd density estimation method based on deep aggregation. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: how to provide more information to improve the accuracy of crowd density estimation.

[0006] The present invention provides a crowd density estimation method based on deep aggregation, comprising the following steps:

[0007] Step 1: Input the original image and perform scaling preprocessing on the width and height of the original image;

[0008] Step 2: Input the preprocessed image into the model backbone to extract the image feature map. Here, Backbone refers to the backbone network and Feature map refers to the feature map.

[0009] Step 3: After the feature map obtained in step 2 passes through two parallel three-layer convolutional layers, two prediction branches are output: the head center point prediction branch and the head classification branch;

[0010] Step 4: Use the head classification branch to output the head confidence, filter out the heads predicted by the head center prediction branch with a confidence lower than 0.5, and complete the final crowd target center location;

[0011] Step 5: Output the crowd head center positioning results and count the number of people to complete the crowd density estimation task.

[0012] Furthermore, in step 1, the width and height of the original image are pre-scaled by a multiple of 32, and the processed image width W and height H are less than or equal to the original image width and height.

[0013] Furthermore, the backbone model in step 2 uses Resnet34 as the basic backbone network, and performs deep aggregation transformation between multiple layers of the backbone network. In addition, a variable convolution module is added to the upsampling in the backbone network to update the network structure and generate an image feature map.

[0014] Furthermore, in step 2, deep aggregation transformation is performed between multiple layers of the network, which means that the backbone network layer is downsampled from low to high to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input respectively to extract semantic information of different levels of the image, and a deep aggregation structure is used to perform multi-layer image feature map fusion.

[0015] Furthermore, the overall backbone downsampling stride = 4, so the backbone network output feature map size is H / 4, W / 4. Here, H is the processed image height, W is the processed image width, and stride is the CNN convolution step size.

[0016] Furthermore, in step 3, the head center prediction branch predicts the offset of the current patch target center based on the point anchor. Here, Patch represents an image block. The specific process is as follows:

[0017] Assume the model backbone network is F s ,F s In the example, s represents the model downsampling rate. In the network structure of this method, s is 4. s The feature map size is H×W;

[0018] F s Each pixel in the feature map corresponds to a patch of size s×s in the input image;

[0019] In each patch, a fixed initial anchor reference point set R and point set coordinates R are predefinedk :

[0020] R={R k |kσ{1,…,K}}

[0021] R k =(x k ,y k )

[0022] The initial reference point set R in each patch k In the form of center distribution, 4 initial anchors are distributed at the center of each patch, and Anchor represents the anchor point.

[0023] After the initial reference anchor point set is initialized, the head center point prediction branch will predict each anchor reference point R k Predicted coordinate offset Finally, the predicted value of the center of the head in the crowd is generated

[0024]

[0025]

[0026]

[0027] After the feature map passes through the softmax layer of the head classification branch, the head classification branch outputs the head classification confidence for each patch. Here, Patch represents an image block.

[0028] Among them, K is the number of anchor reference points, x k ,y k is the point set coordinate, and τ is the weight factor.

[0029] The beneficial effects of the present invention are:

[0030] 1. By performing nearest neighbor scaling preprocessing on the image width and height by multiples of 32, the image resolution can be kept as close to the original size as possible while meeting the model inference requirements. This allows the model to obtain more appearance feature information from the image, achieving better detection results in scenes with dense crowds and small crowds, thereby more accurately estimating the crowd center.

[0031] 2. By performing deep aggregation transformation on the model backbone, more skip connections are created between low-level and high-level features in different layers of the model backbone, allowing the model to fuse multi-layer features and enhance feature map representation capabilities. In addition, deformable convolution is used during upsampling of the model backbone, which can dynamically adapt the receptive field according to the scale and posture of the target. This improvement also helps to improve the detection effect of crowd density scenes.

[0032] 3. Compared with the general method of estimating crowd density through intermediate representation, the present invention adopts a method based on center point estimation to directly predict the center point of the crowd, so that end-to-end closed-loop training can be achieved for the crowd density estimation dataset. Therefore, the model training converges faster and the model detection effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 It is a flow chart of the crowd density estimation method based on deep aggregation;

[0035] Figure 2 This is a network structure diagram of the crowd density estimation method based on deep aggregation;

[0036] Figure 3 It is a backbone network diagram of the crowd density estimation method based on deep aggregation. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] In this embodiment, if Figure 1 As shown, a crowd density estimation method based on deep aggregation includes the following steps:

[0039] Step 1: Input the original image and pre-scale its width and height by multiples of 32. The input image width and height are 32 times the nearest dimension. For example, if the input image width is 1100, the 32-times nearest dimension will result in 1088. This ensures that the resized size is a multiple of 32 and smaller than the original image width. The height processing rules are the same as the width processing rules.

[0040] Step 2: Input the preprocessed image into the model backbone to extract the image feature map. This model uses Resnet34 as the base backbone network and downsamples the backbone network layers to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input, respectively, to extract semantic information at different levels of the image. A deep aggregation structure is used to fuse multiple layers of image feature maps, creating more skip connections between low-level and high-level features. This gives the feature map extracted by the network stronger representational capabilities. A variable convolution module is added to the backbone network for upsampling, updating the network structure to generate the image feature map. Furthermore, since the overall backbone downsampling stride = 4, the backbone network output feature map dimensions are H / 4 and W / 4. Here, H is the processed image height, W is the processed image width, and stride is the CNN convolution stride.

[0041] In step 2, by performing deep aggregation transformation on the model backbone, more skip connections are made between low-level and high-level features in different layers of the model backbone, so that the model can fuse multi-layer features and the feature map representation ability is stronger. In addition, deformable convolution is used during backbone upsampling, which can dynamically adapt the receptive field according to the scale and posture of the target. This improvement also helps to improve the detection effect of crowd density scenes.

[0042] Step 3: After the feature map obtained in step 2 passes through two parallel three-layer convolutional layers, two prediction branches are output: the head center point prediction branch and the head classification branch;

[0043] The head center prediction branch predicts the offset of the current patch target center point based on the point anchor. The specific process is as follows:

[0044] Assume the model backbone network is F s ,F s In the example, s represents the model downsampling rate. In the network structure of this method, s is 4. s The feature map size is H×W;

[0045] F s Each pixel in the feature map corresponds to a patch of size s×s in the input image;

[0046] In each patch, a fixed initial anchor reference point set R and point set coordinates R are predefined k:

[0047] R={R k |kσ{1,…,K}}

[0048] R k =(x k ,y k )

[0049] The initial reference point set R in each patch k Take the form of center distribution, distributing 4 initial anchors at the center of each patch;

[0050] After the initial reference anchor point set is initialized, the head center point prediction branch will predict each anchor reference point R k Predicted coordinate offset Finally, the predicted value of the center of the head in the crowd is generated

[0051]

[0052]

[0053]

[0054] After the feature map passes through the softmax layer of the head classification branch, the head classification branch outputs the head classification confidence for each patch. Here, Patch represents an image block and Anchor represents an anchor point.

[0055] Among them, K is the number of anchor reference points, x k ,y k is the point set coordinate, and τ is the weight factor.

[0056] Step 4: Use the head classification branch to output the head confidence, filter out the heads predicted by the head center prediction branch with a confidence lower than 0.5, and complete the final crowd target center location;

[0057] Step 5: Output the crowd head center positioning results and count the number of people to complete the crowd density estimation task.

[0058] After the above network structure, the crowd head center positioning results are output, and the number of people is counted and output to complete the crowd density estimation task and complete the end-to-end training optimization of the model.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A crowd density estimation method based on deep aggregation, characterized in that: The following steps are involved: Step 1: Input the original image and perform scaling preprocessing on the width and height of the original image; Step 2: Input the preprocessed image into the model backbone to extract the image feature map; Step 3: After the feature map obtained in step 2 passes through two parallel three-layer convolutional layers, two prediction branches are output: the head center point prediction branch and the head classification branch; Step 4: Use the head classification branch to output the head confidence, filter out the heads predicted by the head center prediction branch with a confidence lower than 0.5, and complete the final crowd target center location; Step 5: Output the crowd head center positioning results and count the number of people to complete the crowd density estimation task; In step 2, the model backbone uses Resnet34 as the basic backbone network, and performs deep aggregation transformation between multiple layers of the backbone network. A variable convolution module is added to the upsampling of the backbone network, the network structure is updated, and the image feature map is generated; In step 3, the head center prediction branch predicts the offset of the current patch target center point based on the anchor point. The specific process is as follows: Assume the model backbone network is , In the formula, s represents the model downsampling rate. In the network structure of this method, s is 4. The feature map size is ; Each pixel in the feature map corresponds to the input image Patch size; In each patch, a fixed initial anchor reference point set R and point set coordinates are predefined : Initial reference point set in each patch Take the form of center distribution, distributing 4 initial anchors at the center of each patch; After the initial reference anchor point set is initialized, the head center point prediction branch will predict each anchor reference point Predicted coordinate offset , and finally generate the predicted value of the center of the head in the crowd : After the feature map passes through the softmax layer of the head classification branch, the head classification branch outputs the head classification confidence for each patch; Among them, K is the number of anchor reference points, is the coordinate of the point set, is the weight factor.

2. The crowd density estimation method based on deep aggregation according to claim 1, characterized in that: In the step 1, the width and height of the original image are pre-scaled by a multiple of 32, and the processed image width W and height H are less than or equal to the original image width and height.

3. The method for estimating crowd density based on deep aggregation according to claim 1, characterized in that: In the second step, deep aggregation transformation is performed between multiple layers of the network, which means that the backbone network layer is downsampled from low to high to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input respectively to extract semantic information of different levels of the image, and a deep aggregation structure is used to perform multi-layer image feature map fusion.

4. The method for estimating crowd density based on deep aggregation according to claim 1, characterized in that: The overall backbone downsampling stride=4, and the backbone network output feature map size is H / 4 and W / 4, where H is the processed image height, W is the processed image width, and stride is the CNN convolution step size.

Images