Dense pedestrian detection method

Abstract

A dense pedestrian detection method. In the network training stage, a convolutional neural network is built, and the encoder-decoder and detection head are trained to predict the correct pedestrian bounding box; and according to the prediction results, an optimal prediction candidate box is assigned to each pedestrian; other network parameters outside the density estimation head are fixed, and the density target is generated using the unique prediction candidate box of each pedestrian, and the density estimation head is trained; finally, all network parameters are released, and the entire network is jointly trained. In the test application stage, when post-processing, each time a pedestrian box is selected to be retained, the Gaussian activation map of the corresponding position is subtracted from this predicted density map, and for those pedestrian boxes whose overlap rate with the retained pedestrian is greater than a threshold, the updated density map is used to perform a secondary judgment on these pedestrian boxes. In dense scenes, the present invention solves the problem that the general non-maximum suppression method will mistakenly delete the correctly predicted bounding box, and at the same time does not affect the performance of non-dense scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of image pedestrian detection, and is used to solve the problem that a non-maximum suppression method commonly used in the post-processing stage of a detector may mistakenly delete correctly predicted candidate frames in densely populated pedestrian situations, resulting in a low overall recall rate. Background Art

[0002] Pedestrian detection is a very challenging computer vision task and is widely used in various computer vision systems as a core module. Although pedestrian detection technology has made significant progress in recent years, pedestrian detection in dense conditions is still a very challenging task due to the frequent occlusion in practical applications.

[0003] In recent years, methods based on convolutional neural networks have achieved an absolute advantage in the field of pedestrian detection, with accuracy far higher than the results obtained by manually designed features. Most pedestrian detectors based on deep convolutional neural networks are based on classic deep learning algorithms for general object detection, such as the Faster R-CNN algorithm proposed by Ren et al. in the 2015 NeurIPS conference article Faster r-cnn: Towards real-time object detection with region proposal, and the Fcos algorithm proposed by Tian et al. in the 2019 ICCV conference article Fcos: Fully convolutional one-stage object detection. Targeted improvement plans are proposed based on these frameworks for the characteristics of pedestrians and the dense characteristics of pedestrians in public scenes. These works can be divided into two categories:

[0004] The first category is to improve the candidate frame extraction stage, including improving the network structure to obtain better pedestrian features for subsequent candidate frame classification and regression; designing targeted loss functions to assist network training and improve the accuracy and robustness of network output. Zhang et al. proposed the OR-CNN network in the 2018 ECCV article Occlusion-aware R-CNN: Detecting Pedestrians in a Crowd, which extracts 5 key part features from each candidate frame for scoring, and finally aggregates multiple non-occluded part scores to obtain the final candidate frame score. Pang et al. proposed the MGAN network at the 2019 ICCV conference Mask-GuidedAttention Network for Occluded Pedestrian Detection, which extracts human foreground part features through the attention self-network for candidate frame classification. Wang et al. proposed the Repulsion Loss: Detecting Pedestrians in a Crowd at the 2018 CVPR conference to solve the problem that the prediction box appears between two pedestrians in dense situations by pushing away the prediction boxes belonging to different pedestrians.

[0005] The second category is to improve the post-processing process of redundant candidate boxes. Although the first category of methods significantly improves the accuracy and recall of candidate boxes, since these methods predict multiple candidate boxes for each pedestrian, post-processing is still required to remove redundant predictions and retain only one predicted box for each pedestrian. The widely used non-maximum suppression method will have poor high threshold effects in pedestrian detection, and low thresholds will mistakenly delete the originally correctly predicted bounding boxes in dense situations. Liu et al. proposed the AdaptiveNMS method in the 2019 CVPR article Adaptive NMS: Refining Pedestrian Detection in a Crowd, predicting a density variable for each bounding box, and adaptively determining the threshold used for non-maximum suppression of the current bounding box based on the density. However, since there is no restriction on the position of high threshold suppression, redundant predictions will still occur. The Count and Similarity-Aware R-CNN for Pedestrian Detection method proposed by Xie et al. at the 2020 ECCV conference and the APD method proposed by Zhang et al. in the 2021 IEEE Transactions on Multimedia journal article Attribute-Aware Pedestrian Detection in a Crowd not only consider the position overlap rate and density during non-maximum suppression, but also introduce similarity comparison, and only retain the bounding boxes with high overlap rate when the similarity is low. This method has achieved great improvement, but this method is still different from the human judgment mechanism. People can distinguish whether two pedestrians are the same pedestrian without comparing similarities. In scenes where pedestrians look similar (such as people wearing the same clothes on campus or in factories), this method may fail.

[0006] Based on the above considerations, our patent proposes a method that is completely based on density and overlap rate, and solves the lack of redundant location information in the AdaptiveNMS method. Summary of the invention

[0007] The technical problem to be solved by the present invention is that in dense scenes, even if the detector detects the correct pedestrian target, when non-maximum suppression is used for post-processing, the target that should be retained will be suppressed by the normally used low threshold due to the large target overlap rate. Simply raising the threshold will cause duplicate samples that should be suppressed to be retained, resulting in a decrease in accuracy; using adaptive non-maximum suppression can increase the threshold in dense areas and use a low threshold in simple scenes, but this approach cannot distinguish the specific locations of dense pedestrians, resulting in many false positive samples being retained in dense situations.

[0008] The technical solution of the present invention is: compared with adaptive non-maximum suppression, we redefine the density as the sum of the Gaussian activation maps generated by all true value boxes on the same image at the corresponding positions, and let the network learn to predict the density map during network training. During post-processing, each time a pedestrian box is selected to be retained, the Gaussian activation map at the corresponding position is subtracted from the predicted density map. For those pedestrian boxes whose overlap rate with the retained pedestrian is greater than the threshold, we use the updated density map to make a secondary judgment on these pedestrian boxes. If the Gaussian activation map corresponding to the box can better reconstruct the density map, it will be retained; otherwise, it will be discarded. The specific steps are as follows:

[0009] (1) Network training phase

[0010] Our network structure is as follows Figure 1 As shown in Figure 2, our method is based on the single-stage target detection algorithm CenterNet, and the detector head is as follows: Figure 2 As shown in the figure, the detection head of this method has three output branches, namely, the center point response of the predicted target, which is the prediction box score, the offset of the target's precise position relative to the output grid position, and the width and height of the target;

[0011] Figure 3 For the density estimation branch we proposed, the density is ideally defined as the sum of the Gaussian activation maps generated by all the true value boxes on the same image at the corresponding positions. In the test phase, we need to correspond the predicted candidate boxes to the density map, so here, we define the true value of the density map as the sum of the Gaussian activation maps generated by the predicted candidate boxes that best match each pedestrian bounding box at the corresponding positions. The best matching potential matrix is ​​defined as:

[0012]

[0013] Where Q i，π(i) ∈[0, 1] represents the matching energy between the i-th true value and the π(i)-th predicted candidate box. Ω i Represents the set of all predicted candidate boxes. The first term represents the spatial prior, and only the value of the points inside the bounding box is 1; the second term is the confidence of the prediction score; the third term is the overlap rate between the predicted candidate box and the true value bounding box. α represents the proportion of the overlap rate in the energy function, and the Hungarian algorithm is used to obtain the best matching predicted candidate box for each true value:

[0014]

[0015] Where G represents the number of true bounding boxes, and N is the number of predicted candidate boxes.

[0016] Since the true value of the density needs to be calculated from the better predictions, we adopted a staged training method: first fix the density estimation branch, train the encoder decoder and the detection head first. Then fix the other parameters outside the density estimation branch and train the density estimation branch. Finally, release all parameters for joint training.

[0017] (2) Testing phase

[0018] Input the image into the network and get four outputs, namely the score activation map, the height and width of the prediction box, the slight offset of the actual center point relative to the prediction output grid, and the density prediction map H m The predicted candidate boxes can be calculated through the first three items, and the 2000 highest scoring candidate boxes of each image with scores greater than 0.05 are screened to obtain the predicted candidate box set , and the corresponding scores Set the threshold for non-maximum suppression to N t

[0019] Initialize the output prediction candidate box is an empty set;

[0020] from Select the largest one and get the serial number m; update H m =H m -Gauss(b m ), where Gauss(b m ) is the candidate box b m Generated Gaussian activation map; Update For all , if iou(b m , b i )>N t ,calculate ,if Update Where Norm() is the bi-norm of the matrix.

[0021] Repeat the previous step until is an empty set; output and

[0022] The effect and benefit of the present invention is that it solves the lack of redundant position information in the AdaptiveNMS method, and can reduce false positive samples. Moreover, since it is completely based on the density and overlap rate method, it is more suitable for situations where pedestrians are dense and similar in appearance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall framework diagram.

[0024] Figure 2 This is a diagram of the detection head structure.

[0025] Figure 3 It is the density estimation head structure diagram.

[0026] Figure 4 Each column represents an application example; (a)-1 and (a)-2 are the original images; (b)-1 and (b)-2 are the predicted box result images obtained after standard non-maximum suppression; (c)-1 and (c)-2 are the predicted box result images obtained after using the density estimation non-maximum suppression proposed by us; (d)-1 and (d)-2 are density map visualization images. DETAILED DESCRIPTION

[0027] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0028] Step 1: Prepare a training dataset. The training dataset can be an existing large-scale dense pedestrian detection dataset CrowdHuman, or a specific self-constructed dataset. If it is a self-constructed dataset, it is necessary to annotate the bounding box that surrounds the entire body of the pedestrian, that is, the coordinates of the upper left and lower right positions of the pedestrian. With the training data, the true value of the bounding box can be read to construct the label used for head detection training and conduct training.

[0029] according to Figure 1 As shown in the figure, a convolutional neural network is built based on the single-stage target detection algorithm CenterNet, and the backbone network uses dla-34; the detector head is as follows Figure 2 As shown in Figure 2, the detection head of this method has three output branches, which respectively predict the target center point response (prediction box score), the offset of the target's precise position relative to the output grid position, and the width and height of the target; the density estimation head structure is shown in Figure 2. Figure 3 As shown in the figure, it consists of a deformable convolution and three convolution modules connected in series.

[0030] Step 2: Fix the density estimation head parameters and train the rest until convergence; the heat map estimation part is trained using the focus loss function with reduced center penalty, and the scale estimation and offset estimation branches are trained using the smooth 1-norm loss function.

[0031] Step 3: Calculate the predicted box and the corresponding score based on the output of the detection head. Calculate the matching energy matrix Q between the predicted box and the true value box according to formula (1): i，π(i) , use the Hungarian algorithm to solve formula (2) and obtain the best matching prediction box corresponding to each true value box.

[0032] Step 4: Using the prediction box and corresponding score obtained in the previous step that uniquely matches the true value box, construct a two-dimensional Gaussian activation map with the prediction box as the center and 1 / 6 of the width and height of the prediction box as the standard deviation, and superimpose it at the corresponding position of the prediction box to obtain the training true value label of the density map.

[0033] Step 5: Fix the encoder-decoder part and the detection head, and train the density estimation head parameters separately using the two-norm loss function until convergence.

[0034] Step 6: Release all parameters and jointly train the entire network until convergence.

[0035] Step 7: In the test application phase, an image is input into the network to obtain four outputs, namely the score activation map Hs, the height H and width W of the predicted box, and the slight offset o of the actual center point relative to the predicted output grid x , o y And the density prediction map H m .

[0036] Step 8: The predicted candidate box can be calculated through the first three items. For a certain position (x c , y_c), the prediction box score at this position is H s (x c ,y c ), bounding box position b i =(x1, y1, x2, y2) is calculated as follows:

[0037] x1=(x c -0.5*W(x c ,y c )+o x )×r

[0038] y1=(y c -0.5*H(x c ,y c )+o y )×r

[0039] x2=(x c +0.5*W(x c ,y c )+o x )×r

[0040] y2=(y c +0.5*H(x c ,y c )+o y )×r

[0041] Among them, r is the ratio of the network input to the output size. The 2000 highest scoring candidate boxes with scores greater than 0.05 are screened to obtain the predicted candidate box set. , and the corresponding scores Set the threshold for non-maximum suppression to N t

[0042] Step 9: Initialize the output prediction candidate box is an empty set;

[0043] Step 10: From Select the largest one and get the serial number k; update H m =H m -Gauss(b k ), where Gauss(b k ) is the candidate box b k Generated Gaussian activation map; Update

[0044] Step 11: For all , if iou(b k , b i )>N t ,calculate if Update Where Norm() is the bi-norm of the matrix.

[0045] Step 12: Repeat steps 10 and 11 until is an empty set; output and

Claims

1. A dense pedestrian detection method, characterized in that: Here are the steps: Step 1: Based on the large-scale dense pedestrian detection dataset CrowdHuman, read the true value of the bounding box and construct the label used for head detection training; Build a convolutional neural network based on the single-stage target detection algorithm CenterNet. The detection head of this method has three output branches, namely the center point response of the predicted target, which is the prediction box score, the offset of the target's precise position relative to the output grid position, and the width and height of the target; Step 2: Fix the density estimation head parameters and train the rest until convergence; Step 3: Calculate the prediction box from the output of the detection head and the corresponding score According to formula (1), the matching energy Q between the i-th true value and the π(i)-th predicted candidate box is calculated. i,π(i) ; where Ω i represents the set of all predicted candidate boxes, represents the confidence of the prediction box, IoU(·,·) represents the overlap of the two bounding boxes, and α∈[0,1] is the proportion of the prediction box quality in the matching energy function. The Hungarian algorithm is used to solve formula (2) to obtain the optimal match between the true value box and the prediction box. Step 4: Using the prediction box and its corresponding score obtained in the previous step that uniquely matches the true value box, a two-dimensional Gaussian activation map is constructed with the prediction box as the center and 1 / 6 of the width and height of the prediction box as the standard deviation. The training true value of the density map is superimposed at the corresponding position of the prediction box. Step 5: Fix the encoder-decoder part and the detection head, and train the density estimation head parameters separately using the two-norm loss function until convergence; Step 6: Release all parameters and jointly train the entire convolutional neural network until convergence; Step 7: In the test application phase, the image is input into the convolutional neural network to obtain four outputs, namely the score activation map, the height and width of the prediction box, the slight offset of the actual center point relative to the prediction output grid, and the density prediction map H. m ; Step 8: The predicted candidate boxes are obtained by calculating the first three items in step 7. The 2000 highest scoring candidate boxes with scores greater than 0.05 are selected for each image to obtain the predicted candidate box set. And the corresponding score Set the threshold for non-maximum suppression to N t ; Step 9: Initialize the output prediction candidate box is an empty set; Step 10: From Select the largest one and get the serial number k; update H m =H m -Gauss(b k ), where Gauss(b k ) is the candidate box b k Generated Gaussian activation map; Update Step 11: For all If iou(b k ,b i )>N t ,calculate if Update Where Norm() is the bi-norm of the matrix; Step 12: Repeat steps 10 and 11 until is an empty set; output and

Images