Occluded pedestrian detection method based on anchor-free full frame and visible frame fusion

Abstract

The application relates to a kind of occluded pedestrian detection methods based on anchor frameless full frame and visible frame fusion, comprising the following steps: input training sample, utilize deep convolution network to extract its feature, obtain feature map;Two branches are respectively used anchor frameless method, the confidence map of classification and the heat map of regression are obtained through full connection layer, to predict the center point of pedestrian full frame and visible frame and the height and width of the frame corresponding to the center point;Reserve the center point of full frame confidence greater than the set threshold, generate the full frame set of pedestrian from the full frame center point and the height and width of the frame corresponding thereto;For each pixel point on feature map, the height and width of the frame corresponding to the position generate the visible frame of pedestrian at the position, according to the pedestrian width-height ratio fixed value to develop into full frame;For each full frame in full frame set, match to a full frame developed from visible frame by maximum intersection ratio for each full frame;Calculate fusion score;Non-maximum suppression is carried out to full frame, and the final full frame is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to an occluded pedestrian detection method in the field of computer vision, in particular to a method for occluded pedestrian detection by using anchor-free frame fusion visible frame information. BACKGROUND

[0002] Pedestrian detection is a very important research field in computer vision, which is a task of inferring whether an input image (or video frame) contains a pedestrian, and marking the position of the pedestrian if it contains a pedestrian. Pedestrian detection technology has a wide range of application scenarios in automatic driving, intelligent monitoring, human interaction, etc. Unlike general object detection, the difficulty of pedestrian detection research has always been that in complex dense pedestrian scenes, the mutual occlusion between pedestrians has serious influence and interference, causing the detection accuracy of occluded pedestrians to decrease.

[0003] In recent years, with the popularization of computer vision, big data, cloud computing and other technologies in life, the application requirements for automatic driving to perceive the surrounding environment and intelligent monitoring to track personnel, etc., have put forward higher requirements for the accuracy of pedestrian detection in image processing, especially for the detection of occluded pedestrians in the current defective dense pedestrian scene. Therefore, this patent mainly studies how to improve the detection accuracy of occluded pedestrians.

[0004] Pedestrian detection belongs to the category of object detection, so the current mainstream pedestrian detection algorithm is also general for object detection. The object detection task can be divided into two parts: feature extraction of the backbone network and classification and positioning of the detection head. Pedestrian detection is different from general object detection in that the recognition degree of pedestrians and backgrounds is smaller, and the handling of the occlusion problem commonly existing in pedestrian scenes.

[0005] Pedestrian detection is similar to object detection and can be generally divided into three methods based on anchor frame, anchor-free frame and fusion of the two. Among them, the anchor-based method is to judge and classify a large number of rectangular frames, which is complex in calculation, cumbersome in process, and difficult to configure and debug with many hyperparameters, such as the double-stage detector Faster RCNN[1] with high accuracy and the single-stage detector SSD[2] with fast speed, etc. The anchor-free method converts object detection into the prediction of key points or center points and size scales, and then generates the object detection frame, which is simple in idea, small in operation amount and good in effect, such as CSP[3] predicting the center point and height of the pedestrian, and TLL[4] predicting the head and bottom of the pedestrian.

[0006] For the research of solving the occlusion problem, most methods in pedestrian detection are through the use of visible parts as additional supervision to improve the performance of occlusion. These methods can be roughly divided into four categories: first, train independent detectors for each occlusion mode, and then fuse all occlusion modes for inference, such as Zhou et al. [5] use AdaBoost.MH multi-label learning method to build a decision tree shared by all parts of pedestrian body detectors to capture the overall distribution of all parts; second, use attention mechanism to better capture features, such as MGAN [6] uses pixel-level attention map in the feature layer to highlight the visible parts and suppress the occluded parts; third, add a visible box classifier to integrate the confidence score into the final score, such as Bibox [7] and others use Faster RCNN framework to train full anchor box with positive examples and visible anchor box with positive and negative examples, and fuse the scores of the two anchors in the inference process; fourth, propose a new loss function for crowded scenes, such as RepLoss [8] designs a novel regression loss to prevent target proposal boxes from moving to surrounding pedestrians. However, most of these methods cannot accurately find the correspondence between the visible box and the full box and fuse the information of the two to solve the occlusion problem.

[0007] References:

[0008] [1] S. Q. Ren, K. M. He, G. R, and J Sun. Faster R-CNN: Towards real-time object detection with region proposal networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-49.

[0009] [2] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and B. A. C. SSD: Single shot multibox detector [C]. In Proceedings of the 14th European Conference on Computer Vision, 2016.

[0010] [3] W. Liu, S. C. Liao, W. Q. Ren, W. D. Hu, and Y. N. Yu. High-level semantic feature detection: A new perspective for pedestrian detection [C]. In Proceedings of IEEE International conference on Computer Vision and Pattern Recognition, 2019.

[0011] [4] T. Song, L. Sun, D. Xie, H. Sun, and S. Pu. Small-scale pedestrian detection based on topological line localization and temporal feature aggregation [C]. In Proceedings of the 15th European Conference on Computer Vision, 2018.

[0012] [5] C. Zhou and J. Yuan. Multi-label learning of part detectors for heavily occluded pedestrian detection [C]. In Proceedings of the IEEE International Conference on Computer Vision, 2017.

[0013] [6] Y. W. Pang, J. Xie, M. H. Khan, R. M. Anwer, F. S. Khan, and L. Shao. Mask-guided attention network for occluded pedestrian detection [C]. In Proceedings of the IEEE International Conference on Computer Vision, 2019.

[0014] [7] C. Zhou and J. Yuan. Bi-box regression for pedestrian detection and occlusion estimation[C]. In Proceedings of the 15th European Conference on Computer Vision, 2018.

[0015] [8] X. Wang, T. Xiao, Y. Jiang, S. Shao, J. Sun and C. Shen. Repulsion loss: Detecting pedestrians in a crowd[C]. In Proceedings of IEEE International conference on Computer Vision and Pattern Recognition, 2018. SUMMARY

[0016] The purpose of the present application is to overcome the problem of low accuracy caused by occlusion in dense pedestrian scenes in the existing pedestrian detection process, and to propose a method for detecting occluded pedestrians based on the fusion of anchor-free full frame and visible frame.

[0017] A method for detecting occluded pedestrians based on the fusion of anchor-free full frame and visible frame, comprising the following steps:

[0018] Step 1: input the training sample, use the deep convolutional network to extract the features, and get the feature map F;

[0019] Step 2: use two branches on the feature map F to get the classification confidence map and the regression heat map through the full connection layer to predict the center point of the pedestrian full frame and the visible frame and the height and width of the frame corresponding to the center point;

[0020] Step 3: keep the center point of the full frame whose confidence is greater than a certain threshold, and generate the full frame set {B full} of pedestrians from the full frame center point and the height and width of the frame corresponding to it, which contains the position and size of the full frame and the score s;

[0021] Step 4: generate the pedestrian visible frame at this position from the height and width of the frame corresponding to each pixel point on the feature map, and expand the full frame {B vis2full} according to the pedestrian width-height ratio of 0.41;

[0022] Step 5: for each full frame in the full frame set in step 3, match it to a full frame expanded from the visible frame in step 4 by maximum intersection ratio, and the visible frame isvis}, the position size and score of the visible box vis_s are included;

[0023] Step 6: Set the visible ratio vis_ratio as the intersection over union of the visible box and the full box, and the scores of the full box and the visible box are s and vis_s respectively, then the fusion score merge_s of the full box and the visible box is:

[0024] merge_s = s·vis_ratio + vis_s·(1-vis_ratio),

[0025] Step 7: Perform non-maximum suppression on the full box according to the fusion score of the visible box and the full box to obtain the final full box.

[0026] The detector obtained by the method of the present application through simultaneously predicting the full box and the visible box of the pedestrian and fusing the scores of the two according to the occlusion degree is an occluded pedestrian detector that can fuse the information of the full box and the visible box. Compared with the method of detecting pedestrians based on the traditional full box information, the classifier that simultaneously predicts the visible box and the full box and then fuses the scores of the two effectively fuses the advantage that the visible box is less occluded relative to the full box, thereby reducing the missed detection rate of the occluded pedestrians and improving the accuracy of the detector. At the same time, the method is simple in idea, and only the branch for predicting the visible box in the training stage and the fusion of the scores of the visible box and the full box in the test stage are added to the original anchor-free pedestrian detector, so that the problem of occluded pedestrians can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The application example of the present patent in the occluded pedestrian detection in autonomous driving and intelligent monitoring is described in the.

[0028] Figure 2 The method example of the traditional anchor-free pedestrian detection is described in the.

[0029] Figure 3 The occluded pedestrian detection method based on the fusion of the anchor-free full box and the visible box proposed by the present patent is described.

[0030] Figure 4 The specific implementation method of the fusion module of the full box and the visible box proposed by the present patent is described in the.

[0031] Figure 5 The block diagram of the occluded pedestrian detection method based on the fusion of the anchor-free full box and the visible box proposed by the present patent is described. DETAILED DESCRIPTION

[0032] The present patent will be further described below in combination with the drawings.

[0033] The patent proposes a method for occluded pedestrian detection based on anchor-free full frame and visible frame fusion. First, the CNN learns the features of the image, and the anchor-free method is used to predict the center point and size of the full frame and visible frame in the image. The full frame with confidence greater than the set threshold is taken, the visible frame is expanded into a full frame with a fixed pedestrian width-height ratio, the remaining full frame is matched to a visible frame expanded full frame with the maximum intersection ratio, the intersection ratio of the visible frame and the full frame is calculated as the visibility ratio, and the scores of the full frame and the corresponding visible frame are fused for non-maximum suppression processing. At this time, the full frame is the fusion of the full frame and the visible frame information, and the final pedestrian full frame is obtained.

[0034] Figure 2 Examples of traditional anchor-free pedestrian detection are described. Specifically, this method inputs the original image into the anchor-free detection network, then directly predicts the center point and scale of the pedestrian full frame to obtain the final detection frame. The prediction only contains the full frame, which cannot well deal with the occlusion situation where the pedestrian has a large invisible area.

[0035] Figure 3 The method for occluded pedestrian detection based on anchor-free full frame and visible frame fusion proposed by the patent is described. Specifically, the network includes three main parts: convolutional feature extraction network, full frame and visible frame prediction network, and full frame and visible frame fusion network. The convolutional feature extraction network is mainly used to extract general features. The full frame and visible frame prediction network further predicts the center point heat map and scale map of the pedestrian full frame and visible frame based on the convolutional feature extraction network. In the center point heat map, the response value of the position where the pedestrian exists is significant, while the response value of the position where there is no pedestrian is not significant, which can effectively reflect the position of the pedestrian. Then, due to the less occlusion of the visible frame, the predicted visible frame and full frame are fused through the full frame and visible frame fusion network, thereby significantly reducing the missed detection rate of the occlusion situation of pedestrians and improving the accuracy of the detector. The full frame and visible frame fusion network is only used in the test stage.

[0036] For the sake of convenience in the following description, some brief explanations are first given. After the deep convolutional network extracts the features of the image, two independent anchor-free detection branches are used to predict the classification and regression of the full frame and visible frame. For the full frame {B full} that is retained with a confidence greater than the set threshold, it is matched to a full frame {B vis2full} expanded from the visible frame according to the prior information of the pedestrian frame width-height ratio of 0.41 through the maximum intersection ratio principle. The visible frame is denoted as {B vis} Thus, the matching of the full frame and the visible frame is achieved, and the intersection ratio of each pair of full frame and visible frame is calculated as the visibility ratio vis_ratio, which is used to fuse the scores of the two. It can be represented as

[0037] merge_s = s vis_ratio + vis_s (1 - vis_ratio), (1)

[0038] where s and vis_s are the scores of the matched full box and visible box, respectively.

[0039] Finally, the non-maximum suppression post-processing is performed using the fusion score of the full box and the visible box to obtain the final full box.

[0040] The present application proposes an occluded pedestrian detection method based on full box and visible box fusion without anchor box, starting from the idea of fusing full box and visible box information. The method simultaneously predicts full box and visible box, and performs information fusion after matching the two, thereby solving the occlusion problem. The method includes image feature extraction, full box and visible box prediction without anchor box, and fusion detection of occluded pedestrians, and the specific steps are as follows:

[0041] Step 1: input the training sample, and use the convolution network to extract the features to obtain the feature map F;

[0042] Step 2: use two branches on the feature map F to predict the center points of the pedestrian full box and visible box and the height and width of the frame corresponding to the center points by full connection layer, by using the anchor box-free method;

[0043] Step 3: retain the center points with full box confidence greater than a set threshold, and generate the full box set {B full} of the pedestrian from the full box center points and the height and width of the frame corresponding to the center points, wherein the full box set {B full} contains the position and size of the full box and the score s;

[0044] Step 4: generate the pedestrian visible box at the position from the height and width of the frame corresponding to each pixel point on the feature map, and expand the full box {B vis2full} according to the pedestrian width-height ratio of 0.41;

[0045] Step 5: for each full box in the full box set in step 3, match to a full box expanded from the visible box in step 4 by maximum intersection ratio, and the visible box is {B vis}, wherein the visible box contains the position and size of the visible box and the score vis_s;

[0046] Step 6: set the visible ratio vis_ratio as the intersection ratio of the visible box and the full box, and the scores of the full box and the visible box are s and vis_s, respectively, and then the fusion score merge_s of the full box and the visible box is:

[0047] merge_s = s vis_ratio + vis_s (1 - vis_ratio), (1)

[0048] Step 7: Non-maximum suppression is performed on the full frame based on the visible frame and full frame fusion score described above to obtain the final full frame.

[0049] Specifically, the network parameter training of the present patent comprises the following steps:

[0050] Step 1: Prepare a training image set, each image of which contains at least one pedestrian, and give the pedestrian annotation information of the image in the training set, including the class and annotation box information of the pedestrian region. In addition, in order to improve the detection accuracy of the model, the data can be enhanced. The enhancement methods include but are not limited to: random flipping, color change, random cropping, adding random noise, image scaling, etc.

[0051] Step 2: Set the related hyperparameters in the training stage, including the input image size, input image batch size, iteration number, initial learning rate, learning rate weight decay coefficient, classification score threshold for training positive and negative examples, and the weight of each loss function.

[0052] Step 3: Initialize the weights of the convolutional feature extraction network and the full frame and visible frame prediction network. The convolutional feature extraction network can select ResNet, DenseNet, ResNeXt, etc. In addition, the ImageNet pre-trained model can be used to initialize the related convolutional weights. Set the loss function in the training stage, which includes the loss function of the visible frame and full frame prediction. The commonly used regression loss is L1 Smooth loss function, and the commonly used classification loss is Softmax loss function.

[0053] Step 4: According to the back propagation algorithm commonly used in deep convolutional networks, the gradient of each layer parameter is calculated, and the gradient descent method (SGD) is used to constantly update the weight parameters of the convolutional feature extraction network and the full frame and visible frame prediction network. The stop criterion can be the loss value of the loss function tending to converge (fluctuating around a stable value without significant change) to stop the iteration signal. When the iteration number is over, the learned weight parameters are the final network parameters.

[0054] Then, we introduce the specific test process:

[0055] Step 1: Prepare a training image set. If testing the performance of the model, the test set prepared should have the same data annotation format as the training image, and the mAP index of the test result is calculated as the performance measure. If it is a real application test, it does not need to have annotation information.

[0056] Step 2: input the test image into the network, and output the full frame and visible frame detection results by using the trained full frame and visible frame network. Based on the full frame and visible frame detection results, one-to-one matching is performed through the maximum intersection ratio strategy, the visible frame and full frame scores are fused according to the visible comparison, the non-maximum suppression (NMS) processing is performed on the full frame detection results according to the fusion score, the pedestrian detection frames with large overlap at the same position are removed, and the frame with higher confidence is reserved as the final pedestrian detection output.

[0057] Step 3: the model mAP index can be tested, and the detection frame of the pedestrian is displayed on the image.

Claims

1. An occlusion pedestrian detection method based on the fusion of anchorless full-frame and visible bounding box, comprising the following steps: Step 1: Input training samples and use a deep convolutional network to extract features from them to obtain feature maps. ; Step 2: In the feature map The method uses two branches with no anchor boxes, and obtains the confidence map of classification and the heatmap of regression through a fully connected layer to predict the center point of the full frame and the visible frame of the pedestrian, as well as the height and width of the full frame or the visible frame corresponding to the center point. Step 3: Retain center points with a full frame confidence score greater than a set threshold, and generate the full frame set of pedestrians from the center points and their corresponding frame height and width. This includes the position and size of the entire frame and the score. ; Step 4: For each pixel on the feature map, generate the pedestrian visible bounding box for that location based on the height and width of the corresponding bounding box, and set the value according to the pedestrian's aspect ratio. Expand into full frame ; Step 5: For each full frame in the full frame set from Step 3, match it using the maximum intersection-union ratio (MOU) to a full frame that is expanded from the visible frame in Step 4. This visible frame is... It includes the position and size of the visible box and the score. ; Step 6: Set the visibility ratio The intersection-union ratio (IUU) of the visible bounding box and the full bounding box is given by the scores of the full bounding box and the visible bounding box, respectively. and This indicates the score for blending the full frame and the visible frame. : Step 7: Based on the above visible box and full box fusion scores, perform non-maximum suppression on the full box to obtain the final full box.

Images