Multi-instance pedestrian detection method based on improved soft-nms algorithm

Abstract

The application relates to a multi-instance pedestrian detection method based on an improved Soft-NMS algorithm, and comprises the following steps: acquiring a pedestrian data set, constructing and improving a Faster RCNN network, training the improved Faster RCNN network based on the pedestrian data set, acquiring a detection result based on the trained Faster RCNN network, improving a Soft-NMS algorithm, performing a post-processing operation on the detection result based on the improved Soft-NMS algorithm, deleting repeated detection results, and obtaining a final detection result. The application can fully utilize additional information to assist detection and avoid missed detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a multi-instance pedestrian detection method based on an improved Soft-NMS algorithm. BACKGROUND

[0002] Pedestrian detection, as an important research branch of target detection, has been applied to various aspects of real life. For example, in intelligent driving, it can quickly help the car locate the position of the pedestrian, so that the car can make accurate judgments; in the field of robots, it can make the robot well recognize the pedestrian and make appropriate responses according to the analysis of the pedestrian, so as to better serve human beings. Therefore, the research on pedestrian detection method has considerable significance.

[0003] Most of the traditional pedestrian detection methods adopt sliding window algorithm, which is low in time efficiency, is processed by multiple steps, is complex and has low accuracy. In addition, due to the continuous change of the environment of pedestrians, the performance requirements of pedestrian detection methods in actual scenes are gradually increasing. Especially in a more complex pedestrian scene, the position, size scale and occlusion degree of pedestrians have a greater impact on accuracy. And with the increasing demand for computing power and the increasing amount of data, the traditional detection algorithm cannot meet the needs of reality.

[0004] With the rapid development of deep learning technology, the existing pedestrian detection algorithm based on deep learning extracts pedestrian features through convolutional neural network, uses loss function to train a large number of pedestrian data sets, and obtains a better pedestrian detection model, which can effectively realize pedestrian detection. However, the existing deep learning-based target detection algorithm still has the following problems in detecting pedestrians in dense scenes: (1) it does not fully utilize additional information to assist detection. (2) Due to the overlap between pedestrians, the original NMS algorithm is prone to delete the prediction box of the adjacent pedestrian, resulting in missed detection. SUMMARY

[0005] The purpose of the present application is to provide a multi-instance pedestrian detection method based on an improved Soft-NMS algorithm, which can fully utilize additional information to assist detection and avoid missed detection.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] The multi-instance pedestrian detection method based on the improved Soft-NMS algorithm comprises:

[0008] obtaining a pedestrian dataset, constructing and improving a Faster RCNN network, training the improved Faster RCNN network based on the pedestrian dataset, and obtaining a detection result based on the trained Faster RCNN network;

[0009] improving a Soft-NMS algorithm, performing a post-processing operation on the detection result based on the improved Soft-NMS algorithm, deleting duplicate detection results, and obtaining a final detection result.

[0010] Optionally, the constructing and improving the Faster RCNN network comprises:

[0011] The Faster RCNN network is constructed by connecting a Backbone part, a Neck part, and a Head part in sequence, and an additional prediction branch is added after a fully connected layer of the Head part to improve the Faster RCNN network.

[0012] The additional prediction branch is used to assist in detection by using the pedestrian dataset. The Head part comprises two fully connected layers and a prediction branch. The additional prediction branch has the same structure as the prediction branch.

[0013] Optionally, the obtaining the detection result comprises:

[0014] The pedestrian dataset is input, the Backbone part extracts features of pedestrians in the pedestrian dataset by using a Resnet50 network, and a feature map is obtained.

[0015] The Neck part comprises an RPN network and an ROI Align. The RPN network is used to generate candidate boxes of pedestrians, and the candidate boxes are projected to the feature map to obtain features in the candidate boxes. The features in the candidate boxes are scaled to the same size based on the ROI Align.

[0016] The fully connected layer of the Head part uses the features in each candidate box to predict a full-body box of a pedestrian by using a prediction branch. The additional prediction branch predicts a visible box of the pedestrian based on the features in each candidate box, and outputs the visible box and the full-body box of the pedestrian, i.e., the detection result.

[0017] Optionally, the generating the candidate boxes of pedestrians by using the RPN network comprises:

[0018] Each point on the feature map corresponds to a position of an image in the pedestrian dataset, a plurality of anchor boxes with targets are generated, and target probabilities and regression parameters of the anchor boxes are obtained by using a connection layer of the RPN network.

[0019] correct the position of the anchor frame based on the regression parameter, sort the anchor frame by the score of the target probability, delete the anchor frame exceeding the image boundary, perform NMS algorithm processing on the anchor frame after deleting the anchor frame exceeding the image boundary, delete the repeated anchor frame, sort the anchor frame after NMS algorithm processing by the score of the target probability from large to small, and select a preset number of anchor frames as the candidate frame.

[0020] Optionally, scaling the features in the candidate frame to the same size based on the ROI Align includes:

[0021] dividing the features in the candidate frame into regions of the same size, determining the number of sampling points as N, dividing each region into N squares, taking the center of each square as a sampling point, performing max pooling on the sampling points, and scaling the features in the candidate frame to the same size.

[0022] Optionally, the post-processing operation on the detection result based on the improved Soft-NMS algorithm includes:

[0023] S1.1: sorting the visible frame according to the score of the visible frame, selecting a first visible frame with the highest score, calculating the first visible frame with the visible frame excluding the first visible frame, and obtaining a first IOU value of the visible frame;

[0024] S1.2: selecting a second visible frame with the highest score from the visible frame excluding the first visible frame, calculating the second visible frame with the visible frame excluding the second visible frame, and obtaining a second IOU value of the visible frame;

[0025] S1.3: according to S1.1 and S1.2, obtaining a plurality of IOU values of the visible frame.

[0026] Optionally, the post-processing operation on the detection result based on the improved Soft-NMS algorithm further includes:

[0027] S2.1: sorting the full-body frame according to the score of the full-body frame, selecting a first full-body frame with the highest score, calculating the first full-body frame with the full-body frame excluding the first full-body frame, and obtaining a first IOU value of the full-body frame;

[0028] S2.2: selecting a second full-body frame with the highest score from the full-body frame excluding the first full-body frame, calculating the second full-body frame with the full-body frame excluding the second full-body frame, and obtaining a second IOU value of the full-body frame;

[0029] S2.3: according to S2.1 and S2.2, obtaining a plurality of IOU values of the full-body frame.

[0030] Optionally, the sorting according to the score of the whole body bounding box comprises: updating the score of the whole body bounding box.

[0031] The updating of the score of the whole body bounding box comprises:

[0032] respectively judging whether the IOU values of the visible bounding box and the whole body bounding box are both less than a preset threshold value, if the IOU values of the visible bounding box and the whole body bounding box are both less than the preset threshold value, not updating the score of the whole body bounding box, and if the IOU values of the visible bounding box and the whole body bounding box are both not less than the preset threshold value, weighting the score of the whole body bounding box based on a function of the IOU value in the visible bounding box, and updating the score of the whole body bounding box.

[0033] Optionally, the deleting of the repeated detection result comprises: deleting the repeated whole body bounding box.

[0034] The deleting of the repeated whole body bounding box comprises:

[0035] judging whether the score of the whole body bounding box is greater than a preset score threshold value, if the score of the whole body bounding box is greater than the preset score threshold value, deleting the whole body bounding box, and if the score of the whole body bounding box is not greater than the preset score threshold value, keeping the whole body bounding box.

[0036] Optionally, the improvement of the Soft-NMS algorithm comprises:

[0037] replacing the IOU in the penalty function with the IOU of the visible bounding box based on the IOU value of the visible bounding box as an auxiliary judgment of the overlap condition.

[0038] The present application has the following beneficial effects:

[0039] (1) An additional prediction branch is added to the Head part in the Faster RCNN for multiple instance prediction to obtain the visible bounding box of the pedestrian, which is used to assist in judging the overlap condition between pedestrians and reduce the missed detection of pedestrians due to the overlap of pedestrians.

[0040] (2) The present application improves the Soft-NMS algorithm by using the whole body bounding box and visible bounding box information of the pedestrian, so that the improved Soft-NMS algorithm can better judge the overlap condition between pedestrians, thereby further reducing the missed detection of pedestrians. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0042] Figure 1 The flow chart of the multi-instance pedestrian detection method based on the improved Soft-NMS algorithm is an embodiment of the present application.

[0043] Figure 2 The improved Faster RCNN network structure diagram is an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0046] As shown in Figure 1 The present application discloses a multi-instance pedestrian detection method based on an improved Soft-NMS algorithm, which comprises the following steps: obtaining a pedestrian data set, constructing and improving a Faster RCNN network, training the improved Faster RCNN network based on the pedestrian data set, obtaining a detection result based on the trained Faster RCNN network, improving a Soft-NMS algorithm, performing a post-processing operation on the detection result based on the improved Soft-NMS algorithm, deleting repeated detection results, and obtaining a final detection result, specifically as follows:

[0047] The multi-instance pedestrian detection method based on the improved Soft-NMS algorithm comprises the following steps:

[0048] Step 1: input a pedestrian data set containing a training set and a validation set;

[0049] Step 2: train a pedestrian detection network;

[0050] Step 3: test the pedestrian detection network;

[0051] Step 4: perform post-processing;

[0052] In step 1, the training data set with additional labeled pedestrian visible frames is inputted.

[0053] In step 2, the improved Faster RCNN network is trained using the training data set.

[0054] In step 3, the trained Faster RCNN network is tested using the test data set to obtain a detection result.

[0055] In step 4, the obtained test result is post-processed, that is, the improved Soft-NMS algorithm is used to delete the repeated test results to obtain the final detection result, and the improved Soft-NMS algorithm is as follows.

[0056]

[0057]

[0058] The pedestrians in the image are detected through the above steps.

[0059] The improved Faster RCNN network structure is as shown in the following figure, which is composed of three parts: a Backbone part, a Neck part and a Head part. Figure 2

[0060] The Backbone part uses a Resnet50 network to extract the features of the pedestrians in the image to obtain a feature map of the image, and the Resnet50 network is a deep residual network that can avoid the problem of low learning efficiency and ineffective improvement of accuracy due to deepening of the network depth. The Neck part is composed of an RPN network and an ROI Align. The RPN network is used to generate candidate boxes of pedestrians, and then the candidate boxes generated by the RPN network are projected onto the image feature map extracted by the Resnet50 network to obtain the features in the candidate boxes, and the ROI Align scales the features in the candidate boxes to the same size. The Head part is composed of two fully connected layers and two prediction branches, which respectively predict the visible box and the whole body box of the pedestrian for each feature in the candidate box, and the whole body box represents the whole body of the pedestrian and the visible box represents the visible part of the pedestrian.

[0061] In step 2, the following sub-steps are included:

[0062] ​Step 2-1) Obtain the features of the candidate box: input the pedestrian image to be detected into the backbone network for feature extraction, the backbone network adopts Resnet50, different features of the image are extracted through multiple layers of convolution, the feature map of the image is obtained, and the problem that the network converges more slowly and the accuracy is also reduced due to the depth of the convolutional neural network reaching a certain depth is well solved by adding a residual structure. Then input the feature map extracted by Resnet50 into the RPN network: first, each point on the feature map is mapped to each position in the original image to generate k anchor boxes that may contain targets, i.e. anchor boxes, then two fully connected layers are used to obtain the target probability and regression parameters of the anchor box, then the regression parameters are used to correct the position of the anchor box, then the anchor boxes are sorted in descending order according to the target probability score, the first N anchor boxes are selected, then the anchor boxes that exceed the image boundary are deleted, then the remaining anchor boxes are processed by the NMS algorithm to delete duplicate anchor boxes, finally the anchor boxes processed by the NMS algorithm are sorted in descending order according to the target probability score, and the first M anchor boxes are selected as the output candidate boxes. Then map the candidate boxes to the feature map of the image to obtain the features in the candidate boxes.

[0063] Step 2-2) Multi-instance prediction on the features of the candidate box: input the features of the obtained candidate box into ROIAlign: first, divide the features in the candidate box into regions of the same size; then determine the number of sampling points to be N, divide each region into N squares, and take each square as the center as the sampling point; then perform bilinear interpolation on the sampling point pixels to obtain the pixel value of the sampling point; finally, perform max pooling on the N sampling points in each region to obtain a feature map of the same size. Then input the feature map into the Head part and then perform multi-instance prediction to predict the visible box and the full-body box of the pedestrian respectively.

[0064] In step 2, the Faster RCNN is optimized as follows:

[0065] In dense scenes, the high overlap between pedestrians makes the candidate boxes generated by the RPN network also highly overlapped, which finally leads to the overlap of the final prediction results. If only the final detected full body box is relied on for the subsequent post-processing operation, it will easily cause the adjacent pedestrians to be missed. Therefore, additional auxiliary information needs to be introduced to help the subsequent post-processing operation. Therefore, an additional prediction branch is added after the two fully connected layers in the Head part to predict the visible box of the pedestrian, which is used to assist detection through the pedestrian dataset. The branch has the same structure as the original prediction branch, but the final output result is different; wherein the original prediction branch is the part after the two fully connected layers for prediction and regression, i.e. Figure 2 the Reg and cls parts in the formula.

[0066] The following sub-steps are included in step 4:

[0067] Step 4-1): Calculate the IOU value between the pedestrian visible box and the full body box respectively: first, sort all visible candidate boxes in descending order according to the score of the visible candidate box, then select the highest score visible candidate box from the visible candidate boxes, and calculate the IOU value between it and the remaining visible candidate boxes one by one. After traversing all visible candidate boxes, then select the highest visible candidate box from the remaining visible candidate boxes, and calculate the IOU value between it and the remaining visible candidate boxes one by one, and so on to calculate the IOU value between each visible candidate box and all visible candidate boxes with a lower score. Similarly, the IOU value of the full body box can be obtained.

[0068] Step 4-2): Update the score of the full body box: respectively judge whether the IOU values of the visible box and the full body box between pedestrians are less than the set threshold value, if the IOU values of the visible box and the full body box are less than the set threshold value, do not update the score of the full body box, otherwise use the function of the visible box IOU value to weight the score of the full body box to update the score of the full body box.

[0069] Step 4-3): Delete duplicate full body boxes: judge whether the scores of all full body boxes are greater than the set score threshold, if greater than the set threshold, delete the full body box. After judging all full body boxes, the final detection result is obtained.

[0070] In step 4, the Soft-NMS algorithm is improved as follows:

[0071] In pedestrian detection, after obtaining the prediction results, post-processing algorithm operation is needed, that is, using NMS algorithm to delete duplicate prediction results. The original NMS directly sets 0 to the score of the bounding box with IOU value greater than the set threshold, thereby deleting the bounding box greater than the threshold. However, when there is overlap between adjacent candidate boxes, that is, occlusion, it is easy to cause missed detection. Based on the prior art, the original NMS algorithm is improved. For the bounding box with IOU value greater than the threshold, Soft-NMS adopts a score penalty mechanism to reduce the score of the bounding box, that is, using a penalty function positively correlated with IOU to punish the score. When the adjacent detection box has a large IOU value with the previous box, the score of the detection box is not directly set to 0, but the score of the detection box is suppressed using the penalty function, and the non-overlapping box is not affected. Soft-NMS has two penalty functions, the first penalty function formula is as follows:

[0072]

[0073] Wherein, f(iou(M,b i )) represents a linear decay type penalty function, M represents the current box, b i represents the detection box, iou(M,b i ) represents the overlap degree of M and b i , N t represents the set threshold.

[0074] The second penalty function formula is as follows:

[0075]

[0076] Wherein, f(iou(M,b i )) represents an exponential Gaussian type penalty function, M represents the current box, b i represents the detection box, iou(M,b i ) represents the overlap degree of M and b i , D represents the saved detection box after Soft-NMS suppression, σ is a constant, is any b i .

[0077] Although Soft-NMS can better alleviate the object occlusion missing detection caused by the original NMS violent deletion, there are still the following problems: (1) Due to the high overlap between pedestrians, the IOU value between adjacent full body boxes is large, resulting in the score of adjacent full body boxes after Soft-NMS suppression being less than the set threshold, and further causing missing detection. (2) When two overlapping full body boxes are predicted to be the same pedestrian, the Soft-NMS algorithm should reduce the score of one of the full body boxes as much as possible; when two overlapping full body boxes are predicted to be different pedestrians, the Soft-NMS algorithm should not reduce the score of one of the full body boxes as much as possible, but it is difficult to distinguish between the two cases using only the IOU value between the full body boxes. Therefore, based on the above two problems, the Soft-NMS is improved by introducing additional pedestrian visible box information. When two overlapping full body boxes are predicted to be the same pedestrian, the IOU value between their visible boxes is large; when two overlapping full body boxes are predicted to be different pedestrians, the IOU value between their visible boxes is small. Therefore, the IOU value between the pedestrian visible boxes is used as an auxiliary condition to determine overlap, and the IOU in the penalty function is replaced by the IOU of the visible box. The improved formula is as follows:

[0078]

[0079] wherein f(overlop V ) represents a linear decay type penalty function of the visible box, M v represents the visible box of the current frame, represents the visible box of the detection frame, represents M and b i the overlap degree of the visible box, N t represents the set threshold, overlop V is the IOU value between the pedestrian visible boxes.

[0080]

[0081] wherein f(overlop V ) represents an exponential Gaussian type penalty function of the visible box, M v represents the visible box of the current frame, represents the visible box of the detection frame, represents M and b i the overlap degree of the visible box, D represents the detection frame after Soft-NMS suppression, e is the natural logarithm, σ is a constant, is any

[0082]

[0083] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A multi-instance pedestrian detection method based on an improved Soft-NMS algorithm, characterized in that, The application relates to a pedestrian detection method based on Faster RCNN network. The application comprises the following steps: Obtaining a pedestrian data set, constructing and improving a Faster RCNN network, training the improved Faster RCNN network based on the pedestrian data set, and obtaining a detection result based on the trained Faster RCNN network. The improved Faster RCNN network comprises a Backbone part, a Neck part and a Head part, the Head part is composed of two fully connected layers and two prediction branches, and is used for respectively predicting a visible frame and a full-body frame of a pedestrian in each candidate frame; wherein the full-body frame represents a full body of the pedestrian, and the visible frame represents a visible part of the pedestrian. An improved Soft-NMS algorithm is used: for a frame with an IOU value greater than a threshold value, the Soft-NMS algorithm adopts a score penalty mechanism to reduce the score of the frame, that is, a penalty function positively correlated with the IOU value is used to punish the score, when a neighboring detection frame has an IOU value greater than a threshold value with a previous frame, the score of the detection frame is not set to 0, but the score of the detection frame is suppressed by using the penalty function, and non-overlapping frames are not affected. Based on the improved Soft-NMS algorithm, post-processing operation is performed on the detection result to obtain a final detection result, which comprises the following steps: S1.1, IOU values between visible frames and full-body frames are respectively calculated: first, all visible frames are sorted in descending order according to the scores of the visible frames, and the first visible frame with the highest score is selected, and the first visible frame is sequentially calculated with the remaining visible frames to obtain first IOU values, after all visible frames are traversed, the highest visible frame is then selected from the remaining visible frames, that is, the second visible frame, and the second visible frame is sequentially calculated with the remaining visible frames to obtain second IOU values, and IOU values of each visible frame with all visible frames with a score lower than the frame are calculated in the same way, and IOU values of full-body frames can be obtained in the same way. S1.2, the score of the full-body frame is updated: whether the IOU values of the visible frame and the full-body frame are both less than a set threshold value is respectively judged, if the IOU values of the visible frame and the full-body frame are both less than the set threshold value, the score of the full-body frame is not updated, otherwise the score of the full-body frame is weighted by using a function of the IOU value of the visible frame, and the score of the full-body frame is updated. 2.The improved Soft-NMS algorithm based multi-instance pedestrian detection method according to claim 1, wherein, S1.3, repeated full-body frames are deleted: whether the scores of all full-body frames are greater than a set score threshold value is judged, if the scores of all full-body frames are greater than the set score threshold value, the full-body frame is deleted, and after all full-body frames are judged, the final detection result is obtained. The Faster RCNN network is constructed and improved in the following steps: The Faster RCNN network is constructed by connecting a Backbone part, a Neck part and a Head part in sequence, and the Faster RCNN network is improved by adding an additional prediction branch after the fully connected layer of the Head part. The additional prediction branch is used to assist detection by using the pedestrian data set, the Head part comprises two fully connected layers and a prediction branch, and the additional prediction branch has the same structure as the prediction branch. 3.The improved Soft-NMS algorithm based multi-instance pedestrian detection method according to claim 2, wherein, The detection result comprises: The input pedestrian data set, the Backbone part extracts the features of pedestrians in the pedestrian data set through a Resnet50 network, and obtains a feature map; The Neck part comprises an RPN network and an ROI Align, generates a candidate box of pedestrians through the RPN network, projects the candidate box to the feature map, obtains the features in the candidate box, and scales the features in the candidate box to the same size based on the ROI Align; The full connection layer of the Head part obtains the features in each candidate box, predicts a full body box of pedestrians through a prediction branch, the additional prediction branch predicts a visible box of pedestrians based on the features in each candidate box, and outputs the visible box and the full body box of the pedestrians, that is, the detection result.

4. The method of claim 3, wherein the improved Soft-NMS algorithm-based multi-instance pedestrian detection method is characterized by, The RPN network generates a candidate box of pedestrians, which comprises: Each point on the feature map corresponds to the position of an image in the pedestrian data set, a plurality of anchor boxes with targets are generated, the target probability and regression parameters of the anchor boxes are obtained through the connection layer of the RPN network, the positions of the anchor boxes are corrected based on the regression parameters, the anchor boxes are sorted according to the scores of the target probability, and the anchor boxes exceeding the image boundary are deleted, the anchor boxes after the deletion of the anchor boxes exceeding the image boundary are processed by the NMS algorithm, the repeated anchor boxes are deleted, the anchor boxes processed by the NMS algorithm are sorted in descending order according to the scores of the target probability, and a preset number of anchor boxes are selected as the candidate box. The ROI Align scales the features in the candidate box to the same size, which comprises:

5. The method of claim 3, wherein the improved Soft-NMS algorithm-based multi-instance pedestrian detection method is characterized by, The features in the candidate box are divided into regions of the same size, the number of sampling points is determined as N, each region is divided into N squares, the center of each square is taken as a sampling point, max pooling is performed on the sampling points, and the features in the candidate box are scaled to the same size. The IOU value of the full body box comprises: 6.The improved Soft-NMS algorithm based multi-instance pedestrian detection method according to claim 1, wherein, S2.1: The full body boxes are sorted according to the scores of the full body boxes, the first full body box with the highest score is selected, the first full body box is calculated with the full body boxes excluding the first full body box, and the first IOU value of the full body box is obtained; S2.2: The second full body box with the highest score is selected from the full body boxes excluding the first full body box, the second full body box is calculated with the full body boxes excluding the second full body box, and the second IOU value of the full body box is obtained; S2.3: According to steps S2.1 and S2.2, a plurality of IOU values of the full body box are obtained. The improved Soft-NMS algorithm comprises:

7. The method of claim 6, wherein the improved Soft-NMS algorithm-based multi-instance pedestrian detection method is characterized by, The IOU value of the visible box is used as an auxiliary judgment of the overlap condition, and the IOU in the penalty function is replaced by the IOU of the visible box. ​

Images