A dense pedestrian detection method with controllable label allocation based on global features

Abstract

The present invention discloses a dense pedestrian detection method with controllable label allocation based on global features, comprising the following steps: (1) collecting video data in a traffic scene and extracting images by frame; (2) marking the position of pedestrians and the specific position of the visible part of pedestrians in the image; (3) designing a frame visible weight function based on a loss function; (4) designing a small target weight function based on the loss function; (5) optimizing the label allocation strategy of positive samples; (6) optimizing the label allocation strategy of negative samples; (7) training a model and performing reasoning and prediction. The present invention has a stronger detection capability for occluded pedestrians and small target pedestrians, and the optimized label allocation strategy enables the model to better learn the characteristics of positive and negative samples.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and computer vision, and in particular to a dense pedestrian detection method with controllable label allocation based on global features. Background Art

[0002] Pedestrian detection is an indispensable module for intelligent vehicle assistance systems and autonomous driving. With the development of deep neural networks, object detection networks based on convolutional neural networks, such as YOLOv3 and FCOS, have achieved good results in pedestrian detection scenarios. However, due to the extremely complex pedestrian detection scenarios, the label allocation methods of various detectors cannot meet the requirements for effective detection of dense pedestrians.

[0003] For example, a Chinese patent document with publication number CN112464770A discloses a method for dense pedestrian detection in complex scenes, including collecting pedestrian images with monitoring equipment; extracting key frames through the network and transmitting the images to the MSR algorithm for image enhancement processing; and detecting and identifying the acquired pedestrian images using the YOLOv3 convolutional neural network model. This method realizes dense pedestrian detection in complex environments by combining the existing YOLOv3-based convolutional neural network model with the MSR algorithm.

[0004] The label assignment strategy in the classic YOLOv3 algorithm uses the addition of prior conditions to match each anchor box with a foreground object or background. However, the disadvantage of introducing anchor boxes is that since the number of anchor boxes generated by the algorithm is very large, while the number of foreground objects in the image is very small, the number of negative samples will be much larger than the number of positive samples, resulting in an imbalance between positive and negative samples. In addition, parameters such as the size and number of anchor boxes add more hyperparameters that need to be manually adjusted to the model. The anchor-free FCOS algorithm considers points that fall within the border of the ground truth (GT) of the real data as positive samples. However, this ignores the different positive and negative sample division criteria caused by issues such as size and occlusion, especially the difficulty in assigning blurred samples caused by a point falling into two GTs at the same time. The results of simply assigning positive and negative samples to each anchor box or each point without considering the context are often not the best.

[0005] Therefore, the OTA algorithm combines the context and assigns labels from a global perspective, demonstrating the optimal matching idea under global features, and adopts the Sinkhorn-Knopp iterative algorithm for one-to-many label assignment strategy, ultimately achieving good results in the field of pedestrian detection. However, dense pedestrian detection is still a very challenging task, and the OTA algorithm still has room for improvement, as shown below:

[0006] First, under global feature matching, occluded pedestrians are less likely to be assigned positive sample labels than nearby unoccluded pedestrians due to less effective information.

[0007] Secondly, due to the large differences in the sizes of pedestrians caused by the distance in the image, pedestrians at a long distance are too small to be correctly labeled by the detector.

[0008] Most importantly, since the OTA algorithm uses the Sinkhorn-Knopp iterative algorithm for one-to-many global feature label assignment, the algorithm will automatically assign the remaining anchor frames that have been assigned as positive samples as negative samples, that is, background. This results in some anchor frames that have a certain degree of overlap with GT being assigned as negative samples, which is not conducive to the training and learning of neural networks. Summary of the invention

[0009] The present invention provides a dense pedestrian detection method with controllable label allocation based on global features. By designing a box-visible weight function and a small target weight function and optimizing the existing label allocation strategy, the detection performance of occluded pedestrians and pedestrians that are smaller in the image due to their long distance is significantly improved.

[0010] A dense pedestrian detection method with controllable label allocation based on global features includes the following steps:

[0011] (1) Use monitoring equipment or vehicle-mounted video equipment to collect video data in traffic scenes and extract key frames from the video data;

[0012] (2) marking the position of pedestrians in the image data composed of key frames, further marking the specific position of the visible part of the pedestrians, and obtaining the size of each image;

[0013] (3) Construct a single-stage anchor-free box detection model FCOS and design a box visibility weight function based on the loss function to improve the detection capability of occluded pedestrians;

[0014] (4) Based on the loss function, a small target weight function is designed to improve the detection capability of pedestrians whose scale is small in the image due to their long distance.

[0015] (5) Design a label assignment strategy for positive samples to dynamically assign the number of positive samples that match each GT box;

[0016] (6) Design a label assignment strategy for negative samples, assigning anchor boxes that partially overlap with the GT box as ignored samples, and assigning the rest as negative samples based on the IoU threshold;

[0017] (7) Iterate the detection model for no less than 100 epochs and monitor the loss function until the model converges;

[0018] (8) Input the image to be detected into the trained detection model to perform dense pedestrian detection.

[0019] Furthermore, in step (2), the specific position of the visible part of the pedestrian is marked for designing the box visible weight function; and the size of each image is obtained for designing the small object weight function.

[0020] In step (3), the specific process of designing the box visible weight function based on the loss function is:

[0021] First, for a GT box, the position of the box (x l ,y l , x r ,y r ) to obtain the area A of the box, where x l ,y l , x r ,y r Represent the horizontal and vertical coordinates of the upper left corner and the lower right corner of the GT box respectively; the area A of the visible part of the box v and the area of ​​the entire box A f Divide to get the ratio of the visible part box to the entire box i represents the i-th GT, the ratio of the visible part box

[0022] After obtaining the ratio of the visible part frame, it is adjusted by the weight function f(x); the weight function f(x) is positively correlated with the ratio of the visible part frame. The lower the visible part ratio, the smaller the final weight. The range of the obtained visible part weight is

[0023] Finally, the weight of the visible part is multiplied by the loss function to obtain a new loss function. Each GT box will have a different weight size. The GT box with a larger visible part will eventually get a higher loss function. Therefore, when the OTA algorithm selects a smaller loss function, the GT with a smaller visible part will be more easily selected.

[0024] Weight function f(x) and visible part weight The formula is:

[0025] f(x)=x 2 +1

[0026]

[0027]

[0028] in, represents the weight of the visible part in the i-th GT box, Represents the loss function of the i-th GT box corresponding to the j-th anchor box, and the visible part weight Multiply by the loss function Get the updated loss function

[0029] In step (4), the specific process of designing the small target weight function based on the loss function is as follows:

[0030] (4-1) Calculate the area of ​​each GT based on the coordinates of each GT box

[0031]

[0032] in, Indicates the horizontal and vertical coordinate values ​​of the upper left corner of the i-th GT, Indicates the horizontal and vertical coordinate values ​​of the lower right corner of the i-th GT.

[0033] (4-2) Calculate the size A of the current image p ;

[0034] A p =W p ×H p

[0035] (4-3) Calculate the ratio of each GT area to the current image area

[0036]

[0037] (4-4) This ratio Normalize to get the final small target ratio

[0038]

[0039] (4-5) The obtained ratio Substitute it into the weight function f(x) for adjustment to get the small target weight

[0040] f(x)=x 2 +1

[0041]

[0042] (4-6) Small target weight Multiply by the loss function to get the updated loss function

[0043]

[0044] The weight function is positively correlated with the proportion of small targets. The lower the proportion of small targets, the smaller the final weight. Each GT will have a different weight. The larger the target, the higher the loss function. Therefore, when a smaller loss function is selected through the Sinkhorn-Knopp iterative algorithm, a smaller GT will be more easily selected.

[0045] In step (5), the label allocation strategy for positive samples is designed to improve the manually set value of positive samples in the OTA algorithm, changing the allocation from a static equal amount to a dynamic allocation strategy.

[0046] The label assignment strategy for positive samples is as follows:

[0047] By calculating the number p of anchor box center points in each GT, and taking p as the number of positive samples that the GT needs to match; and introducing the central sampling radius to determine that the samples within the GT central radius belong to the positive sample area, and the rest of the area is used as negative samples. By changing the size of the radius, the number of anchor boxes falling into the GT can be controlled; finally, the Soft-NMS post-processing algorithm is used to replace the NMS non-maximum suppression algorithm for post-processing operations, where the Gaussian factor σ is equal to 1.8.

[0048] In step (6), the label assignment strategy for negative samples is designed as follows:

[0049] Before the Sinkhorn-Knopp iterative algorithm performs label assignment, the anchor boxes with IoU in a certain interval [m, n] are screened out and the maximum IoU of these anchor boxes for all GTs is less than n; m and n are taken as 0.3 and 0.5 respectively; these selected anchor boxes become ignored samples, that is, they are neither positive samples nor negative samples, and these ignored samples are not used for subsequent loss function calculation; the rest are assigned as negative samples according to the IoU threshold.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. Compared with other target detection networks and OTA algorithms, the detection capability of the present invention for obscured pedestrians and small-target pedestrians is significantly improved.

[0052] 2. The present invention further optimizes the label assignment strategy for positive and negative samples. It not only dynamically assigns the number of positive samples for each GT match, but also removes the fuzzy anchor frames of positive and negative samples, so that the model can better learn the characteristics of positive and negative samples. Experiments have proved the effectiveness of the label assignment strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flowchart of a dense pedestrian detection method with controllable label allocation based on global features of the present invention;

[0054] Figure 2 It is a comparison diagram of examples of weight functions in the box of the present invention;

[0055] Figure 3 This is a diagram showing the predicted effect of the present invention on the input actual image;

[0056] Figure 4 This is a diagram showing the prediction effect of the present invention when the NMS threshold of the input actual image is equal to 1. DETAILED DESCRIPTION

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0058] like Figure 1 As shown, a dense pedestrian detection method with controllable label allocation based on global features includes the following steps:

[0059] Step 1: Collect video data from traffic scenes and road monitoring equipment, and extract the video data into frames of image data.

[0060] Step 2: Mark the position of pedestrians in the image data, and mark the position of the visible part of the pedestrians for the subsequent design of the frame visible weight function, and obtain the size of each image for the subsequent design of the small target weight function.

[0061] Step 3, based on the loss function, a box-visible weight function is designed to improve the detection capability of obscured pedestrians. The present invention designs different GTs with different importance. For obscured pedestrians, such GTs should be the first objects to be considered for matching. They should have a higher matching priority than ordinary GTs, that is, unobstructed pedestrians. Therefore, a box-visible weight function optimization strategy is designed, that is, different weights are given to pedestrian GTs with different degrees of occlusion. This can increase the probability that the GT of obscured pedestrians will be selected first. The specific method is described as follows:

[0062] First, for a GT box, it has the coordinates of the entire box and the coordinates of the visible part of the box. Therefore, the position of the box (x l ,y l , x r ,y r ) to get the area A of the box, where x l ,y l , x r ,y rRepresent the horizontal and vertical coordinates of the upper left corner and the lower right corner of the GT box respectively. Therefore, the area of ​​the visible part box A v and the area of ​​the entire box A f Dividing the two gives the ratio of the visible part to the entire frame. i represents the i-th GT, the visible part ratio

[0063] A=(x l -x r )×(y l -y r )

[0064]

[0065]

[0066]

[0067] After obtaining the visible portion ratio, it is adjusted by the weight function f(x). This function is positively correlated with the visible portion ratio. Therefore, the lower the visible portion ratio, the smaller the final weight. So the range of the visible part weight is Finally, the visible part weight is multiplied into the loss function, which means that each GT will have a different weight. The GT with a larger visible part will get a higher loss function. Therefore, when the OTA algorithm selects a smaller loss function, the GT with a smaller visible part will be more easily selected.

[0068] f(x)=x 2 +1

[0069]

[0070]

[0071] in, Represents the visible weight of the box of the i-th GT, Represents the loss function of the i-th GT corresponding to the j-th anchor box.

[0072] like Figure 2As shown in the figure, Q11 in the left figure represents the loss function value of the prediction box 1 for GT1, Q12 represents the loss function value of the prediction box 1 for GT2, and the same applies to Q21 and Q22. The first figure on the left shows the loss function of the prediction box and GT corresponding to the prediction box without the box visible weight function. The second figure on the left shows the label assigned according to the global feature in this case. It can be seen that Q22 is severely occluded and has poor quality, but it is also screened out. The third figure on the left shows the corresponding loss function value after using the box visible weight function. For GT with low visibility, its loss function value is reduced. The result on the right shows that the prediction box of the occluded pedestrian is matched first, and the detection ability of the occluded pedestrian is further improved.

[0073] Step 4: Based on the loss function, a small target weight function is designed to improve the detection capability of pedestrians whose scale is small in the image due to the distance. Small target objects have low confidence in the prediction box because they contain very few semantics. In this case, due to the global strategy of the OTA algorithm, small objects are difficult to be selected compared to large objects because of their higher loss function. Therefore, in response to this problem, this paper proposes an optimization scheme for the small target weight function. The specific steps are as follows.

[0074] (4-1) Calculate the area of ​​each GT based on the coordinates of each GT box

[0075]

[0076] in Indicates the horizontal and vertical coordinate values ​​of the upper left corner of the i-th GT, Represents the horizontal and vertical coordinate values ​​of the lower right corner of the i-th GT.

[0077] (4-2) Calculate the size A of the current image p ;

[0078] A p =W p ×H p

[0079] (4-3) Calculate the ratio of each GT area to the current image area

[0080]

[0081] (4-4) This ratio Normalize to get the final small target ratio

[0082]

[0083] (4-5) The obtained ratio Substitute it into the weight function f(x) for adjustment to get the small target weight

[0084] f(x)=x 2 +1

[0085]

[0086] (4-6) Small target weight Multiply by the loss function to get the updated loss function

[0087]

[0088] The weight function is positively correlated with the proportion of small targets. Therefore, the lower the proportion of small targets, the smaller the final weight. So the value range of the weight of the small target part is Finally, the weight of the small target is multiplied into the loss function. That is to say, each GT will have a different weight. The larger the target, the higher the loss function. Therefore, when the smaller loss function is selected by the Sinkhom-Knopp iterative algorithm, the smaller GT will be more easily selected.

[0089] Step 5, design a label allocation strategy for positive samples to dynamically allocate the number of positive samples that each GT matches. Calculate the number p of anchor box center points in each GT, and use p as the number of positive samples that the GT needs to match. The manually set value of positive samples in the OTA algorithm is improved, from a static equal number allocation to a dynamic allocation strategy. In addition, the central sampling radius is introduced to determine that the samples within the GT central radius belong to the positive sample area, and the remaining areas are negative samples. In this way, by changing the size of the radius, the number of anchor boxes that fall into the GT can be controlled. Finally, the Soft-NMS post-processing algorithm is used to replace the NMS non-maximum suppression algorithm for post-processing operations, where the Gaussian factor σ is equal to 1.8.

[0090] Step 6, design a label assignment strategy for negative samples, assign anchor boxes that partially overlap with GT as ignored samples, and assign the rest as negative samples based on the IoU threshold. For the label assignment matching strategy for negative samples, the OTA algorithm directly assigns all the remaining positive samples as negative samples, that is, background classes. Obviously, some anchor boxes that have a certain degree of overlap with GT are assigned as negative samples even though their quality is not particularly high. Therefore, in view of the problem of violent assignment of negative samples by the OTA algorithm, we have optimized the label assignment strategy for negative samples. The specific details are as follows:

[0091] Before the Sinkhorn-Knopp iterative algorithm performs label assignment, the anchor boxes with IoU in a certain interval [m, n] are screened out and the maximum IoU of these anchor boxes for all GTs must be less than n. These selected anchor boxes become ignored samples, that is, they are neither positive samples nor negative samples, and these samples are not used for subsequent loss function calculations. The specific implementation process is shown in Algorithm 1.

[0092]

[0093]

[0094] Among them, m and n are 0.3 and 0.5 respectively, and π represents the label sequence assigned by the Sinkhom-Knopp iterative algorithm.

[0095] Step 7: Perform iterative training for no less than 100 epochs and monitor the loss function until the model converges.

[0096] like Figure 3 As shown, it is the prediction effect diagram of the present invention for the input actual image, the two pictures on the left are the detection effect diagrams of the present invention, and the two pictures on the right are the detection effect diagrams of the original OTA algorithm. The pedestrians blocked at the places indicated by the arrows are detected by the present invention.

[0097] like Figure 4 As shown in the figure, it is the prediction effect diagram of the present invention for the input actual image with the NMS threshold equal to 1. Because there is no NMS post-processing algorithm to filter the prediction box, the box displayed in the image is the prediction result of the module; the two pictures on the right are the detection effect diagrams of the original OTA algorithm, and its prediction box is spread all over the image, while the two pictures on the left are the detection effect diagrams of the present invention, and its prediction box can basically be accurately concentrated near the GT, and the number of invalid negative samples is small.

[0098] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dense pedestrian detection method with controllable label allocation based on global features, characterized in that: The following steps are involved: (1) Use monitoring equipment or vehicle-mounted video equipment to collect video data in traffic scenes and extract key frames from the video data; (2) marking the position of pedestrians in the image data composed of key frames, further marking the specific position of the visible part of the pedestrians, and obtaining the size of each image; (3) Construct a single-stage anchor-free box detection model FCOS and design a box visibility weight function based on the loss function; specifically, the weight function f(k) is related to the visible part weight The formula is: f(k)=k 2 +1 in, represents the weight of the visible part in the i-th GT box, Represents the loss function of the i-th GT box corresponding to the j-th anchor box, and the visible part weight Multiply by the loss function Get the updated loss function The area of ​​the visible part of the frame is divided by the area of ​​the entire frame to get the ratio of the visible part of the frame to the entire frame. (4) Design a small target weight function based on the loss function; the specific process is as follows: (4-1) Calculate the area of ​​each GT based on the coordinates of each GT box in, Indicates the horizontal and vertical coordinate values ​​of the upper left corner of the i-th GT, Indicates the horizontal and vertical coordinate values ​​of the lower right corner of the i-th GT; (4-2) Calculate the size A of the current image p ; A p =W p ×H p (4-3) Calculate the ratio of each GT area to the current image area (4-4) This ratio Normalize to get the final small target ratio (4-5) The obtained ratio Substitute it into the weight function f(k) for adjustment to get the small target weight f(k)=k 2 +1 (4-6) Small target weight Multiply by the loss function to get the updated loss function The weight function is positively correlated with the proportion of small targets. The lower the proportion of small targets, the smaller the final weight. Each GT will have a different weight. The larger the target, the higher the loss function. (5) Design a label assignment strategy for positive samples to dynamically assign the number of positive samples that match each GT box; (6) Design a label assignment strategy for negative samples, assigning anchor boxes that partially overlap with the GT box as ignored samples, and assigning the rest as negative samples based on the IoU threshold; (7) Iterate the detection model for no less than 100 epochs and monitor the loss function until the model converges; (8) Input the image to be detected into the trained detection model to perform dense pedestrian detection.

2. The dense pedestrian detection method with controllable label allocation based on global features according to claim 1 is characterized in that: In step (2), the specific position of the visible part of the pedestrian is marked for designing the box visibility weight function; the size of each image is obtained for designing the small object weight function.

3. The dense pedestrian detection method with controllable label allocation based on global features according to claim 1 is characterized in that: In step (3), the specific process of designing the box visible weight function based on the loss function is: First, for a GT box, the position of the box (x l ,y l , x r ,y r ) to get the area of ​​the box Among them, x l ,y l , x r ,y r Represent the horizontal and vertical coordinates of the upper left corner and the lower right corner of the GT box respectively; the area A of the visible part of the box v and the area of ​​the entire box A f Divide to get the ratio of the visible part box to the entire box Represents the ratio of the visible part box of the i-th GT After obtaining the ratio of the visible part frame, it is adjusted by the weight function f(k); the weight function f(k) is positively correlated with the ratio of the visible part frame. The lower the visible part ratio, the smaller the final weight. The range of the obtained visible part weight is Finally, the weight of the visible part is multiplied by the loss function to obtain a new loss function. Each GT box will have a different weight size. The GT box with a larger visible part will eventually get a higher loss function. Therefore, when the OTA algorithm selects a smaller loss function, the GT with a smaller visible part will be more easily selected.

4. The dense pedestrian detection method with controllable label allocation based on global features according to claim 1 is characterized in that: In step (5), the label assignment strategy for the positive sample is designed to be a dynamic assignment strategy; the label assignment strategy for the positive sample is specifically: The number d of anchor box center points in each GT is calculated, and d is used as the number of positive samples that the GT needs to match. The central sampling radius is introduced to determine that the samples within the central radius of the GT belong to the positive sample area, and the remaining areas are used as negative samples. By changing the size of the radius, the number of anchor boxes falling into the GT can be controlled. Finally, the Soft-NMS post-processing algorithm is used for post-processing operations, where the Gaussian factor σ is equal to 1.

8.

5. The dense pedestrian detection method with controllable label allocation based on global features according to claim 4 is characterized in that: In step (6), the label assignment strategy for negative samples is designed as follows: Before the Sinkhorn-Knopp iterative algorithm performs label assignment, the anchor boxes with IoU in a certain interval [m, n] are screened out and the maximum IoU of these anchor boxes for all GTs is less than n; m and n are taken as 0.3 and 0.5 respectively; these selected anchor boxes become ignored samples, that is, they are neither positive samples nor negative samples, and these ignored samples are not used for subsequent loss function calculation; the rest are assigned as negative samples according to the IoU threshold.

Images