A method for detecting occluded pedestrians based on head and visible area clues

By adopting a detection method based on head and visible area clues in advanced assisted driving systems, combined with SRPN, VGA and HRCNN, the problem of high detection missed detection rate of shading pedestrians in complex road environments is solved, and the detection accuracy and performance are significantly improved.

CN113610037BActive Publication Date: 2024-11-12BEIJING INST OF COMP TECH & APPL
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202110942302.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-17
Publication Date
2024-11-12
Estimated Expiration
2041-08-17

AI Technical Summary

Technical Problem

The missed detection rate of occlusion pedestrian detection in complex road environments is high, resulting in insufficient occlusion pedestrian detection performance of advanced assisted driving systems.

Method used

Using a detection method based on head and visible area cues, combined with residual network and feature pyramid network, the sharpening feature response area recommendation network (SRPN) and visible area guidance attention module (VGA) and the head cascade detection network ( HRCNN), extracts pedestrian whole body and head features, enhances visible area features, suppresses occlusion noise, and improves detection accuracy.

Benefits of technology

It significantly improves the detection accuracy of occluding pedestrians, reduces the missed detection rate, and improves the detection performance of occluding pedestrians of advanced assisted driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113610037B_ABST
    Figure CN113610037B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for detecting occluded pedestrians based on head and visible area clues, and belongs to the field of assisted driving. The present invention uses head, visible area, and whole body information to improve the detection performance of occluded pedestrians in complex road scenes, and proposes a sharp feature response region recommendation network SRPN to simultaneously detect the whole body and head RoIs of pedestrians. For whole body RoIs, a visible area attention-guiding VGA module is added before the original whole body detection branch Fast RCNN to enhance the visible area features while suppressing the occluded area noise, and a parallel head cascade detection branch is added to detect severely occluded pedestrians based on the step-by-step expansion of the perception domain of head RoIs. A large number of experimental results show that the method proposed in this patent can significantly improve the detection accuracy of occluded pedestrians.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of assisted driving, and in particular relates to a method for detecting occluded pedestrians based on head and visible area clues. Background Art

[0002] The pedestrian detection module in the Advanced Driver Assistance Systems (ADAS) automatically detects pedestrians from images based on computer vision methods, and then reminds the driver to pay attention to pedestrians in front or triggers the driving system to automatically perform protective measures in time, which can significantly reduce the occurrence of collision accidents. It is one of the core modules for realizing intelligent driving. The pedestrian detection algorithm for driving applications is different from general object detection. The former has stricter restrictions on the number of missed detections caused by occluded pedestrians. The high missed detection rate of occluded pedestrian detection will cause traffic accidents, resulting in loss of life and property.

[0003] At present, the pedestrian detection algorithm improved based on the object detection framework has the following difficulties in detecting occluded pedestrians: 1) lack of human body information of the occluded part; 2) diverse appearance of the occluded object; 3) diverse occlusion modes of pedestrians; 4) diverse appearance, scale, and posture of pedestrians themselves. The occlusion problem of pedestrians in the crowd can be considered as an intra-class occlusion problem. When different pedestrians overlap, the detection frame of the target may be mistakenly transferred to another pedestrian and ignore the occluded pedestrian because the features cannot distinguish the individual instances.

[0004] At present, the solution to the pedestrian occlusion problem can be divided into a holistic model and a part-based model. The holistic method usually pays more attention to the partial occlusion problem of general pedestrians (reasonable), whose occlusion area is less than 35%. However, for severely occluded pedestrians, the detector performance will drop rapidly as the proportion of the intruding occluding object area increases. In response to the problems of the holistic model, some researchers have proposed a part model, which learns a series of body part detectors to detect various parts of the human body respectively. When a part of the human body is occluded, the detector of the visible part can still work normally. Then the part model will comprehensively judge whether the current object is a pedestrian or a virtual scene by combining the results of all part detectors. However, the current pedestrian detection datasets usually lack part labels. The method based on the part model obtains part annotation information by designing part templates and statistically analyzing the visible area pattern. However, due to the diversity of postures and occlusion patterns, the part annotations generated by such methods are biased. Therefore, how to reduce the missed detection rate of occluded pedestrian detection in complex road environments for driving scenarios and improve the occluded pedestrian detection performance of advanced driver assistance systems is a problem worth studying. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is how to provide a method for detecting obstructed pedestrians based on head and visible area clues, so as to reduce the missed detection rate of obstructed pedestrian detection in complex road environments for driving scenes and improve the obstructed pedestrian detection performance of advanced driver assistance systems.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention proposes a method for detecting occluded pedestrians based on head and visible area clues, which comprises the following steps:

[0009] The backbone network uses the residual network ResNet50 and the feature pyramid network FPN, and the backbone network outputs a multi-scale feature pyramid;

[0010] The multi-scale feature pyramid is sharpened through the feature response region proposal network SRPN to output the pedestrian's whole body RoIs and head RoIs;

[0011] The features obtained by pooling the whole body RoIs enter the whole body detection branch. The visible area guided attention module VGA estimates the pedestrian visible area probability map in the whole body rectangular frame based on this feature, and then optimizes the original features by dot multiplication, suppresses the noise features of occluded objects and enhances the features of the pedestrian visible area. After that, it is input to Fast RCNN for detection and outputs the whole body detection results.

[0012] The pooled features obtained by pooling the head RoIs are input to the head cascade detection branch, which includes the head cascade detection network HRCNN. The network gradually obtains the visible area detection frame from the head detection frame in stages, and finally obtains the pedestrian full body detection frame;

[0013] The whole body detection results output by the whole body detection branch and the head cascade detection branch are post-processed using Soft-NMS.

[0014] Furthermore, the sharpened feature response region recommendation network SRPN includes two identical generation modules for outputting whole body and head RoIs; the generation module includes a 3×3×256 convolutional layer, and three parallel 1×1×256 convolutional layers output the target center probability map, the scale estimation map, and the center offset map respectively; the center score, scale, and center offset based on the above output are converted into a target external rectangular frame through RoIs conversion, and the final region of interest is obtained after NMS post-processing, and the whole body RoIs or head RoIs are output.

[0015] Furthermore, the SRPN extracts RoIs on the feature pyramids of five sizes from P3 to P7 output by the FPN.

[0016] Furthermore, the center probability map branch is trained using the average binary cross-entropy (BCE) loss function; the target center probability map GT center The generation of is achieved by superimposing a two-dimensional Gaussian distribution map at each target center position (i, j). The specific formula is as follows:

[0017] G(i,j,x,y,σ)=exp(-(ix) 2 / (2σ 2 )-(jy) 2 / (2σ 2 )) (1)

[0018]

[0019] Where K is the number of positive samples in the image, (x k ,y k ) is the center position of the circumscribed rectangular box of the kth target, the value of the variance σ of the Gaussian distribution map is related to the pyramid level 1 of the current feature, and (i, j) is the coordinate position of the feature map.

[0020] Furthermore, the target scale estimation graph GT scale Generated according to the following formula:

[0021]

[0022] In the formula, h k is the height of the kth target, and r is the radius of the positive sample sampling range.

[0023] Furthermore, the center offset of the center offset map branch is calculated according to the following formula:

[0024]

[0025] Where, t ij It is the deviation of the actual center position of the target in the coordinates of the 1st layer feature map (i, j).

[0026] Furthermore, the head rectangle annotation can be calculated as follows:

[0027]

[0028] In the formula, (x f ,y f )The center coordinate of the whole body annotation box, (x h ,y h )Head center, h f is the pedestrian height, s h It is the head size.

[0029] Furthermore, the visible region guided attention module VGA is implemented as follows: RoI pooling feature is denoted as F r ∈[C×H r ×W r ], where C is the number of channels, H r and W r is the feature resolution; F r After inputting VGA, the attention features are encoded through four C×3×3 convolutional layers and one 1×1 convolutional layer. Finally, the normalized visible area attention template is obtained after the pixel-by-pixel Sigmoid function, which is recorded as A. V ∈[1×H r ×W r ]. By r With A V Multiply pixel by pixel to get the filtered feature F VGA (i, :, :) = F r (i, :, :)·A V , the filtered features are input to Fast RCNN for detection.

[0030] Furthermore, the calculation process of the head cascade detection network HRCNN is divided into three stages: the first stage, B 0 are the head RoIs generated by SRPN, based on B 0 The RoIs features are extracted by RoI pooling on the multi-scale feature pyramid and then input to the head detection module D h , D h Output head classification score C h And the head rectangle position regression result B h ; The second stage is the visible area detection module D v , B h Convert to full body rectangular frame B f0 , based on B f0 After extracting full-body features through the second stage RoI pooling on the multi-scale feature pyramid, the visible region detection module D v Based on the whole body features, the visible area rectangle B can be successfully detected v and its confidence score C v ; In the third stage, B is still used f0 As RoI, full-body features are extracted through the third stage RoI pooling on the multi-scale feature pyramid. v To B f0 The extracted features are filtered and then input into the whole body detection module D f , and finally D f Output the rectangular box B of the whole body f and the confidence score C of the whole bodyf .

[0031] Furthermore, the optimization loss function of the overall network is

[0032] L=λ 1 L SRPN +λ 2 L vGA +λ 3 L FRCNN +λ 4 L HRCNN (17)

[0033] In the formula, λ 1-4 is the loss weight for each module, L SRPN is the overall loss function of SRPN, L VGA is the loss function of VGA, L FRCNN is the loss function of FRCNN, L HRCNN is the loss function of HRCNN.

[0034] (III) Beneficial effects

[0035] The present invention proposes a method for detecting occluded pedestrians based on head and visible area clues, using head, visible area and full body information to improve the detection performance of occluded pedestrians in complex road scenes, and proposes a sharp-response region proposal network (SRPN) to simultaneously detect the full body and head RoIs of pedestrians. For full body RoIs, a visible-guided attention (VGA) module is added before the original full body detection branch Fast RCNN to enhance the features of the visible area while suppressing the noise of the occluded area, and a parallel head cascade detection (Head-cascade R-CNN) branch is added to detect severely occluded pedestrians based on the step-by-step expansion of the perception domain of head RoIs. A large number of experimental results show that the method proposed in this patent can significantly improve the detection accuracy of occluded pedestrians. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the human body cascade detection network framework of the present invention;

[0037] Figure 2 A schematic diagram of a network structure recommended for sharpening feature response regions of the present invention;

[0038] Figure 3 This is a schematic diagram of marking pedestrians blocked by the present invention;

[0039] Figure 4A flowchart of filtering RoI features for the visible region guided attention module of the present invention;

[0040] Figure 5 This is a schematic diagram of the head cascade detection network of the present invention;

[0041] Figure 6 It is a schematic diagram of the regression transformation process from the head rectangular frame to the whole body rectangular frame of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0043] The purpose of this patent is to propose a human cascade (Human-Cascade-RCNN, HCRCNN) network, which uses head, visible area, and full body information to improve the performance of occluded pedestrian detection in complex road scenes. In actual road scenes, the probability of pedestrian heads being occluded is usually low, so they have better visibility. Compared with other body parts, the visibility of head information has better robustness to various types of occluded obstacles and crowd scenes. Therefore, we propose a sharp feature response region proposal network (Sharp-response Region Proposal Network, SRPN) to simultaneously detect pedestrian full-body and head RoIs. For full-body RoIs, the actual effective information is the human body information in the visible area, while the occluded part will only produce noise features that affect the resolution of the detector. Therefore, we add a visible-guided attention (VGA) module in front of the original full-body detection branch Fast RCNN to enhance the features of the visible area while suppressing the noise in the occluded area. In addition to the full-body RoIs detection branch, we added a parallel head cascade detection (Head-cascadeR-CNN) branch to detect severely occluded pedestrians based on the step-by-step expansion of the perception domain of head RoIs. A large number of experimental results show that the method proposed in this patent can significantly improve the detection accuracy of occluded pedestrians.

[0044] This patent studies the occlusion problem in pedestrian detection on complex roads and proposes a human cascade detection network (Human-Cascade R-CNN). The framework is as follows Figure 1 As shown in the figure, it mainly includes the backbone network (backbone), the sharp feature region recommendation network (SRPN), the detection network of the visible area guided attention module (VGA+Fast RCNN), and the head cascade detection network (HRCNN).

[0045] The backbone network in the figure is composed of a residual network ResNet 50 and a feature pyramid network FPN, which is the mainstream backbone network combination in object detection and pedestrian detection networks. The backbone network outputs a multi-scale feature pyramid.

[0046] First, the sharp feature response region recommendation network SRPN is used to output the pedestrian's full body RoIs and head RoIs.

[0047] The features obtained by pooling the whole body RoIs enter the whole body detection branch, that is, the Fast RCNN branch of the original Faster RCNN framework. At this time, the features contain the noise features of the occluded objects. The visible area guided attention module VGA estimates the probability map of the pedestrian visible area in the whole body rectangular frame based on this feature, and then optimizes the original features by dot multiplication, suppresses the noise features of the occluded objects, and enhances the features of the pedestrian visible area, and then inputs them to Fast RCNN for detection, and outputs the whole body detection results.

[0048] The pooled features obtained by pooling the head RoIs are input into the head cascade detection branch, which includes the head cascade detection network HRCNN. The network gradually obtains the visible area detection frame from the head detection frame in stages, and finally obtains the pedestrian full body detection frame in a progressive regression manner, thereby utilizing the head clues while avoiding a large number of false alarms caused by the head RoIs.

[0049] Both the whole body detection branch and the head cascade detection branch output whole body detection results. In order to avoid redundant results, Soft-NMS is used for post-processing, ultimately achieving the goal of significantly improving the detection accuracy of occluded pedestrians.

[0050] Step 1: Sharpening feature response region recommendation

[0051] Feature response adhesion can cause the detection frame to shift or erroneously suppress the detection frame when detecting crowded scenes. Therefore, this patent proposes to use a region recommendation network that sharpens feature responses for RoIs extraction. In the training mode based on anchor matching and external rectangular frame annotation, due to the overlap between pedestrian annotations and the incorrect marking of background and occluded areas introduced by the rectangular frame, the convolution feature responses inevitably diffuse and stick together. Therefore, in order to sharpen the feature response, it is necessary to change the anchor matching method and the rectangular frame training mode. In fact, when observing an object, the human eye can only observe a very small area in detail and will not preset a rectangular template of the object. Inspired by this, SRPN uses an anchor-free method to convert the pedestrian rectangular frame annotation into a center probability distribution map and a scale distribution map as supervision information. SRPN judges the center probability for each spatial coordinate on the convolution feature map and regresses the size estimate.

[0052] Figure 2 The detailed structure of the sharp feature response region proposal network is shown, which mainly includes two identical generation modules (ProposalModule) for outputting whole body and head RoIs. The generation module contains a 3×3×256 convolution layer, and three parallel 1×1×256 convolution layers output the target center probability map (center map), scale estimation map (scalemap), and center shift map (shift map) respectively. The results based on the above outputs (center score, scale, center shift) are converted into the target external rectangular box through RoIs conversion, and the final region of interest is obtained after NMS post-processing, and the whole body RoIs or head RoIs are output.

[0053] Different from CSP, which deconvolves multi-scale feature maps to a unified size, SRPN extracts RoIs on the feature pyramids of five sizes from P3 to P7 output by FPN. Similar to the different focal lengths of human eyes when observing objects of different sizes, the convolution kernels have the same size, so large-resolution feature maps are suitable for detecting small targets, while small-resolution feature maps are suitable for detecting large targets. This adapts to the scale diversity of pedestrians and makes the receptive field of the convolution kernel of the generation module match the target size.

[0054] SRPN uses a non-anchor matching region recommendation method, so it is necessary to convert the pedestrian rectangle annotation into the objective benchmark (ground truth) of the three estimated images output by SRPN and then conduct supervised training separately. The following introduces the objective benchmark generation method and loss function of the three estimated images.

[0055] The center probability map represents the probability that each feature map coordinate position is the target center. It belongs to the classification problem, so the average binary cross-entropy (BCE) loss function is used for training this branch. center The generation of is achieved by superimposing a two-dimensional Gaussian distribution map at each target center position (i, j). The specific formula is as follows:

[0056] G(i,j,x,y,σ)=exp(-(ix) 2 / (2σ 2 )-(jy) 2 / (2σ 2 )) (1)

[0057]

[0058] Where K is the number of positive samples in the image, (X k ,y k) is the center position of the bounding rectangle of the kth target, the variance σ of the Gaussian distribution map is related to the pyramid level 1 of the current feature, and (i, j) is the coordinate position of the feature map. Although SRPN does not use a large number of preset anchor point matching methods, the number of positive and negative samples in the image is still very different. In order to avoid the problem of extreme imbalance between positive and negative samples, the focal loss strategy is used to re-weight the samples and calculate the loss value of a batch. Therefore, the loss function of the target center probability map is:

[0059]

[0060]

[0061]

[0062] In the above formula, p ij ∈[0, 1] target center probability value, l is the feature pyramid level, W and H are the width and height of the image, γ and β are hyperparameters that control the focus loss weight. According to empirical experience, the parameters can be set to γ ​​= 2 and β = 4 respectively.

[0063] The coordinate position of each feature map in the scale estimation map is the target size value, which belongs to the regression problem. Therefore, the classic SmoothL1 loss function is used to train this branch. In the problem of pedestrian detection, the circumscribed rectangular box of the human body is generally used for annotation. However, due to the posture of the pedestrian, the pedestrian is sometimes not in the center of the rectangular box, which will cause great confusion for the positioning of the circumscribed rectangular box of the pedestrian detector. In response to the above problems, some scholars have proposed a human centerline annotation method, that is, draw a straight line between the top of the head and the two feet of the human body, and then generate a rectangular box annotation based on a fixed aspect ratio. This annotation form makes the rectangular box tightly surround the human torso, reduces the diversity of the rectangular box, and focuses more on the task goal of pedestrian detection itself. Based on the above characteristics, for the pedestrian detection task, the height of the pedestrian can be directly estimated when the center of the pedestrian is known, and the final rectangular box result of the pedestrian can be obtained based on the fixed aspect ratio. Therefore, the target scale estimation map GT scale Generated according to the following formula:

[0064]

[0065] In the formula, h k is the height of the kth target, r is the radius of the positive sample sampling range, and is empirically set to 2 to balance the precise positioning and scale estimation training tasks. The loss function of size estimation on the 1-level feature map is:

[0066]

[0067] In the formula, represents the target size estimated by the network, N is a normalization term, which indicates the number of feature map coordinate positions for size estimation, which is the same as the average value of the size estimation loss. [GT scale >0] is the indicator function, GT scale When it is greater than 0, it is 1, otherwise it is 0. The Smooth L1 formula is as follows:

[0068]

[0069] The function of the center offset map is to fine-tune the target center position. Since the feature map is smaller than the original image, the feature map coordinates have deviations as the target center, so a small range regression of the center position is required. This is a regression problem and is also trained using the Smooth L1 loss function. The actual center offset can be obtained by a simple calculation:

[0070]

[0071] Where, t ij is the deviation of the actual center position of the target in the coordinates of the 1-level feature map (i, j). The loss function of the center offset on the 1-level feature map is:

[0072]

[0073] In the formula, It is the offset predicted by SRPN at the position (i, j) of the feature map.

[0074] The loss function of SRPN at a feature pyramid level 1 is:

[0075]

[0076] In the formula, λ c , s and λ t are the weights of the three branches, which are set to 0.01, 1 and 0.1, respectively, referring to the settings in CSP. Finally, SRPN contains the losses of five feature pyramid levels P3 to P7, and SRPN outputs head and body RoIs at the same time, so the overall loss function of SRPN is:

[0077]

[0078] In the formula, and Represent the loss functions of the head and full body region recommendation branches of layer 1 respectively. However, most pedestrian detection datasets do not have a marked head rectangle, but due to the special annotation method of the CityPersons dataset, it basically ensures that the head is in the vertical center of the pedestrian annotation box, and the size and aspect ratio are related to the height of the pedestrian. Through statistical observation, the height of the head is about 20% of the height, so the head rectangle annotation can be calculated as follows:

[0079]

[0080] In the formula, (x f ,y f )The center coordinate of the whole body annotation box, (x h ,y h )Head center, h f is the pedestrian height, s h is the head size. The aspect ratio of pedestrians in CityPersons is 0.41. The aspect ratio of the head is simply set to 1.0. The relationship between the pedestrian full body annotation, visible area, and head annotation is as follows Figure 3 As shown, the whole body frame is green (rectangular frame), the visible area frame is blue (large square frame), the head frame is red (small square frame), the green dotted line (long dotted line) is the center line of the whole body frame, and the red dotted line (short dotted line) is the center line of the head frame.

[0081] Step 2: Visible areas guide attention to filter RoI features

[0082] The structure of the visible region guided attention module is as follows Figure 4 After the RoI pooling feature, a visible area guided attention template estimation branch is added before the FastRCNN detection module. After multiplying the attention template with the original RoI feature, it can filter out the interference features of occluded objects in the whole body RoIs.

[0083] The RoI pooling feature is denoted as F r ∈[C×H r ×W r ], where C is the number of channels, H r and W r is the feature resolution, usually set to 14×14. r After inputting VGA, the attention features are encoded through four C×3×3 convolutional layers and one 1×1 convolutional layer. Finally, the normalized visible area attention mask is obtained after the sigmoid function is applied pixel by pixel, which is recorded as A. V ∈[1×H r ×W r ]. By r With A VMultiply pixel by pixel to get the filtered feature F VGA (i,:,:)=Fr(i,:,:)·A V In the formula, i is the number of feature channels, and the attention template filters the features of each channel. VGA It is the feature after the attention template transformation, which reduces the interference of occluded objects. The detection network can improve the confidence score of occluded pedestrians.

[0084] In order to train VGA, segmentation and annotation of the visible area of ​​pedestrians are required. However, segmentation and annotation require a lot of manual operations. Most datasets for pedestrian detection tasks lack segmentation and annotation of visible areas, but they basically mark the circumscribed rectangular box of the visible area. And due to the particularity of the annotation of the CityPersons dataset, its visible area contains fewer noise areas, so it can be used as a rough segmentation mark for the VGA visible area attention template. The objective annotation of VGA attention can be defined as:

[0085]

[0086] In the formula, is the outer bounding rectangle of the visible area of ​​the kth pedestrian. If the pixel is within the rectangle, it is marked as 1, otherwise it is 0. VGA is essentially also to judge the visible area of ​​pedestrians, so it belongs to the pixel-by-pixel classification task, so the binary cross entropy loss BCE is used as the inspection function. The loss function of VGA is as follows:

[0087]

[0088] Step 3: Head Cascade Detection

[0089] The head cascade detection network HRCNN obtains the pedestrian full body detection results by step-by-step detection and regression from the head RoIs. Its structure is as follows: Figure 5As shown in the figure. The structure of HRCNN refers to the design of Cascade RCNN, which is essentially a multi-stage regression detection added to the Faster RCNN structure. Cascade regression is a process of multiple resampling. Each stage modifies the reference position of the RoI and re-extracts the RoI features for classification detection. Therefore, although the efficiency of cascade detection is lower than that of Faster RCNN, it can obtain a more accurate detection frame. From a numerical point of view, the IoU between its detection frame and the GT rectangular frame is higher. In the human visual system, when we search for individual pedestrians in a crowd, we usually locate their heads first and then estimate their whole bodies. However, due to the small size of the human head and the lack of sufficient discriminant information, the accuracy of pedestrian detection based on the head is not as good as that based on the pedestrian torso. Based on the above analysis, we adopt the idea of ​​cascade detection, starting from the head clue, and gradually expand the sampling range in stages to support the previous detection results. It detects the sampling features of the head, visible area, and full body rectangular frame respectively, and obtains the final detection results based on the head clue.

[0090] like Figure 5 ,HRCNN calculation process is divided into three stages: ,the first stage, B 0 are the head RoIs generated by SRPN, based on B 0 The RoIs features are extracted by RoI pooling on the multi-scale feature pyramid and then input to the head detection module D h . D h Output head classification score C h And the head rectangle position regression result B h The second stage is the visible area detection module D v , but it is obvious that based on the head rectangle B h Regression pedestrian visible area rectangle B v Therefore, B h Convert to full body rectangular frame B f0 , based on B f0 After extracting full-body features through the second stage RoI pooling on the multi-scale feature pyramid, the visible region detection module D v Based on the whole body features, the visible area rectangle Bv and its confidence score C can be successfully detected v In the third stage, B is still used f0 As RoI, full-body features are extracted through the third stage RoI pooling on the multi-scale feature pyramid. v The function is similar to VGA. f0 The extracted features are filtered and then input into the whole body detection module D f . Finally D f Output the rectangular box B of the whole body f and the confidence score C of the whole body fThe regression transformation process from the head rectangle to the whole body rectangle is as follows: Figure 6 shown.

[0091] In order to train the head cascade detection network, the head G is required h , visible area G v , whole body G f The CityPersons dataset provides a visible area G v and whole body G f The annotation of the head G h It can be automatically calculated according to formula 12. Each detection module in the cascade detector contains a classifier and a regressor, so the loss function of HRCNN can be expressed as:

[0092] L HRCNN =L h +L v +L f (15)

[0093] L i =L cls (C i , Y i )+[Y i >0]L reg (B i , G i ), i∈h,v,f (16)

[0094] Where, L i is the loss function of the detector at the i-th stage, L cls is the classification cross entropy loss function, L reg is the regression SmoothL1 loss function, Y i is the RoI at the IoU threshold u i = 0.7, [Y i >0] is the indicator function, when Y i >0, it is 1, otherwise it is 0.

[0095] Step 4: Multi-task loss function training

[0096] Based on the above-mentioned multiple occluded pedestrian detection modules, multi-task simultaneous training is adopted during training, and the optimization loss function of the overall network can be defined as

[0097] L=λ 1 L SRPN +λ 2 L VGA +λ 3 L FRCNN +λ 4 L HRCNN (17)

[0098] In the formula, λ x The loss weight for each module is simply set to 1 during actual training.

[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting occluded pedestrians based on head and visible area clues, It is characterized in that The method comprises the following steps: The backbone network uses the residual network ResNet50 and the feature pyramid network FPN, and the backbone network outputs a multi-scale feature pyramid; The multi-scale feature pyramid is sharpened through the feature response region proposal network SRPN to output the pedestrian's whole body RoIs and head RoIs; The features obtained by pooling the whole body RoIs enter the whole body detection branch. The visible area guided attention module VGA estimates the pedestrian visible area probability map in the whole body rectangular frame based on this feature, and then optimizes the original features by dot multiplication, suppresses the noise features of occluded objects and enhances the features of the pedestrian visible area. After that, it is input to Fast RCNN for detection and outputs the whole body detection results. The pooled features obtained by pooling the head RoIs are input to the head cascade detection branch, which includes the head cascade detection network HRCNN. The network gradually obtains the visible area detection frame from the head detection frame in stages, and finally obtains the pedestrian full body detection frame; The whole body detection results output by the whole body detection branch and the head cascade detection branch are post-processed using Soft-NMS; in, The sharp feature response region recommendation network SRPN includes two identical generation modules for outputting whole body and head RoIs; the generation module includes a 3×3×256 convolution layer, and three parallel 1×1×256 convolution layers respectively output a target center probability map, a scale estimation map, and a center offset map; the center score, scale, and center offset based on the above output are converted into a target external rectangular frame through RoIs conversion, and the final region of interest is obtained after NMS post-processing, and the whole body RoIs or head RoIs are output; The SRPN extracts RoIs on the feature pyramids of five sizes from P3 to P7 output by FPN respectively; The visible area guided attention module VGA is implemented as follows: RoI pooling feature is denoted as F r ∈[C×H r ×W r ], where C is the number of channels, H r and W r is the feature resolution; F r After inputting VGA, the attention features are encoded through four C×3×3 convolutional layers and one 1×1 convolutional layer. Finally, the normalized visible area attention template is obtained after the pixel-by-pixel Sigmoid function, which is recorded as A. V ∈[1×H r ×W r ]; by r With A V Multiply pixel by pixel to get the filtered feature F VGA (i,:,:)=F r (i,:,:)·A V , the filtered features are input to Fast RCNN for detection; The calculation process of the head cascade detection network HRCNN is divided into three stages: the first stage, B 0 are the head RoIs generated by SRPN, based on B 0 The RoIs features are extracted by RoI pooling on the multi-scale feature pyramid and then input to the head detection module D h , D h Output head classification score C h And the head rectangle position regression result B h ; The second stage is the visible area detection module D v , B h Convert to full body rectangular frame B f0 , based on B f0 After extracting full-body features through the second stage RoI pooling on the multi-scale feature pyramid, the visible region detection module D v Based on the whole body features, the visible area rectangle B can be successfully detected v and its confidence score C v ; In the third stage, B is still used f0 As RoI, full-body features are extracted through the third stage RoI pooling on the multi-scale feature pyramid. v To B f0 The extracted features are filtered and then input into the whole body detection module D f , and finally D f Output the rectangular box B of the whole body f and the confidence score C of the whole body f .

2. The method for detecting occluded pedestrians based on head and visible area clues as claimed in claim 1, It is characterized in that The center probability map branch is trained using the average binary cross-entropy (BCE) loss function; the target center probability map GT center The generation of is achieved by superimposing a two-dimensional Gaussian distribution map at each target center position i, j). The specific formula is as follows: G(i,j,x,y,σ)=exp(-(i-x) 2 / (2σ 2 )-(j-y) 2 / (2σ 2 )) (1) Where K is the number of positive samples in the image, (x k ,y k ) is the center position of the circumscribed rectangular box of the kth target, the value of the variance σ of the Gaussian distribution map is related to the pyramid level of the current feature, and (i, j) is the coordinate position of the feature map.

3. The method for detecting occluded pedestrians based on head and visible area clues as claimed in claim 2, It is characterized in that Target scale estimation graph GT scale Generated according to the following formula: In the formula, h k is the height of the kth target, and r is the radius of the positive sample sampling range.

4. The method for detecting occluded pedestrians based on head and visible area clues as claimed in claim 3, It is characterized in that The center offset of the center offset map branch is calculated according to the following formula: Where, t ij It is the deviation of the actual center position of the target in the coordinates of the feature map (i, j) of layer l.

5. The method for detecting occluded pedestrians based on head and visible area clues as claimed in claim 4, It is characterized in that The head rectangle can be annotated by the following calculation: In the formula, (x f ,y f )The center coordinate of the whole body annotation box, (x h ,y h )Head center, h f is the pedestrian height, s h It is the head size.

6. The method for detecting occluded pedestrians based on head and visible area clues as claimed in claim 1, It is characterized in that The optimization loss function of the overall network is L=λ 1 L SRPN +λ 2 L VGA +λ 3 L FRCNN +λ 4 L HRCNN (17) In the formula, λ 1-4 is the loss weight for each module, L SRPN is the overall loss function of SRPN, L VGA is the loss function of VGA, L FRCNN is the loss function of FRCNN, L HRCNN is the loss function of HRCNN.

Citation Information

Cited By

  • Dangerous pedestrian perception and cognition method and system in parking lot environment

    CN114852055A