A pedestrian detection method and device that resists occlusion overlap and scale change

Abstract

The present invention relates to the field of computer vision technology, and specifically provides a pedestrian detection method and device that is resistant to occlusion, overlap, and scale changes. In the feature extraction link, a cyclic multi-scale feature extraction network integrating an attention mechanism is constructed to improve the ability to obtain image feature information and the ability to express feature information, and to improve the detection sensitivity of the pedestrian detector to multi-scale pedestrian targets; in the core part of the model, a head and overall dual detection branch is designed for joint detection, and head detection is used to assist pedestrian detection to improve the detection capability of densely occluded pedestrian targets. In the post-processing link of the model, a suppression discrimination mechanism is constructed for the overall detection frame of the pedestrian, and a pedestrian head-body matching strategy is constructed to match and bind the pedestrian head detection frame with the overall detection frame. A secondary matching check is performed on the head detection frame that fails to match successfully, which is used to restore the overall detection frame that is incorrectly suppressed and eliminate the head detection frame that is misdetected, so as to further improve the detection accuracy of pedestrian targets.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically provides a pedestrian detection method and device that are resistant to occlusion, overlap and scale changes. Background Art

[0002] In the field of computer vision, pedestrian detection technology is one of the most widely used and popular research directions. The pedestrian detection task mainly refers to accurately detecting and locating pedestrian target instances in images, videos and video stream data through computer vision related technologies. This task is essentially a classification and regression process. In real life, pedestrian detection plays an important role in the fields of autonomous driving, intelligent monitoring, intelligent robots and human-computer interaction and has high application value. Among them, in the fields of autonomous driving and assisted safe driving, pedestrian detection technology can be used to detect in real time whether there are pedestrians suddenly breaking into the front of the vehicle, so as to make timely adjustments according to the actual situation to ensure pedestrian safety and driving safety. Especially in congested sections and crowded scenes, the auxiliary role of pedestrian detection technology in autonomous driving safety is more obvious. In the field of intelligent monitoring, most public places today use cameras to monitor the entire scene and count crowd flow data in real time. Especially during the epidemic, most large public places will strictly control the density of pedestrian flow. The use of pedestrian detection technology can accurately count the pedestrian flow in the venue in real time, and analyze and predict these data, which will help managers take corresponding adjustment measures. In the field of intelligent robots, sensors such as cameras transmit corresponding environmental scene signals to intelligent robots. Pedestrian detection algorithms, as an important thinking perception network in the brain of intelligent robots, can help intelligent robots quickly and accurately perceive pedestrian targets and make corresponding decisions in time to make adjustments. In the field of human-computer interaction, smart food delivery vehicles and smart express delivery vehicles in campuses and restaurants have integrated multiple functions such as pedestrian detection algorithms, and through interaction with pedestrians, they achieve the purpose of using artificial intelligence technology to better serve people's daily lives.

[0003] Pedestrian detection algorithms based on deep learning can be divided into single-stage pedestrian detection algorithms and two-stage pedestrian detection algorithms according to the detection ideas. Among them, the single-stage pedestrian detection algorithm is mainly represented by the YOLO series network model, and the two-stage pedestrian detection algorithm is represented by the RCNN series network model. However, both types of algorithms lack research and design for specific detection scenarios, and have weak detection capabilities for small-scale pedestrian targets at farther viewing angles and pedestrian targets with severe occlusion and overlap, which is mainly manifested in low detection accuracy and high missed detection rate. Therefore, it is very necessary to design a pedestrian detection method that is resistant to occlusion, overlap and scale changes.

[0004] In response to the above needs, there are currently many relevant solutions at home and abroad.

[0005] Chinese patent publication number CN111767882A, published on October 13, 2020, proposes a multimodal pedestrian detection method based on an improved YOLO model. By integrating the CBAM attention mechanism and optimizing the loss function, the pedestrian detection effect is improved. However, there are also some problems such as: (1) The sensitivity to multi-scale pedestrian targets in the feature extraction link is not high enough, especially the feature information learning ability of small-scale pedestrians at a farther viewing angle is weak; (2) There is still room for improvement in the phenomenon of missed detection of pedestrian targets with serious occlusion and overlap. Chinese patent application publication number CN115082855A, published on September 20, 2022, a pedestrian occlusion detection method based on an improved YOLOX algorithm is similar to a multimodal pedestrian detection method based on an improved YOLO model with publication number CN111767882A, published on October 13, 2020, and also has the same problems. Chinese patent publication number CN113989939A, published on January 28, 2022, proposes a method for detecting small-scale pedestrian targets, but the actual detection scene inevitably has dense and crowded pedestrians, so the method has weak detection capabilities for occluded pedestrian targets and has certain limitations in practical applications. Chinese patent application publication number CN114882527A, published on August 9, 2022, proposes a pedestrian detection algorithm and system based on dynamic group convolution, which mainly uses group convolution to achieve pedestrian detection, but lacks consideration of specific scenes for pedestrian detection, and has certain limitations in execution efficiency in complex crowd-dense scenes.

[0006] For existing pedestrian detection technologies, most solutions still rely on classic target detection algorithms to detect pedestrian targets in scenes. Based on the detection ideas, they can be divided into single-stage detection algorithms represented by the YOLO series network model and two-stage detection algorithms represented by the Faster RCNN network model. However, most classic target detection algorithms lack specific considerations for detection scenarios, especially in complex crowd-dense scenes. The robustness of these mainstream detection algorithms is affected, mainly including the following two points:

[0007] (1) The different scales of pedestrian targets affect the overall performance of the detector. Since most of the current pedestrian detection data sets are obtained based on camera shooting and calibration, and the camera has the rule of "large near and small far" when shooting, the overall scale of pedestrian targets at a closer perspective is larger, and the overall scale of pedestrian targets at a farther perspective is smaller. Since the resolution of small-scale pedestrian targets is relatively low, the algorithm is prone to problems such as limited learned feature information or weak feature expression ability during feature extraction, making it difficult to have high sensitivity for pedestrian targets of different scales, which can easily lead to missed detection or false detection.

[0008] (2) Pedestrian targets are seriously occluded, which affects the overall performance of the detector. In pedestrian detection tasks in complex and crowded scenes, pedestrian targets are often occluded to a certain extent. By analyzing the images in the pedestrian detection dataset, it can be seen that the pedestrian occlusion problem mainly includes two situations: intra-class occlusion and inter-class occlusion. Among them, intra-class occlusion refers to the mutual occlusion between pedestrian targets. Inter-class occlusion refers to the interference of pedestrians by background information, which mainly includes buildings, trees, vehicles, items carried by pedestrians themselves, and items carried by other nearby pedestrians. Intra-class occlusion and inter-class occlusion lead to a decrease in the proportion of the visible area of ​​the pedestrian's whole body, which will cause difficulties in the feature extraction link of the algorithm, and the information required for the algorithm detection module to infer is also reduced. In addition, it will affect the accuracy of pedestrian target positioning, thereby affecting the overall performance of the pedestrian detection algorithm. Summary of the invention

[0009] In order to solve the above problems, the present invention provides a pedestrian detection method and device that are resistant to occlusion overlap and scale change, which can improve the detection accuracy of small-scale pedestrian targets and severely occluded pedestrian targets at a farther viewing angle, and reduce their missed detection rate.

[0010] In a first aspect, the present invention provides a pedestrian detection method that is resistant to occlusion overlap and scale change, comprising:

[0011] Build the Faster RCNN network model;

[0012] The backbone network of the Faster RCNN network model is integrated with the improved feature extraction network, and the image to be detected is input into the Faster RCNN network model for feature extraction to obtain an extracted feature map;

[0013] Generate a set of candidate frames for pedestrian heads through a region proposal network, perform statistical calculations based on the combination of the calibrated pedestrian head regions in the data set and the overall regions corresponding to the pedestrian head regions, obtain the head-to-body ratio, and generate a set of overall candidate frames for pedestrians based on the head-to-body ratio;

[0014] Constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result in the image to be detected through the pedestrian head detection branch module and the pedestrian overall detection branch module;

[0015] Performing preliminary post-processing on the pedestrian head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection results to obtain preliminary post-processing detection results, and filtering out redundant detection results;

[0016] Constructing a pedestrian head-body matching module, and inputting the preliminary post-processing detection result into the pedestrian head-body matching module, matching and binding the head detection frame and the overall detection frame;

[0017] A secondary post-processing module based on head suppression and full-body recall is constructed, and the secondary post-processing module is used to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result, restore the pedestrian overall frame that was erroneously suppressed and eliminate the pedestrian head frame that was erroneously detected, so as to obtain the final pedestrian detection result.

[0018] As a preferred solution, the backbone network of the Faster RCNN network model is fused with an improved feature extraction network, and the image to be detected is input into the Faster RCNN network model for feature extraction to obtain an extracted feature map, including:

[0019] Use the ResNet50 network as the backbone network of the Faster RCNN network model;

[0020] Fusing the improved feature extraction network with the Faster RCNN network model;

[0021] Constructing a feature reconstruction module, wherein the feature reconstruction module includes a feature scale scaling module and a feature transformation module;

[0022] Construct a hybrid attention enhancement module to increase the weight of small-scale pedestrian target features in the training process by focusing on them in both channel and spatial dimensions, while suppressing background features;

[0023] The hybrid attention enhancement module is composed of a channel attention module and a spatial attention module connected in series. The channel attention module is used to extract channel attention information from the input feature map, and the spatial attention module is used to enhance spatial attention and learn spatial weight information. Finally, a weighted multiplication operation is performed to obtain the final feature map.

[0024] As a preferred solution, the improved feature extraction network is integrated with the Faster RCNN network model, including:

[0025] Using the backbone network ResNet50 to learn the feature information of the image to be detected;

[0026] The feature information obtained by the backbone network ResNet50 is subjected to feature fusion.

[0027] As a preferred solution, feature fusion is performed on the feature information obtained by the backbone network ResNet50, including:

[0028] The feature fusion is performed through the feature pyramid FPN structure. The specific process is expressed by formula (1) and formula (2):

[0029] (1)

[0030] (2)

[0031] in, Represents the backbone network The convolution operation of the stage, Represents the top-down feature pyramid network Stage feature fusion operation, For the The features learned by the layer network, The feature pyramid The characteristics of the stage output, is the number of feature layers.

[0032] Combining the idea of ​​cyclic recursion, a feedback splicing channel from back to front is added on the basis of the feature pyramid FPN structure to construct a cyclic recursive feature pyramid. The cyclic recursive feature pyramid is used to feedback and splice the feature information output by the feature pyramid FPN structure to the backbone network, so that the backbone network can perform secondary purification processing on the features. The specific process is expressed by formula (3) and formula (4):

[0033] (3)

[0034] (4)

[0035] in, It represents the feature transformation operation before splicing the learned feature feedback to the backbone network, which is used to realize the cascade connection of the two stages, repeat the feature extraction and fusion operation of the first stage, and obtain the output features of the second stage. , the features of the two stages output in the recursive feature pyramid and It can be expressed by formula (5) to formula (8):

[0036] (5)

[0037] (6)

[0038] (7)

[0039] (8)

[0040] The feature transformation operation before splicing the learned feature feedback to the backbone network The specific implementation process is:

[0041] After transformation, the features conduct Convolution operation;

[0042] The features output by the first residual block in each stage of the backbone network are fused to complete the feedback splicing operation; and for the feature transformation processing link before the feedback is spliced ​​to the backbone network, the spatial pyramid pooling module ASPP integrated with the dilated convolution is used to achieve the cascade connection of the two stages, wherein the spatial pyramid pooling module ASPP includes four parallel branches, wherein the first three branches all adopt a convolution layer structure, and the sizes of the convolution kernels are 1×1, 3×3 and 3×3, respectively, and a successively increasing dilation rate is set, and a ReLU layer is connected at the back to increase the receptive field range. The fourth branch is based on the global average pooling operation, and is connected to a 1×1 ordinary convolution layer and a ReLU layer at the back to evenly extract effective information and reduce local information loss; the features output by the four branches are connected along the channel dimension to generate a feature with the same scale as the input feature of the second stage to complete the feature transformation processing operation;

[0043] The backbone network performs secondary learning to obtain the image features of the image to be detected, and converts the features output from the two stages into and The specific process is as follows: construct an adaptive feature fusion method and assign weights to the features output from the two stages , weight It is calculated by the convolutional layer and Sigmoid function after the output feature of the second stage, according to the weight The two-stage features are weightedly fused, which is specifically expressed by formula (9):

[0044] (9)

[0045] in, and are the weights assigned to the output features of the two stages, and the weights and The relationship between , It is the final output feature of the feature extraction network.

[0046] As a preferred solution, the feature reconstruction module is constructed, and the feature reconstruction module includes a feature scale scaling module and a feature transformation module, including:

[0047] The feature scale scaling module generates features of different scales through upsampling or downsampling operations;

[0048] The feature transformation module is used to combine all features of the same scale in series to form new features. The specific method is as follows: for the intermediate-level features, the feature scale scaling module is used to generate features of the current-level feature scale, features of the previous-level feature scale, and features of the next-level feature scale for the intermediate-level features; for the deepest-level features, features of the current-level feature scale and features of the next-level feature scale are generated for the deepest-level features; for the shallowest-level features, features of the current-level feature scale and features of the previous-level feature scale are generated for the shallowest-level features, and features of the same scale are connected in series along the channel dimension.

[0049] As a preferred solution, the hybrid attention enhancement module is constructed to increase the weight of small-scale pedestrian target features in the training process by focusing on small-scale pedestrian target features in two dimensions, channel and space, while suppressing background features, including:

[0050] In the channel attention module, the input features are first subjected to maximum pooling and average pooling operations to obtain a compressed one-dimensional feature vector and , and for and Perform element-by-element addition operations to enable the one-dimensional feature vector to obtain the global receptive field of space, and perform two convolution operations to obtain the scales of and The one-dimensional feature vector of is the compression coefficient, which is used to learn the channel weight information, and the channel weight information is weighted multiplied with the input feature to obtain the channel attention feature map that integrates the channel attention;

[0051] In the spatial attention module, the channel attention feature map is subjected to average pooling and maximum pooling operations to obtain a compressed two-dimensional feature vector and , the two-dimensional feature vector is spliced ​​along the channel dimension so that the two-dimensional feature vector can obtain the channel global receptive field, and a convolution operation is performed to obtain a scale of The two-dimensional feature vector is obtained and the spatial weight information is obtained, and the spatial weight information is weighted multiplied with the channel attention feature map to obtain the final feature after processing.

[0052] As a preferred solution, the preliminary post-processing of the head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection result to obtain the preliminary post-processing detection result, and filtering out redundant detection results, includes:

[0053] Perform non-maximum suppression processing on the head detection results based on the non-maximum suppression algorithm;

[0054] A non-maximum suppression algorithm based on target occlusion overlap rate compensation is constructed to perform non-maximum suppression on the overall pedestrian detection results. The occlusion overlap rate of pedestrian targets corresponding to the area of ​​interest is calculated. , so that the actual suppression threshold is adaptively adjusted as the degree of occlusion overlap changes. The occlusion overlap rate corresponding to the target box The calculation method can be expressed as:

[0055] (12)

[0056] (13)

[0057] in, Indicates The overlapping area of ​​a target frame is the maximum value of the overlapping area between the target frame and all other target frames. Indicates The area of ​​the target box itself;

[0058] The calculated The occlusion overlap rate corresponding to the target box The preset threshold in the NMS algorithm Compare them and take the larger value between the two as the decision threshold for non-maximum suppression processing.

[0059] As a preferred solution, the step of constructing a pedestrian head-body matching module, inputting the preliminary post-processing detection result into the pedestrian head-body matching module, and matching and binding the head detection frame and the overall detection frame includes:

[0060] The pedestrian head-body matching module includes three constraints, which are specifically expressed as follows:

[0061] Calculate the pedestrian head box Overall frame with pedestrians The overlap ratio between , the specific method is:

[0062] (14)

[0063] in, represents the overlapping area between the head frame and the pedestrian's overall frame, Represents the pedestrian head frame If the overlap ratio Below the preset threshold , then it is determined that the head and the whole body cannot be matched. If the overlap ratio Above the preset threshold , then proceed to the following steps;

[0064] Constructing a virtual scaled overall frame of pedestrians , and calculate the intersection-and-combination ratio between the pedestrian virtual zoomed overall frame and the pedestrian overall frame, and set the pedestrian head frame The coordinate information is , the width and height of the pedestrian head frame are respectively and To represent, the virtual zoom overall frame corresponding to the pedestrian head frame Coordinate information of It can be expressed as:

[0065] (15)

[0066] (16)

[0067] (17)

[0068] (18)

[0069] Calculate the virtual scaled overall frame With the overall frame If the intersection ratio between Below the preset threshold , it is determined that the head and the whole body cannot be matched; if it is higher than the preset threshold , then continue to judge the L1 distance between the two frame scales , which can be expressed as:

[0070] (19)

[0071] in, and Respectively represent the width and height of the pedestrian's overall frame, and Respectively represent the width and height of the virtual zoom overall frame. If The value is greater than the preset threshold , indicating that there is a large difference in scale between the head frame and the overall frame, so they cannot be matched, otherwise proceed to the next step;

[0072] Calculate the discrimination score of the relationship between the pedestrian head and the whole body. By constructing a relationship discrimination module consisting of three stacked fully connected layers, and extracting the information of each head corresponding to the whole matching pair for the relationship discrimination module, calculate the relationship matching score between the head and the whole body. If the maximum value of the relationship matching score is lower than the preset threshold Or when the relationship matching score set is an empty set, it means that the head frame cannot be matched to the corresponding overall frame.

[0073] As a preferred solution, the secondary post-processing module based on head suppression and full body recall is constructed, and the secondary post-processing module is used to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result to obtain the final pedestrian detection result, including:

[0074] The header frame set that failed to match successfully The elements in the image are matched again with the overall frame set that has not been processed with non-maximum suppression. After the first two matching constraints are met at the same time, the score calculated in the relationship judgment link is greater than a higher preset threshold. When , it is considered a successful match;

[0075] If the head and the whole are successfully paired, the pedestrian overall detection result previously suppressed by the non-maximum value is restored. If the maximum value of the relationship matching score is still lower than the preset threshold Or when the relationship matching score set is an empty set, the head detection result is determined to be a false detection, and the final pedestrian detection result is obtained.

[0076] In a second aspect, the present invention provides a pedestrian detection device that is resistant to occlusion, overlap and scale change, comprising:

[0077] Construction unit, used to build Faster RCNN network model;

[0078] A fusion unit, used to fuse the backbone network of the Faster RCNN network model with the improved feature extraction network, and input the image to be detected into the Faster RCNN network model for feature extraction to obtain an extracted feature map;

[0079] A construction unit is used to generate a set of candidate frames for pedestrian heads through a region proposal network, and to perform statistical calculations based on the combination of the calibrated pedestrian head regions in the data set and their corresponding overall regions to obtain the head-to-body ratio of the pedestrian target, and to generate a set of overall candidate frames for the pedestrian based on the ratio;

[0080] A preliminary detection unit, constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result of the image to be detected through the pedestrian head detection branch module and the pedestrian overall detection branch module;

[0081] A post-processing detection unit, used for performing preliminary post-processing on the head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection result to obtain a preliminary post-processing detection result, and filtering out redundant detection results;

[0082] An input unit, used to construct a pedestrian head-body matching module, and input the preliminary post-processing detection result into the pedestrian head-body matching module, and match and bind the head detection frame and the overall detection frame;

[0083] The secondary verification unit is used to construct a secondary post-processing module based on head suppression and full-body recall, and use the secondary post-processing module to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result to obtain the final pedestrian detection result.

[0084] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0085] The pedestrian detection method and device for resisting occlusion, overlap and scale change provided in the embodiment of the present invention enhances the ability of the feature extraction network to obtain image feature information and the expression ability of feature information by enhancing feature fusion and introducing the strategy of attention enhancement module in the feature extraction link, and makes the semantic information and detail texture information contained in the feature information more balanced, and uses the feature reconstruction module to make the semantic information and detail texture information covered in the acquired feature information more balanced, thereby improving the algorithm's detection ability for multi-scale pedestrian targets. In the core part of the model, the head and overall dual detection branches are designed for joint detection, and then the head detection is fully utilized to assist pedestrian detection to improve the detection ability of densely occluded pedestrian targets. In the post-processing link of the model, a more reasonable suppression discrimination mechanism is constructed for the overall detection frame of pedestrians with severe occlusion and overlap, and then the constructed pedestrian head-body matching strategy is used to match and bind the pedestrian head detection frame and the overall detection frame, and the head detection frame that fails to match successfully is checked for secondary matching, which is used to restore the wrongly suppressed pedestrian overall detection frame and eliminate the falsely detected head detection frame, so as to further improve the detection accuracy of pedestrian targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 is a flow chart of a pedestrian detection method that is resistant to occlusion overlap and scale change in an embodiment of the present invention;

[0087] Figure 2 It is a structural schematic diagram of a feature extraction network in a pedestrian detection method that is resistant to occlusion overlap and scale change provided in an embodiment of the present invention;

[0088] Figure 3 It is a schematic diagram of the structure of a ResNet50 network in a pedestrian detection method that is resistant to occlusion overlap and scale change provided in an embodiment of the present invention;

[0089] Figure 4 It is a structural schematic diagram of a cyclic recursive feature pyramid module in a pedestrian detection method against occlusion overlap and scale change provided in an embodiment of the present invention;

[0090] Figure 5 It is a schematic diagram of the structure of a spatial pyramid pooling module ASPP that integrates dilated convolution in a pedestrian detection method that resists occlusion overlap and scale change, provided in an embodiment of the present invention;

[0091] Figure 6 is a structural schematic diagram of a hybrid attention enhancement module of a pedestrian detection method that is resistant to occlusion overlap and scale change in an embodiment of the present invention;

[0092] Figure 7 It is a schematic diagram of the comprehensive performance comparison between a pedestrian detection method resistant to occlusion, overlap and scale change provided in an embodiment of the present invention and some mainstream detection algorithms on the CrowdHuman dataset;

[0093] Figure 8 It is a schematic diagram of the comprehensive performance comparison between a pedestrian detection method resistant to occlusion, overlap and scale change provided in an embodiment of the present invention and some mainstream detection algorithms on the CityPersons dataset;

[0094] Fig. 9 It is a structural block diagram of a pedestrian detection device that is resistant to occlusion, overlap and scale changes in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0096] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0097] Combination Figure 1 As shown, the present invention provides a pedestrian detection method that is resistant to occlusion overlap and scale change, comprising:

[0098] S101, build Faster RCNN network model;

[0099] S102, integrating the backbone network of the Faster RCNN network model with the improved feature extraction network, and inputting the image to be detected into the Faster RCNN network model for feature extraction to obtain an extracted feature map;

[0100] S103, generating a set of candidate frames for pedestrian heads through a region proposal network, and performing statistical calculations based on a combination of a pedestrian head region calibrated in a data set and an overall region corresponding to the pedestrian head region to obtain a head-to-body ratio relationship of a pedestrian target, and generating a set of overall candidate frames for pedestrians based on the head-to-body ratio relationship of a pedestrian target.

[0101] S104, constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result in the image to be detected through the pedestrian head detection branch module and the pedestrian overall detection branch module.

[0102] S105 , performing preliminary post-processing on the pedestrian head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection results to obtain preliminary post-processing detection results, and filtering out redundant detection results.

[0103] S106: construct a pedestrian head-body matching module, and input the preliminary post-processing detection result into the pedestrian head-body matching module to match and bind the head detection frame and the overall detection frame.

[0104] S107. Construct a secondary post-processing module based on head suppression and full-body recall, and use the secondary post-processing module to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result, restore the overall detection frame that was incorrectly suppressed and eliminate the falsely detected head detection frame, so as to obtain the final pedestrian detection result.

[0105] Combination Figure 2 and Figure 3 As shown, in S102, the backbone network of the Faster RCNN network model is fused with the improved feature extraction network, and the image to be detected is input into the Faster RCNN network model for feature extraction to obtain an extracted feature map, including:

[0106] Use the ResNet50 network as the backbone network of the Faster RCNN network model;

[0107] Fusing the improved feature extraction network with the Faster RCNN network model;

[0108] Constructing a feature reconstruction module, wherein the feature reconstruction module includes a feature scale scaling module and a feature transformation module;

[0109] Construct a hybrid attention enhancement module to increase the weight of small-scale pedestrian target features in the training process by focusing on them in both channel and spatial dimensions, while suppressing background features;

[0110] The hybrid attention enhancement module is composed of a channel attention module and a spatial attention module connected in series. The channel attention module is used to extract channel attention information from the input feature map, and the spatial attention module is used to enhance spatial attention and learn spatial weight information. Finally, a weighted multiplication operation is performed to obtain the final feature map.

[0111] Furthermore, the improved feature extraction network is integrated with the Faster RCNN network model, including:

[0112] Using the backbone network ResNet50 to learn the feature information of the image to be detected;

[0113] The feature information obtained by the backbone network ResNet50 is subjected to feature fusion.

[0114] Combination Figure 4 , further, using the feature information obtained from the backbone network ResNet50 to perform feature fusion, including:

[0115] The feature fusion is performed through the feature pyramid FPN structure. The specific process is expressed by formula (1) and formula (2):

[0116] (1)

[0117] (2)

[0118] in, Represents the backbone network The convolution operation of the stage, Represents the top-down feature pyramid network Stage feature fusion operation, For the The features learned by the layer network, The feature pyramid The characteristics of the stage output, is the number of feature layers.

[0119] Combining the idea of ​​cyclic recursion, a feedback splicing channel from back to front is added on the basis of the feature pyramid FPN structure to construct a cyclic recursive feature pyramid. The cyclic recursive feature pyramid is used to feedback and splice the feature information output by the feature pyramid FPN structure to the backbone network, so that the backbone network can perform secondary purification processing on the features. The specific process is expressed by formula (3) and formula (4):

[0120] (3)

[0121] (4)

[0122] in, It represents the feature transformation operation before splicing the learned feature feedback to the backbone network, which is used to realize the cascade connection of the two stages, repeat the feature extraction and fusion operation of the first stage, and obtain the output features of the second stage. , the features of the two stages output in the recursive feature pyramid and It can be expressed by formula (5) to formula (8):

[0123] (5)

[0124] (6)

[0125] (7)

[0126] (8)

[0127] Combination Figure 5 As shown, further, the learned feature feedback is spliced ​​to the feature transformation operation before the backbone network The specific implementation process is:

[0128] After transformation, the features conduct Convolution operation;

[0129] Fuse with the features output by the first residual block in each stage of the backbone network to complete the feedback splicing operation. As for the feature transformation processing link before the feedback is spliced ​​to the backbone network, the spatial pyramid pooling module ASPP integrated with the hole convolution is adopted to realize the cascade connection of the two stages, wherein the spatial pyramid pooling module ASPP includes four parallel branches, wherein the first three branches all adopt the convolution layer structure, the sizes of the convolution kernels are 1×1, 3×3 and 3×3 respectively, and the hole rates are set to increase successively, and the ReLU layer is connected at the back to increase the receptive field range. The fourth branch is based on the global average pooling operation, and is connected to a 1×1 ordinary convolution layer and a ReLU layer at the back to evenly extract effective information and reduce local information loss; the features output by the four branches are connected along the channel dimension to generate a feature with the same scale as the input feature of the second stage to complete the feature transformation processing operation;

[0130] The backbone network performs secondary learning to obtain the image features of the image to be detected, and converts the features output from the two stages into and The specific process is as follows: construct an adaptive feature fusion method and assign weights to the features output from the two stages , weight It is calculated by the convolutional layer and Sigmoid function after the output feature of the second stage, according to the weight The two-stage features are weightedly fused, which is specifically expressed by formula (9):

[0131] (9)

[0132] in, and are the weights assigned to the output features of the two stages, and the weights and The relationship between , It is the final output feature of the feature extraction network.

[0133] Furthermore, the feature reconstruction module is constructed, and the feature reconstruction module includes a feature scale scaling module and a feature transformation module, including:

[0134] The feature scale scaling module generates features of different scales through upsampling or downsampling operations;

[0135] The feature transformation module is used to combine all features of the same scale in series to form new features. Specifically, for the intermediate-level features, the feature scale scaling module is used to generate features of the feature scale of the current layer, features of the feature scale of the previous layer of the current layer, and features of the feature scale of the next layer of the current layer; for the deepest layer features, features of the feature scale of the current layer and features of the feature scale of the next layer of the current layer are generated. For the shallowest layer features, features of the feature scale of the current layer and features of the feature scale of the previous layer of the current layer are generated, and features of the same scale are connected in series along the channel dimension.

[0136] Combination Figure 6 As shown, further, the hybrid attention enhancement module is constructed, by focusing on the small-scale pedestrian target features in the two dimensions of channel and space, increasing the weight of the small-scale pedestrian target features in the training process, and suppressing the background features, including:

[0137] In the channel attention module, the input features are first subjected to maximum pooling and average pooling operations to obtain a compressed one-dimensional feature vector and , and for and Perform element-by-element addition operations to enable the one-dimensional feature vector to obtain the global receptive field of space, and perform two convolution operations to obtain the scales of and The one-dimensional feature vector of is the compression coefficient, which is used to learn the channel weight information, and the channel weight information is weighted multiplied with the input feature to obtain the channel attention feature map that integrates the channel attention;

[0138] In the spatial attention module, the channel attention feature map is subjected to average pooling and maximum pooling operations to obtain a compressed two-dimensional feature vector and , the two-dimensional feature vector is spliced ​​along the channel dimension so that the two-dimensional feature vector can obtain the channel global receptive field, and a convolution operation is performed to obtain a scale of The two-dimensional feature vector is obtained and the spatial weight information is obtained, and the spatial weight information is weighted multiplied with the channel attention feature map to obtain the final feature after processing.

[0139] Further, the preliminary post-processing of the head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection result to obtain a preliminary post-processing detection result, and filtering out redundant detection results, includes:

[0140] Perform non-maximum suppression processing on the head detection results based on the non-maximum suppression algorithm;

[0141] A non-maximum suppression algorithm based on target occlusion overlap rate compensation is constructed to perform non-maximum suppression on the overall pedestrian detection results. The occlusion overlap rate of pedestrian targets corresponding to the area of ​​interest is calculated. , so that the actual suppression threshold is adaptively adjusted as the degree of occlusion overlap changes. The occlusion overlap rate corresponding to the target box The calculation method can be expressed as:

[0142] (12)

[0143] (13)

[0144] in, Indicates The overlapping area of ​​a target frame is the maximum overlapping area of ​​the target frame and all other target frames. Indicates The area of ​​the target box itself;

[0145] The calculated The occlusion overlap rate corresponding to the target box The preset threshold in the NMS algorithm Compare them and take the larger value between the two as the decision threshold for non-maximum suppression processing.

[0146] Furthermore, the step of constructing a pedestrian head-body matching module and inputting the preliminary post-processing detection result into the pedestrian head-body matching module to match and bind the head detection frame and the overall detection frame includes:

[0147] The pedestrian head-body matching module includes three constraints, which are specifically expressed as follows:

[0148] Calculate the pedestrian head box Overall frame with pedestrians The overlap ratio between , the specific method is:

[0149] (14)

[0150] in, represents the overlapping area between the head frame and the pedestrian's overall frame, Represents the pedestrian head frame If the overlap ratio Below the preset threshold , then it is determined that the head and the whole body cannot be matched. If the overlap ratio Above the preset threshold , then proceed to the following steps;

[0151] Constructing a virtual scaled overall frame of pedestrians , and calculate the intersection-and-combination ratio between the pedestrian virtual zoomed overall frame and the pedestrian overall frame, and set the pedestrian head frame The coordinate information is , the width and height of the pedestrian head frame are respectively and To represent, the virtual zoom overall frame corresponding to the pedestrian head frame Coordinate information of It can be expressed as:

[0152] (15)

[0153] (16)

[0154] (17)

[0155] (18)

[0156] Calculate the virtual scaled overall frame With the overall frame If the intersection ratio between Below the preset threshold , it is determined that the head and the whole body cannot be matched; if it is higher than the preset threshold , then continue to judge the L1 distance between the two frame scales , which can be expressed as:

[0157] (19)

[0158] in, and Respectively represent the width and height of the pedestrian's overall frame, and Respectively represent the width and height of the virtual zoom overall frame. If The value is greater than the preset threshold , indicating that there is a large difference in scale between the head frame and the overall frame, so they cannot be matched, otherwise proceed to the next step;

[0159] Calculate the discrimination score of the relationship between the pedestrian head and the whole body. By constructing a relationship discrimination module consisting of three stacked fully connected layers, and extracting the information of each head corresponding to the whole matching pair, calculate the relationship matching score between the head and the whole body. If the maximum value of the relationship matching score is lower than the preset threshold Or when the relationship matching score set is an empty set, it means that the head box cannot be matched to the corresponding overall box.

[0160] As a preferred solution, the secondary post-processing module based on head suppression and full body recall is constructed, and the secondary post-processing module is used to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result to obtain the final pedestrian detection result, including:

[0161] The header frame set that failed to match successfully The elements in the image are matched again with the overall frame set that has not been processed with non-maximum suppression. After the first two matching constraints are met at the same time, the score calculated in the relationship judgment link is greater than a higher preset threshold. When , it is considered a successful match;

[0162] If the head and the whole are successfully paired, the pedestrian overall detection result previously suppressed by the non-maximum value is restored. If the maximum value of the relationship matching score is still lower than the preset threshold Or when the relationship matching score set is an empty set, the head detection result is determined to be a false detection, and the final pedestrian detection result is obtained.

[0163] The pedestrian detection method against occlusion, overlap and scale change provided in the embodiment of the present invention improves the ability of the feature extraction network to obtain image feature information and the expression ability of feature information by enhancing feature fusion and introducing the strategy of attention enhancement module in the feature extraction link, and makes the semantic information and detail texture information contained in the feature information more balanced, and uses the feature reconstruction module to make the semantic information and detail texture information covered in the acquired feature information more balanced, thereby improving the algorithm's detection ability for multi-scale pedestrian targets. In the core part of the model, the head and overall dual detection branches are designed for joint detection, and then the head detection is fully utilized to assist pedestrian detection to improve the detection ability of densely occluded pedestrian targets. In the post-processing link of the model, a more reasonable suppression discrimination mechanism is constructed for the overall detection frame of pedestrians with severe occlusion and overlap, and then the constructed pedestrian head-body matching strategy is used to match and bind the pedestrian head detection frame and the overall detection frame, and the head detection frame that fails to match successfully is checked for secondary matching, which is used to restore the pedestrian overall detection frame that is incorrectly suppressed and eliminate the head detection frame that is misdetected, so as to further improve the detection accuracy of pedestrian targets.

[0164] In order to further facilitate the understanding of the solution of the present invention, the following describes a pedestrian detection method for resisting occlusion overlap and scale change under an embodiment of the present invention, and the method includes:

[0165] Step 1: Build the Faster RCNN network model;

[0166] Step 2: Fuse the backbone network of the Faster RCNN network model with the improved feature extraction network, as follows Figure 2 As shown, the image to be detected is input into the Faster RCNN network model integrated with the improved feature extraction network for feature extraction to obtain an extracted feature map;

[0167] Step 2.1: Use the ResNet50 network as the backbone network of the Faster RCNN network model. The specific structure of the ResNet50 network is as follows: Figure 3 As shown;

[0168] Step 2.2: Fuse the improved feature extraction network with the Faster RCNN network model, specifically:

[0169] Step 2.2.1: Use the backbone network ResNet50 to learn the feature information of the image to be detected;

[0170] Step 2.2.2: Perform feature fusion on the feature information obtained by the backbone network, specifically:

[0171] Step 2.2.2.1: Feature fusion is performed through the feature pyramid FPN structure. The specific process can be expressed as:

[0172] (1)

[0173] (2)

[0174] in, Represents the backbone network The convolution operation of the stage, Represents the top-down feature pyramid network Stage feature fusion operation, For the The features learned by the layer network, The feature pyramid The characteristics of the stage output, is the number of feature layers.

[0175] Step 2.2.2.2: Combined with the idea of ​​cyclic recursion, a feedback splicing channel from back to front is added on the basis of the feature pyramid FPN structure to construct a cyclic recursive feature pyramid. Figure 4 As shown, it is used to feed back the feature information output by the FPN structure and splice it into the backbone network, so that the backbone network can perform secondary purification processing on the features. The specific process can be expressed as:

[0176] (3)

[0177] (4)

[0178] in, It represents the feature transformation operation before splicing the learned feature feedback to the backbone network, which is used to realize the cascade connection of the two stages. Then repeat the feature extraction and fusion operation of the first stage to obtain the output feature of the second stage. , the features of the two stages output in the recursive feature pyramid and It can be expressed as:

[0179] (5)

[0180] (6)

[0181] (7)

[0182] (8)

[0183] In step 2.2.2.2, the learned feature feedback is spliced ​​to the feature transformation operation before the backbone network The specific implementation process is as follows: First, the transformed features conduct The convolution operation is then performed with the features output by the first residual block in each stage of the backbone network to complete the feedback splicing operation. As for the feature transformation processing before the feedback is spliced ​​to the backbone network, the spatial pyramid pooling module ASPP fused with dilated convolution is used to implement it. Figure 5 As shown, it is used to realize the cascade connection of two stages.

[0184] The ASPP module includes four parallel branches. The first three branches all adopt convolutional layer structures, with the sizes of convolution kernels being 1×1, 3×3, and 3×3, respectively, and setting successively increasing void rates, and then connecting ReLU layers at the back to increase the receptive field range. The fourth branch is based on the global average pooling operation, and is connected to a 1×1 ordinary convolutional layer and a ReLU layer at the back to evenly extract effective information and reduce local information loss. Finally, the features output by the four branches are connected along the channel dimension to generate a feature with the same scale as the input feature of the second stage, thereby completing the feature transformation processing operation.

[0185] Step 2.2.2.3: The backbone network performs secondary learning to obtain image features, and then converts the features output from the two stages into and The specific implementation process is: construct an adaptive feature fusion method and assign weights to the features output from the two stages , weight It is calculated by the convolution layer and Sigmoid function after the output feature of the second stage. Then according to the weight The two-stage features are weighted and fused, which is specifically expressed as follows:

[0186] (9)

[0187] in, and are the weights assigned to the output features of the two stages, and the weights and The relationship between . It is the final output feature of the feature extraction network.

[0188] Step 2.3: Construct a feature reconstruction module, and use the correlation between adjacent layers of the feature pyramid to construct a new feature layer, add detailed texture information to the deeper features, and add semantic information to the shallower features to enhance the expressiveness of the features. The feature reconstruction module consists of two parts, namely the feature scale scaling module and the feature transformation module. Its specific execution process can be expressed as:

[0189] Step 2.3.1: The feature scale scaling module generates features of different scales through upsampling or downsampling operations.

[0190] Step 2.3.2: Use the feature transformation module to combine all features of the same scale in series to form new features. The specific method is as follows: for the intermediate-level features, use the feature scale scaling module to generate features of the current layer feature scale, features of the previous layer feature scale, and features of the next layer feature scale. For the deepest layer features, generate features of the current layer feature scale and features of the next layer feature scale. For the shallowest layer features, generate features of the current layer feature scale and features of the previous layer feature scale. Finally, the features of the same scale are connected in series along the channel dimension.

[0191] Step 2.4: Construct a hybrid attention enhancement module, as follows Figure 6 As shown. By focusing on the features of small-scale pedestrian targets in both channel and space dimensions, increasing their weight in the training process, and suppressing background features, the expressiveness of target features is enhanced. The hybrid attention enhancement module is composed of a channel attention module and a spatial attention module in series. First, channel attention information is extracted from the input feature map to measure the importance of each feature channel in all channels. Then spatial attention enhancement is performed, and spatial weight information is learned. Finally, a weighted multiplication operation is performed to obtain the final feature map. The specific execution process can be expressed as:

[0192] (10)

[0193] (11)

[0194] in, represents the input feature map, represents the channel attention enhancement link, represents the spatial attention enhancement link, The feature map representing the fused channel attention, Represents the feature map obtained after final processing.

[0195] Step 2.4.1: In the channel attention module, first perform maximum pooling and average pooling operations on the input features to obtain a compressed one-dimensional feature vector and , and for and Perform element-by-element summation to obtain the global spatial receptive field. Then perform two convolution operations to obtain the scales of and The one-dimensional feature vector of is the compression coefficient, which is used to learn the channel weight information, and the channel weight information is weighted multiplied with the input features to obtain the feature map of the fused channel attention.

[0196] Step 2.4.2: In the spatial attention module, firstly, the channel attention feature map obtained in step 2.4.1 is averaged and max-pooled to obtain a compressed two-dimensional feature vector. and , and concatenate them along the channel dimension to obtain the global receptive field of the channel. Then perform a convolution operation to obtain a scale of The two-dimensional feature vector of the fusion channel is obtained and the spatial weight information is obtained. The spatial weight information is weighted multiplied with the feature map of the fusion channel attention to obtain the final feature after processing.

[0197] Step 3: Generate a set of candidate frames for the pedestrian head through the region proposal network, and perform statistical calculations based on the {head~body} region combination of the calibrated pedestrian targets in the dataset to approximate the head-to-body ratio. The width of the head region is about 0.3 times the width of the pedestrian's overall region, and the height is about 1 / 6 of the height of the pedestrian's overall region. Based on this head-to-body ratio, a set of candidate frames for the pedestrian's overall region is generated.

[0198] Step 4: Construct a pedestrian head detection branch and a pedestrian overall detection branch, and obtain preliminary target detection results through the pedestrian head detection branch module and the pedestrian overall detection branch module;

[0199] Step 5: Perform preliminary post-processing on the obtained head detection frame and pedestrian overall detection frame to filter out some redundant detection results. The specific process is as follows:

[0200] Step 5.1: Perform non-maximum suppression on the head detection results based on the non-maximum suppression algorithm (NMS algorithm), where the NMS preset threshold Set to 0.5.

[0201] Step 5.2: Construct a non-maximum suppression algorithm based on target occlusion overlap rate compensation to perform non-maximum suppression on the overall pedestrian detection results. First, calculate the occlusion overlap rate of the pedestrian target corresponding to the area of ​​interest. , so that the actual suppression threshold can be adaptively adjusted as the degree of occlusion overlap changes, thereby replacing the judgment strategy based on manually preset thresholds in the traditional NMS algorithm. The occlusion overlap rate corresponding to the target box The calculation method can be expressed as:

[0202] 12)

[0203] (13)

[0204] in, Indicates The overlapping area of ​​a target frame is the maximum overlapping area of ​​the target frame and all other target frames. Indicates The area of ​​the target box itself.

[0205] Step 5.3: Substitute the first The occlusion overlap rate corresponding to the target box The preset threshold in the NMS algorithm Compare them and take the larger value between the two as the decision threshold for non-maximum suppression processing.

[0206] Step 6: Build a pedestrian head-body matching module, and send the preliminary post-processed detection results to the pedestrian head-body matching module to match and bind the head detection frame and the overall detection frame. The head-body matching module includes three constraints, which can be specifically expressed as:

[0207] Step 6.1: Calculate the pedestrian head box Overall frame with pedestrians The overlap ratio between , the specific method is:

[0208] (14)

[0209] in, Indicates the overlapping area between the head frame and the whole body frame. Represents the pedestrian head frame If the overlap ratio Below the preset threshold , it is determined that the head and the whole body cannot be matched. In the present invention, after adjusting the reasoning, the The value is 0.7. If the overlap ratio Above the preset threshold , then go to step 6.2;

[0210] Step 6.2: Construct the virtual scaled overall box of the pedestrian , and calculate the intersection-over-union ratio between it and the pedestrian's overall frame. Set the pedestrian head frame The coordinate information is , whose width and height are respectively and To represent, the virtual scaled overall frame corresponding to the target frame Coordinate information of It can be expressed as:

[0211] (15)

[0212] (16)

[0213] (17)

[0214] (18)

[0215] Then calculate Overall frame with pedestrians If the intersection ratio between Below the preset threshold , it is determined that the head and the whole body cannot be matched. In the present invention, after adjusting the reasoning, the The value is 0.5; if it is higher than the preset threshold , then continue to judge the L1 distance between the two frame scales , which can be expressed as:

[0216] (19)

[0217] in, and Respectively represent the width and height of the pedestrian's overall frame, and Respectively represent the width and height of the virtual zoom overall frame. The value is greater than the preset threshold , indicating that there is a large difference in scale between the head frame and the overall frame, so they cannot be matched, otherwise go to step 6.3; in the present invention, after adjusting the reasoning, set The value is 60.

[0218] Step 6.3: Calculate the discrimination score of the relationship between the pedestrian head and the whole body. By constructing a relationship discrimination module consisting of three stacked fully connected layers, and extracting the information of each {head~whole} matching pair for it. Then calculate the relationship matching score between the head and the whole body. If the maximum value of the relationship matching score is lower than the preset threshold Or when the relationship matching score set is an empty set, it means that the head frame cannot match the corresponding overall frame. In the present invention, after the adjustment, the reasoning is set The value is 0.1.

[0219] Step 7: Construct a secondary post-processing module based on head suppression and full body recall, and use the secondary post-processing module to perform secondary verification on the head detection frames that failed to match successfully. The specific method is as follows: for the head frame set that failed to match successfully in step 6 For the elements in , it is matched again with the overall frame set that has not been processed with non-maximum suppression. After the first two matching constraints are met at the same time, only when the score calculated in the relationship judgment link is greater than a higher preset threshold Only when it is considered a successful match, the present invention is set after the adjustment participation reasoning The value is 0.9. If the head and the whole can be successfully matched, the pedestrian overall detection result previously suppressed by the non-maximum value is restored. If the maximum value of the relationship matching score is still lower than the preset threshold Or when the relationship matching score set is an empty set, the head detection result is determined to be a false detection. At this point, the final pedestrian detection result is obtained.

[0220] The loss function used in the method involved in the present invention can be expressed as:

[0221] (20)

[0222] (twenty one)

[0223] (twenty two)

[0224] (twenty three)

[0225] (twenty four)

[0226] in, represents the overall loss function, represents the classification loss function in the candidate box extraction network, represents the regression loss function in the candidate box extraction network, represents the classification loss function in the candidate box classification network, represents the regression loss function in the candidate box extraction network, Indicates the number of samples involved in training in the candidate box extraction network. Indicates the number of samples involved in training in the candidate box classification network. Represents the score calculated by the predicted category in the candidate box extraction network, express The corresponding true value label, Represents the score calculated by the predicted category in the candidate box classification network, express The corresponding true value label, represents the bounding box regression offset predicted in the candidate box extraction network, express The corresponding true value label, represents the bounding box regression offset predicted in the candidate box classification network, express The corresponding true value label.

[0227] Compared with the prior art, the present invention has the following characteristics:

[0228] 1. Compared with the existing YOLO series network models and RCNN series network models, the method involved in the present invention enhances the ability of the feature extraction network to obtain image feature information and the expression ability of feature information by enhancing feature fusion and introducing the strategy of attention enhancement module in the feature extraction link, and makes the semantic information and detail texture information contained in the feature information more balanced, and uses the feature reconstruction module to make the semantic information and detail texture information covered in the acquired feature information more balanced, thereby improving the algorithm's detection ability for multi-scale pedestrian targets.

[0229] 2. Compared with the existing YOLO series network models and RCNN series network models, the method involved in the present invention designs a head and overall dual detection branch in the core part of the model for joint detection, and then makes full use of head detection to assist pedestrian detection to improve the detection capability of densely occluded pedestrian targets.

[0230] 3. Compared with the existing YOLO series network models and RCNN series network models, the method involved in the present invention constructs a more reasonable suppression discrimination mechanism for the overall detection frame of pedestrians with severe occlusion and overlap in the post-processing link of the model, and then uses the constructed pedestrian head-body matching strategy to match and bind the pedestrian head detection frame and the overall detection frame, and performs a secondary matching check on the head detection frame that fails to match successfully, so as to restore the incorrectly suppressed pedestrian overall detection frame and eliminate the misdetected head detection frame, so as to further improve the detection accuracy of pedestrian targets.

[0231] The effectiveness of the present invention can be verified by the following experiments:

[0232] 1. Experimental conditions and contents

[0233] 1. The ImageNet dataset is used as the pre-training dataset for the pedestrian detection model, and then the CrowdHuman dataset and CityPersons dataset are used to train the pedestrian detection model. During the training process, some enhancement strategies are used for the dataset, including random cropping, random horizontal flipping, and random interference of image brightness. In the actual training process, the stochastic gradient descent optimizer is used for 35 epochs of training, the momentum factor momentum is set to 0.9, the learning rate adopts the warm up strategy, and the initial learning rate is set to a lower value. In this invention, it is set to , from the 1st iteration to the 800th iteration of each epoch, the learning rate increases linearly to , and then remain unchanged. By the 24th epoch, the learning rate was reduced to 10% of the original, and by the 28th epoch, the learning rate became 1% of the original.

[0234] 2. Use Average Precision (AP) and Logarithmic Mean Missed Detection Rate (MR) -2 ) is used as an evaluation index to evaluate the comprehensive performance of the method involved in the present invention. The specific calculation method of the average precision and the logarithmic average missed detection rate can be expressed as:

[0235] (25)

[0236] (26)

[0237] (27)

[0238] (28)

[0239] (29)

[0240] (30)

[0241] in, represents the accuracy, which is the ratio of the number of samples whose predicted results are pedestrian targets to the number of samples that are actually marked as pedestrian targets in the image; TP represents the case where both the predicted result and the true result are pedestrian targets, indicating that the pedestrian target is correctly predicted; FN represents the case where the predicted result is background information and the true result is a pedestrian target, indicating that a pedestrian is missed; FP represents the case where the predicted result is a pedestrian target and the true result is background information, indicating that a pedestrian is falsely detected; Recall is the ratio of the number of correctly predicted pedestrian targets to all predicted results in the process of pedestrian target detection. represents the missed detection rate, represents the average number of false positives in each image, Indicates the number of all pedestrian targets in the image.

[0242] (II) Experimental results

[0243] Combination Figure 7 and Figure 8 It can be seen that on the CrowdHuman and CityPersons pedestrian detection datasets, compared with some popular pedestrian detection algorithms and the benchmark Faster RCNN algorithm before improvement, the average detection accuracy of the present invention is higher and the logarithmic average missed detection rate is lower. Therefore, the experimental results can fully demonstrate that the present invention has better performance than some popular pedestrian detection algorithms and can better detect some small-scale pedestrian targets and some severely occluded pedestrian targets.

[0244] Compared with the prior art, the present invention has the following characteristics:

[0245] 1. Compared with the existing YOLO series network models and RCNN series network models, the method involved in the present invention enhances the ability of the feature extraction network to obtain image feature information and the expression ability of feature information by enhancing feature fusion and introducing the strategy of attention enhancement module in the feature extraction link, and makes the semantic information and detail texture information contained in the feature information more balanced, and uses the feature reconstruction module to make the semantic information and detail texture information covered in the acquired feature information more balanced, thereby improving the algorithm's detection ability for multi-scale pedestrian targets.

[0246] 2. Compared with the existing YOLO series network models and RCNN series network models, the method involved in the present invention designs a head and overall dual detection branch in the core part of the model for joint detection, and then makes full use of head detection to assist pedestrian detection to improve the detection capability of densely occluded pedestrian targets.

[0247] 3. Compared with the existing YOLO series network models and RCNN series network models, the method involved in the present invention constructs a more reasonable suppression discrimination mechanism for the overall detection frame of pedestrians with severe occlusion and overlap in the post-processing link of the model, and then uses the constructed pedestrian head-body matching strategy to match and bind the pedestrian head detection frame and the overall detection frame, and performs a secondary matching check on the head detection frame that fails to match successfully, so as to restore the incorrectly suppressed pedestrian overall detection frame and eliminate the misdetected head detection frame, so as to further improve the detection accuracy of pedestrian targets.

[0248] The technical solution designed by the present invention can be well applied to the fields of intelligent monitoring, assisted safe driving, video monitoring of large venues, intelligent robots, etc.

[0249] Combination Fig. 9 Accordingly, the present invention provides a pedestrian detection device that is resistant to occlusion, overlap and scale change, comprising:

[0250] A construction unit 901 is used to construct a Faster RCNN network model;

[0251] A fusion unit 902 is used to fuse the backbone network of the Faster RCNN network model with the improved feature extraction network, and input the image to be detected into the Faster RCNN network model for feature extraction to obtain an extracted feature map;

[0252] A construction unit 903 is used to generate a set of candidate frames of a pedestrian head through a region proposal network, and perform statistical calculations based on a combination of a pedestrian head region calibrated in a data set and an overall region corresponding to the pedestrian head region to obtain a head-to-body ratio relationship of a pedestrian target, and generate a set of overall candidate frames of a pedestrian based on the head-to-body ratio relationship of a pedestrian target;

[0253] A preliminary detection unit 904 is used to construct a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtain a preliminary target detection result in the image to be detected through the pedestrian head detection branch module and the pedestrian overall detection branch module;

[0254] A post-processing detection unit 905 is used to perform preliminary post-processing on the head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection result to obtain a preliminary post-processing detection result, and filter out redundant detection results;

[0255] An input unit 906 is used to construct a pedestrian head-body matching module, and input the preliminary post-processing detection result into the pedestrian head-body matching module to match and bind the head detection frame and the overall detection frame;

[0256] The secondary verification unit 907 is used to construct a secondary post-processing module based on head suppression and full-body recall, and use the secondary post-processing module to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result, restore the overall detection frame that was incorrectly suppressed and eliminate the falsely detected head detection frame, so as to obtain the final pedestrian detection result.

[0257] The pedestrian detection device with resistance to occlusion, overlap and scale change provided in the embodiment of the present invention improves the ability of the feature extraction network to obtain image feature information and the ability to express feature information by enhancing feature fusion and introducing the strategy of attention enhancement module in the feature extraction link, and makes the semantic information and detail texture information contained in the feature information more balanced, and uses the feature reconstruction module to make the semantic information and detail texture information covered in the acquired feature information more balanced, thereby improving the algorithm's detection ability for multi-scale pedestrian targets. In the core part of the model, the head and overall dual detection branches are designed for joint detection, and then the head detection is fully utilized to assist pedestrian detection to improve the detection ability of densely occluded pedestrian targets. In the post-processing link of the model, a more reasonable suppression discrimination mechanism is constructed for the overall detection frame of pedestrians with serious occlusion and overlap, and then the constructed pedestrian head-body matching strategy is used to match and bind the pedestrian head detection frame and the overall detection frame, and the head detection frame that fails to match successfully is checked for secondary matching, which is used to restore the pedestrian overall detection frame that is incorrectly suppressed and eliminate the head detection frame that is misdetected, so as to further improve the detection accuracy of pedestrian targets.

[0258] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0259] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A pedestrian detection method that is resistant to occlusion, overlap and scale changes, characterized in that: include: Build the Faster RCNN network model; The backbone network of the Faster RCNN network model is integrated with the improved feature extraction network, and the image to be detected is input into the Faster RCNN network model for feature extraction to obtain an extracted feature map; Generate a set of candidate frames for pedestrian heads through a region proposal network, and perform statistical calculations based on the combination of the calibrated pedestrian head region in the data set and the overall region corresponding to the pedestrian head region to obtain the head-to-body ratio of the pedestrian target, and generate a set of overall candidate frames for the pedestrian based on the head-to-body ratio of the pedestrian target; Constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result in the image to be detected through the pedestrian head detection branch module and the pedestrian overall detection branch module; Performing preliminary post-processing on the pedestrian head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection results to obtain preliminary post-processing detection results, and filtering out redundant detection results; Constructing a pedestrian head-body matching module, and inputting the preliminary post-processing detection result into the pedestrian head-body matching module, matching and binding the head detection frame and the overall detection frame; A secondary post-processing module based on head suppression and full-body recall is constructed, and the secondary post-processing module is used to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result, restore the overall detection frame that was incorrectly suppressed and eliminate the falsely detected head detection frame, so as to obtain the final pedestrian detection result.

2. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 1, characterized in that: The backbone network of the Faster RCNN network model is integrated with the improved feature extraction network, and the image to be detected is input into the Faster RCNN network model for feature extraction to obtain an extracted feature map, including: Use the ResNet50 network as the backbone network of the Faster RCNN network model; Fusing the improved feature extraction network with the Faster RCNN network model; Constructing a feature reconstruction module, wherein the feature reconstruction module includes a feature scale scaling module and a feature transformation module; Construct a hybrid attention enhancement module to increase the weight of small-scale pedestrian target features in the training process by focusing on them in both channel and spatial dimensions, while suppressing background features; The hybrid attention enhancement module is composed of a channel attention module and a spatial attention module connected in series. The channel attention module is used to extract channel attention information from the input feature map, and the spatial attention module is used to enhance spatial attention and learn spatial weight information. Finally, a weighted multiplication operation is performed to obtain the final feature map.

3. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 2, characterized in that: The improved feature extraction network is integrated with the Faster RCNN network model, including: Using the backbone network ResNet50 to learn the feature information of the image to be detected; The feature information obtained by the backbone network ResNet50 is subjected to feature fusion.

4. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 3, characterized in that: The feature information obtained by the backbone network ResNet50 is subjected to feature fusion, including: The feature fusion is performed through the feature pyramid FPN structure. The specific process is expressed by formula (1) and formula (2): (1) (2) in, Represents the backbone network The convolution operation of the stage, Represents the top-down feature pyramid network Stage feature fusion operation, For the The features learned by the layer network, The feature pyramid The characteristics of the stage output, is the number of feature layers; Combining the idea of ​​cyclic recursion, a feedback splicing channel from back to front is added on the basis of the feature pyramid FPN structure to construct a cyclic recursive feature pyramid. The cyclic recursive feature pyramid is used to feedback and splice the feature information output by the feature pyramid FPN structure to the backbone network, so that the backbone network can perform secondary purification processing on the features. The specific process is expressed by formula (3) and formula (4): (3) (4) in, It represents the feature transformation operation before splicing the learned feature feedback to the backbone network, which is used to realize the cascade connection of the two stages, and then repeat the feature extraction and fusion operation of the first stage to obtain the output feature of the second stage , the features of the two stages output in the recursive feature pyramid and It can be expressed by formula (5) to formula (8): (5) (6) (7) (8) The feature transformation operation before splicing the learned feature feedback to the backbone network The specific implementation process is: After transformation, the features conduct Convolution operation; The features output by the first residual block in each stage of the backbone network are fused to complete the feedback splicing operation. For the feature transformation processing link before the feedback is spliced ​​to the backbone network, the spatial pyramid pooling module ASPP fused with the hole convolution is adopted to realize the cascade connection of the two stages. The spatial pyramid pooling module ASPP includes four parallel branches, wherein the first three branches all adopt the convolution layer structure, the sizes of the convolution kernels are 1×1, 3×3 and 3×3 respectively, and the hole rates are set to increase successively, and the ReLU layer is connected at the back to increase the receptive field range. The fourth branch is based on the global average pooling operation, and the 1×1 ordinary convolution layer and the ReLU layer are connected at the back to evenly extract effective information and reduce local information loss; the features output by the four branches are connected along the channel dimension to generate a feature with the same scale as the input feature of the second stage, and the feature transformation processing operation is completed; The backbone network performs secondary learning to obtain the image features of the image to be detected, and converts the features output from the two stages into and The specific process is as follows: construct an adaptive feature fusion method and assign weights to the features output from the two stages , weight It is calculated by the convolutional layer and Sigmoid function after the output feature of the second stage, according to the weight The two-stage features are weightedly fused, which is specifically expressed by formula (9): (9) in, and are the weights assigned to the output features of the two stages, and the weights and The relationship between , It is the final output feature of the feature extraction network.

5. The pedestrian detection method against occlusion overlap and scale change according to claim 1 or 2, characterized in that: Construct a feature reconstruction module, wherein the feature reconstruction module includes a feature scale scaling module and a feature transformation module, including: The feature scale scaling module generates features of different scales by upsampling or downsampling operations; The feature transformation module is used to combine all features of the same scale in series to form new features. The specific method is as follows: for intermediate-level features, the feature scale scaling module is used to generate features of the feature scale of the current layer, features of the feature scale of the previous layer of the current layer, and features of the feature scale of the next layer of the current layer for the intermediate-level features; for the deepest-level features, features of the feature scale of the current layer and features of the feature scale of the next layer of the current layer are generated for the deepest-level features, and for the shallowest-level features, features of the feature scale of the current layer and features of the feature scale of the previous layer of the current layer are generated for the shallowest-level features, and features with the same scale are connected in series along the channel dimension.

6. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 2, characterized in that: The hybrid attention enhancement module is constructed to increase the weight of small-scale pedestrian target features in the training process by focusing on the small-scale pedestrian target features in the two dimensions of channel and space, while suppressing background features, including: In the channel attention module, the input features are first subjected to maximum pooling and average pooling operations to obtain a compressed one-dimensional feature vector and , and for and Perform element-by-element addition operations to enable the one-dimensional feature vector to obtain the global receptive field of space, and perform two convolution operations to obtain the scales of and The one-dimensional feature vector of is the compression coefficient, which is used to learn the channel weight information, and the channel weight information is weighted multiplied with the input feature to obtain the channel attention feature map that integrates the channel attention; In the spatial attention module, the channel attention feature map is subjected to average pooling and maximum pooling operations to obtain a compressed two-dimensional feature vector and , the two-dimensional feature vector is spliced ​​along the channel dimension so that the two-dimensional feature vector can obtain the channel global receptive field, and a convolution operation is performed to obtain a scale of The two-dimensional feature vector is obtained and the spatial weight information is obtained, and the spatial weight information is weighted multiplied with the channel attention feature map to obtain the final feature after processing.

7. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 6, characterized in that: The head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection result are preliminarily post-processed to obtain a preliminary post-processed detection result, and redundant detection results are screened out, including: Perform non-maximum suppression processing on the head detection results based on the non-maximum suppression algorithm; A non-maximum suppression algorithm based on target occlusion overlap rate compensation is constructed to perform non-maximum suppression on the overall pedestrian detection results. The occlusion overlap rate of pedestrian targets corresponding to the area of ​​interest is calculated. , so that the actual suppression threshold is adaptively adjusted as the degree of occlusion overlap changes. The occlusion overlap rate corresponding to the target box The calculation method is expressed as: (12) (13) in, Indicates The overlapping area of ​​the target frame is the maximum value of the overlapping area between the target frame and all other target frames. Indicates The area of ​​the target box itself; The calculated The occlusion overlap rate corresponding to the target box The preset threshold in the NMS algorithm Compare them and take the larger value between the two as the decision threshold for non-maximum suppression processing.

8. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 7, characterized in that: The step of constructing a pedestrian head-body matching module, inputting the preliminary post-processing detection result into the pedestrian head-body matching module, and matching and binding the head detection frame and the overall detection frame includes: The pedestrian head-body matching module includes three constraints, which are specifically expressed as follows: Calculate the pedestrian head box Overall frame with pedestrians The overlap ratio between , the specific method is: (14) in, represents the overlapping area between the head frame and the pedestrian's overall frame, Represents the pedestrian head frame If the overlap ratio Below the preset threshold , then it is determined that the head and the whole body cannot be matched. If the overlap ratio Above the preset threshold , then proceed to the following steps; Constructing a virtual scaled overall frame of pedestrians , and calculate the intersection-and-combination ratio between the pedestrian virtual zoomed overall frame and the pedestrian overall frame, and set the pedestrian head frame The coordinate information is , the width and height of the pedestrian head frame are respectively and To represent, the virtual zoom overall frame corresponding to the pedestrian head frame Coordinate information of It is expressed as: (15) (16) (17) (18) Calculate the virtual scaled overall frame With the overall frame If the intersection ratio between Below the preset threshold , it is determined that the head and the whole body cannot be matched; if it is higher than the preset threshold , then continue to judge the L1 distance between the two frame scales , specifically expressed as: (19) in, and Respectively represent the width and height of the pedestrian's overall frame, and Respectively represent the width and height of the virtual zoom overall frame. If The value is greater than the preset threshold , indicating that there is a large difference in scale between the head frame and the overall frame, so they cannot be matched, otherwise proceed to the next step; Calculate the discrimination score of the relationship between the pedestrian head and the whole body. By constructing a relationship discrimination module consisting of three stacked fully connected layers, and extracting the information of each head corresponding to the whole matching pair, calculate the relationship matching score between the head and the whole body. If the maximum value of the relationship matching score is lower than the preset threshold Or when the relationship matching score set is an empty set, it means that the head box cannot be matched to the corresponding overall box.

9. The pedestrian detection method against occlusion overlap and scale change as claimed in claim 8, characterized in that: The secondary post-processing module based on head suppression and full body recall is constructed, and the secondary post-processing module is used to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result to obtain the final pedestrian detection result, including: The header frame set that failed to match successfully The elements in the image are matched again with the overall frame set that has not been processed with non-maximum suppression. After the first two matching constraints are met at the same time, the score calculated in the relationship judgment link is greater than a higher preset threshold. When , it is considered a successful match; If the head and the whole are successfully paired, the pedestrian overall detection result previously suppressed by the non-maximum value is restored. If the maximum value of the relationship matching score is still lower than the preset threshold Or when the relationship matching score set is an empty set, the head detection result is determined to be a false detection, and the final pedestrian detection result is obtained.

10. A pedestrian detection device that is resistant to occlusion, overlap and scale changes, characterized in that: include: Construction unit, used to build Faster RCNN network model; A fusion unit, used to fuse the backbone network of the Faster RCNN network model with the improved feature extraction network, and input the image to be detected into the Faster RCNN network model for feature extraction to obtain an extracted feature map; A construction unit is used to generate a set of candidate frames of pedestrian heads through a region proposal network, and perform statistical calculations based on a combination of a pedestrian head region calibrated in a data set and an overall region corresponding to the pedestrian head region to obtain a head-to-body ratio relationship of a pedestrian target, and generate a set of overall candidate frames of a pedestrian based on the head-to-body ratio relationship of the pedestrian target; A preliminary detection unit, used to construct a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtain a preliminary target detection result in the image to be detected through the pedestrian head detection branch module and the pedestrian overall detection branch module; A post-processing detection unit, used for performing preliminary post-processing on the head detection frame and the pedestrian overall detection frame in the obtained preliminary target detection result to obtain a preliminary post-processing detection result, and filtering out redundant detection results; An input unit, used to construct a pedestrian head-body matching module, and input the preliminary post-processing detection result into the pedestrian head-body matching module, and match and bind the head detection frame and the overall detection frame; The secondary verification unit is used to construct a secondary post-processing module based on head suppression and full-body recall, and use the secondary post-processing module to perform secondary verification processing on the head detection frame that failed to match successfully in the preliminary post-processing detection result, restore the overall detection frame that was incorrectly suppressed and eliminate the falsely detected head detection frame, so as to obtain the final pedestrian detection result.

Images