Multi-scale pedestrian detection method and device based on head and overall information combination

Abstract

The invention relates to the technical field of computer vision, in particular to a multi-scale pedestrian detection method and device based on combination of head and overall information, and aims to improve the sensitivity of a network to a multi-scale pedestrian target by densely connecting different hierarchical features; secondly, the sampling mode of the regional suggestion network is optimized, and the adaptive capacity of the model to a shielded pedestrian target is improved by calculating the shielding overlapping rate of each sample in a sample set; and then a joint detection framework of pedestrian head and overall information is constructed, and adverse effects on detection caused by shielding of a pedestrian body are reduced. A post-processing link and a loss function module are optimized, the interference of adjacent pedestrian targets on detection is weakened, the intelligence and rationality of screening redundant frames are improved, and the omission ratio and false detection ratio of pedestrian detection are further reduced. According to the invention, the detection capability of multi-scale pedestrian targets and sheltered pedestrian targets in a complex crowd dense scene is stronger, and the pedestrian omission ratio can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically provides a multi-scale pedestrian detection method and device based on the combination of head and overall information. Background Art

[0002] In the field of computer vision, pedestrian detection technology is one of the most widely used and popular research directions. The pedestrian detection task mainly refers to accurately detecting and locating pedestrian target instances in images, videos and video stream data through computer vision related technologies. This task is essentially a classification and regression process. In real life, pedestrian detection plays an important role in the fields of autonomous driving, intelligent monitoring, intelligent robots and human-computer interaction and has high application value. Among them, in the fields of autonomous driving and assisted safe driving, pedestrian detection technology can be used to detect in real time whether there are pedestrians suddenly breaking into the front of the vehicle, so as to make timely adjustments according to the actual situation to ensure pedestrian safety and driving safety. Especially in congested sections and crowded scenes, the auxiliary role of pedestrian detection technology in autonomous driving safety is more obvious. In the field of intelligent monitoring, most public places today use cameras to monitor the entire scene and count crowd flow data in real time. Especially during the epidemic, most large public places will strictly control the density of pedestrian flow. The use of pedestrian detection technology can accurately count the pedestrian flow in the venue in real time, and analyze and predict these data, which will help managers take corresponding adjustment measures. In the field of intelligent robots, sensors such as cameras transmit corresponding environmental scene signals to intelligent robots. Pedestrian detection algorithms, as an important thinking perception network in the brain of intelligent robots, can help intelligent robots quickly and accurately perceive pedestrian targets and make corresponding decisions in time to make adjustments. In the field of human-computer interaction, smart food delivery vehicles and smart express delivery vehicles in campuses and restaurants have integrated multiple functions such as pedestrian detection algorithms, and through interaction with pedestrians, they achieve the purpose of using artificial intelligence technology to better serve people's daily lives.

[0003] Pedestrian detection algorithms based on deep learning can be divided into single-stage pedestrian detection algorithms and two-stage pedestrian detection algorithms according to the detection ideas. Among them, the single-stage pedestrian detection algorithm is mainly represented by the YOLO series network model, and the two-stage pedestrian detection algorithm is represented by the R-CNN series network model. However, both types of algorithms lack research and design for specific detection scenarios, and have weak detection capabilities for small-scale pedestrian targets at farther viewing angles in dense scenes and pedestrian targets with severe occlusion and overlap, which is mainly manifested in low detection accuracy and high missed detection rate. In addition, both types of algorithms ignore an important issue: although the body of pedestrians is easily occluded and overlapped in complex and dense scenes, their head areas are often less occluded or even completely unoccluded. Even if the body of pedestrians is severely occluded, the head area can still provide important feature information, which is very important for detecting pedestrian targets in dense scenes. However, the overall scale of the head area is small, which is easy to cause ambiguity on the hands, elbows and surrounding small objects, thereby causing false detection. It can be seen that it is difficult to achieve accurate pedestrian detection by relying solely on overall detection and head detection. It is necessary to effectively combine head detection with overall detection, and obtain more abundant and detailed multi-scale feature information in the feature extraction process, so as to better play the respective advantages of head detection and overall detection and improve the accuracy of pedestrian detection. Therefore, it is very necessary to design a multi-scale pedestrian detection algorithm based on the combination of head and overall information, which is of great significance for improving the detection accuracy of occluded pedestrian targets and multi-scale pedestrian targets in dense scenes.

[0004] In response to the above needs, there are currently many relevant solutions at home and abroad.

[0005] The Chinese patent publication number is CN111767882A, the publication date is October 13, 2020, and the patent name is "A multimodal pedestrian detection method based on an improved YOLO model". The invention patent application improves the pedestrian detection effect by integrating the CBAM attention mechanism and optimizing the loss function. However, there are also some problems, such as (1) the sensitivity to multi-scale pedestrian targets in the feature extraction link is not high enough, especially the feature information learning ability of small-scale pedestrians at a farther viewing angle is weak. (2) There is still room for improvement in the phenomenon of missed detection of pedestrian targets with serious occlusion and overlap. The Chinese patent publication number is CN113989939A, the publication date is January 28, 2022, and the patent name is "A small target pedestrian detection system based on an improved YOLO algorithm". However, in actual detection scenes, there is inevitably dense crowding of pedestrians. Therefore, this method has weak detection ability for occluded pedestrian targets and has certain limitations in practical applications. The Chinese patent publication number is CN114882527A, the publication date is August 9, 2022, and the patent name is "Pedestrian detection method and system based on dynamic group convolution". The invention patent application mainly uses group convolution to achieve pedestrian detection, but lacks consideration of specific scenarios for pedestrian detection, and there are certain limitations in the execution efficiency in complex and crowded scenes.

[0006] For existing pedestrian detection technologies, most solutions still rely on classic target detection algorithms to detect pedestrian targets in scenes. Based on the detection ideas, they can be divided into single-stage detection algorithms represented by the YOLO series network model and two-stage detection algorithms represented by the Faster R-CNN network model. However, most classic target detection algorithms lack specific considerations for detection scenarios, especially in complex crowd-dense scenes. The robustness of these mainstream detection algorithms is affected, mainly including the following two points: (1) The different scales of pedestrian targets affect the overall performance of the detector. Since most of the current pedestrian detection data sets are obtained based on camera shooting and calibration, and the camera has the rule of "large near and small far" when shooting, the overall scale of pedestrian targets at a closer perspective is larger, and the overall scale of pedestrian targets at a farther perspective is smaller. Since the resolution of small-scale pedestrian targets is relatively low, the algorithm is prone to problems such as limited learned feature information or weak feature expression ability during feature extraction, making it difficult to have high sensitivity for pedestrian targets of different scales, which can easily lead to missed detection or false detection.

[0007] (2) Pedestrian targets are seriously occluded, which affects the overall performance of the detector. In pedestrian detection tasks in complex and crowded scenes, pedestrian targets are often occluded to a certain extent. By analyzing the images in the pedestrian detection dataset, it can be seen that the pedestrian occlusion problem mainly includes two situations: intra-class occlusion and inter-class occlusion. Among them, intra-class occlusion refers to the mutual occlusion between pedestrian targets. Inter-class occlusion refers to the interference of pedestrians by background information, which mainly includes buildings, trees, vehicles, items carried by pedestrians themselves, and items carried by other nearby pedestrians. Intra-class occlusion and inter-class occlusion lead to a decrease in the proportion of the visible area of ​​the pedestrian's whole body, which will cause difficulties in the feature extraction link of the algorithm, and the information required for the algorithm detection module to infer is also reduced. In addition, it will affect the accuracy of pedestrian target positioning, thereby affecting the overall performance of the pedestrian detection algorithm. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a multi-scale pedestrian detection method and device based on the combination of head and overall information, which improves the detection accuracy of the detector for multi-scale pedestrian targets and occluded pedestrian targets in complex crowded scenes and reduces its missed detection rate.

[0009] In a first aspect, the present invention provides a multi-scale pedestrian detection method based on the combination of head and overall information, comprising: Build the Faster R-CNN network model; The backbone network of the Faster R-CNN network model is integrated with the improved feature extraction network, and the image to be detected is input into the Faster R-CNN model integrated with the improved feature extraction network to extract features, thereby obtaining an extracted feature map; The sampling method of the region proposal network is improved, and a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination is constructed. By calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, the region proposal network is used to simultaneously generate head candidate boxes and overall candidate box sets for all pedestrian instances in the scene. Constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result through the pedestrian head detection branch module and the pedestrian overall detection branch module, wherein the preliminary target detection result includes a pedestrian head detection frame and a pedestrian overall detection frame; Post-processing the obtained pedestrian head detection frame and the pedestrian overall detection frame, and filtering out redundant detection frames generated in the joint detection process to obtain a final pedestrian detection result; In order to further suppress false detection and missed detection in the post-processing stage, a joint loss function is constructed in the loss function part. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate.

[0010] As a preferred solution, the backbone network of the Faster R-CNN network model is fused with an improved feature extraction network, and the image to be detected is input into the Faster R-CNN model fused with the improved feature extraction network for feature extraction to obtain an extracted feature map, including: Use the ResNet50 network as the backbone network of the Faster R-CNN network model; Using the backbone network to learn feature information of the image to be detected; The feature information obtained by the backbone network is fused, and the feature splicing strategy of the feature pyramid network FPN is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of output feature information, absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and then the prediction value corresponding to each layer of scale features is ~ Specifically as shown in formula 1 and formula 2; (1) (2) in, are weight parameters, is the feature mapping function, ~ The scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training.

[0011] As a preferred solution, the sampling method of the region proposal network is improved, and a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination is constructed. By calculating the occlusion overlap rate of each sample in the sample set and giving a higher weight to samples with a higher occlusion overlap rate, the region proposal network is used to simultaneously generate head candidate boxes and overall candidate box sets for all pedestrian instances in the scene, including: The non-uniform difficult sample mining strategy based on occlusion overlap ratio judgment introduces a judgment threshold , the sample set is divided into two groups according to the average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories, the probability of each sample being drawn is defined as , the specific calculation method is shown in formula 3; (3) in, Indicates The occlusion coefficient of the sample is used to reflect the The degree to which each sample is occluded; For The occlusion coefficient of samples , further expressed as: (4) in, Indicates the number of samples required for testing. represents the total number of candidate samples, and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set are used to determine the threshold Different values ​​can be set according to the overall occlusion level of the data set; For the sample set The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , further expressed as: (5) (6) As a preferred solution, the obtained pedestrian head detection frame and the pedestrian overall detection frame are post-processed, and redundant detection frames generated in the joint detection process are screened out to obtain the final pedestrian detection result, including: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively. By gradually reducing the confidence score of the overlapping detection frame, the competition between the overlapping frames is reduced without excessively suppressing them. The score of the overall pedestrian detection box and The score of head detection boxes It is expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set; The confidence scores of the head frame and the overall frame are weighted summed to obtain the joint confidence score. , specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or the overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

[0012] As a preferred solution, in order to further suppress the false detection and missed detection in the post-processing link, a joint loss function is constructed in the loss function part, and the joint loss function is used to punish the false detection that is not suppressed by the correct non-maximum value and the missed detection that is suppressed by the wrong non-maximum value, so that the final detection result is more accurate, including: The joint loss function Specifically expressed as: (12) Among them, the coefficient , , and are the weights of the balanced loss, and It is used to bring the falsely detected head frame and the overall frame closer for elimination. and It is used to push away the missed head frame and the overall frame that are suppressed by the erroneous non-maximum value, so that the head frame and the overall frame correctly correspond to the target frame; For the missed head frame and the overall frame that are used to suppress the erroneous non-maximum value, and The loss function is further expressed as: (13) (14) For the purpose of bringing the falsely detected head frame and the overall frame closer together for elimination and The loss function is further expressed as: (15) (16) in, and are the NMS thresholds corresponding to the head detection branch and the overall detection branch respectively. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

[0013] In a second aspect, the present invention provides a multi-scale pedestrian detection device based on the combination of head and overall information, comprising: Construction unit, used to build Faster R-CNN network model; An extraction unit is used to fuse the backbone network of the Faster R-CNN network model with the improved feature extraction network, and input the image to be detected into the Faster R-CNN model fused with the improved feature extraction network to perform feature extraction, so as to obtain an extracted feature map; A generation unit is used to improve the sampling method of the region proposal network, construct a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination, calculate the occlusion overlap rate of each sample in the sample set, and give a higher weight to the sample with a higher occlusion overlap rate, and use the region proposal network to simultaneously generate a head candidate box and a whole candidate box set for all pedestrian instances in the scene; A preliminary detection unit, used to construct a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtain preliminary target detection results through the pedestrian head detection branch module and the pedestrian overall detection branch module, wherein the preliminary target detection results include a pedestrian head detection frame and a pedestrian overall detection frame; A post-processing unit, used to post-process the obtained pedestrian head detection frame and the pedestrian whole detection frame, and filter out redundant detection frames generated in the joint detection process to obtain a final pedestrian detection result; The output unit is used to further suppress false detection and missed detection in the post-processing link. A joint loss function is constructed in the loss function part. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate.

[0014] As a preferred solution, the extraction unit is specifically used for: Use the ResNet50 network as the backbone network of the Faster R-CNN network model; Using the backbone network to learn feature information of the image to be detected; The feature information obtained by the backbone network is fused, and the feature splicing strategy of the feature pyramid network FPN is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of output feature information, absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and then the prediction value corresponding to each layer of scale features is ~ The details are shown in formula 1 and formula 2; (1) (2) in, are weight parameters, is the feature mapping function, ~ The scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training.

[0015] As a preferred solution, the generating unit is specifically used for: The non-uniform difficult sample mining strategy based on occlusion overlap ratio judgment introduces a judgment threshold , the sample set is divided according to the average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories, the probability of each sample being drawn is defined as , the specific calculation method is shown in formula 3; (3) in, Indicates The occlusion coefficient of the sample is used to reflect the The degree to which each sample is occluded; For The occlusion coefficient of samples , further expressed as: (4) in, Indicates the number of samples required for testing. represents the total number of candidate samples, and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set are used to determine the threshold Different values ​​can be set according to the overall occlusion level of the data set; For the sample set The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , further expressed as: (5) (6) As a preferred solution, the post-processing unit is specifically used for: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively. By gradually reducing the confidence score of the overlapping detection frame, the competition between the overlapping frames is reduced without excessively suppressing them. The score of the overall pedestrian detection box and The score of head detection boxes It is expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set; The confidence scores of the head frame and the overall frame are weighted summed to obtain the joint confidence score. , specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or the overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

[0016] As a preferred solution, the output unit is specifically used for: The joint loss function Specifically expressed as: (12) Among them, the coefficient , , and are the weights of the balanced loss, and It is used to bring the falsely detected head frame and the overall frame closer for elimination. and It is used to push away the missed head frame and the overall frame that are suppressed by the erroneous non-maximum value, so that the head frame and the overall frame correctly correspond to the target frame; For the missed head frame and the overall frame that are used to suppress the erroneous non-maximum value, and The loss function is further expressed as: (13) (14) For the purpose of bringing the falsely detected head frame and the overall frame closer together for elimination and The loss function is further expressed as: (15) (16) in, and are the NMS thresholds corresponding to the head detection branch and the overall detection branch respectively. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

[0017] Compared with the prior art, the present invention can achieve the following beneficial effects: In an embodiment of the present invention, a multi-scale pedestrian detection method and device based on the combination of head and overall information are proposed. First, by densely connecting features at different levels, that is, making all scale features participate in the calculation of the final output features, each level of features finally obtained can cover semantic information and detail texture information, thereby improving the network's sensitivity to multi-scale pedestrian targets; secondly, the present invention optimizes the sampling method of the region proposal network, by calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, thereby strengthening the learning of severely occluded samples in the difficult sample set during training, thereby improving the model's detection ability for occluded pedestrian targets; then, the present invention constructs a joint detection framework for pedestrian heads and overall information, aiming to use head detection to assist pedestrian detection, thereby reducing the adverse effects of pedestrian body occlusion on detection. In addition, the post-processing link and loss function module are optimized to reduce the interference of adjacent pedestrian targets on detection, and at the same time improve the intelligence and rationality of screening out redundant frames, so that the detection frames in dense areas are not overly suppressed while eliminating the false detection frames generated by the two detection branches, thereby further reducing the missed detection rate and false detection rate of pedestrian detection. Therefore, compared with previous algorithms, the pedestrian detection algorithm provided by the present invention has a stronger detection capability for multi-scale pedestrian targets and occluded pedestrian targets in complex crowd-dense scenes, and can reduce the missed detection rate of pedestrian targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of a multi-scale pedestrian detection method based on the combination of head and overall information provided according to an embodiment of the present invention.

[0019] Figure 2 It is an overall logical schematic diagram of a multi-scale pedestrian detection method based on the combination of head and overall information provided according to an embodiment of the present invention.

[0020] Figure 3 3 is a schematic diagram of the structure of the ResNet50 network in the multi-scale pedestrian detection method based on the combination of head and overall information provided according to an embodiment of the present invention.

[0021] Figure 4 It is a feature extraction network structure diagram in a multi-scale pedestrian detection method based on the combination of head and overall information provided according to an embodiment of the present invention.

[0022] Figure 5 It is a pseudo code flow chart of a joint post-processing algorithm in a multi-scale pedestrian detection method based on joint head and overall information provided according to an embodiment of the present invention.

[0023] Figure 6This is a comprehensive performance comparison chart of the multi-scale pedestrian detection method based on the combination of head and overall information provided according to an embodiment of the present invention and some mainstream detection algorithms on the CrowdHuman dataset.

[0024] Figure 7 This is a comprehensive performance comparison chart of the multi-scale pedestrian detection method based on the combination of head and overall information provided according to an embodiment of the present invention and some mainstream detection algorithms on the CityPersons dataset.

[0025] Figure 8a This is a performance comparison result of the multi-scale pedestrian detection method based on the combination of head and overall information provided by an embodiment of the present invention and some mainstream detection algorithms in the TJU-Ped-campus subset.

[0026] Figure 8b This is the performance comparison result of the multi-scale pedestrian detection method based on the combination of head and overall information provided by an embodiment of the present invention and some mainstream detection algorithms in the TJU-Ped-traffic subset.

[0027] Fig. 9 4 is a structural block diagram of a multi-scale pedestrian detection device based on the combination of head and overall information provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0030] Combination Figure 1 As shown, an embodiment of the present invention provides a multi-scale pedestrian detection method based on the combination of head and overall information, including: S101. Build Faster R-CNN network model; S102, integrating the backbone network of the Faster R-CNN network model with the improved feature extraction network, and inputting the image to be detected into the Faster R-CNN model integrated with the improved feature extraction network to perform feature extraction, thereby obtaining an extracted feature map; S103, improving the sampling method of the region proposal network, constructing a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination, calculating the occlusion overlap rate of each sample in the sample set, and assigning a higher weight to samples with a higher occlusion overlap rate, and using the region proposal network to simultaneously generate head candidate boxes and overall candidate box sets for all pedestrian instances in the scene; S104, constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result through the pedestrian head detection branch module and the pedestrian overall detection branch module, wherein the preliminary target detection result includes a pedestrian head detection frame and a pedestrian overall detection frame; S105, post-processing the obtained pedestrian head detection frame and the pedestrian whole detection frame, and filtering out redundant detection frames generated in the joint detection process to obtain a final pedestrian detection result; S106. In order to further suppress false detection and missed detection in the post-processing stage, a joint loss function is constructed in the loss function part. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate.

[0031] In S102, the backbone network of the Faster R-CNN network model is fused with the improved feature extraction network, and the image to be detected is input into the Faster R-CNN model fused with the improved feature extraction network for feature extraction to obtain an extracted feature map, including: Use the ResNet50 network as the backbone network of the Faster R-CNN network model; Using the backbone network to learn feature information of the image to be detected; The feature information obtained by the backbone network is fused, and the feature splicing strategy of the feature pyramid network FPN is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of output feature information, absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and then the prediction value corresponding to each layer of scale features is ~ The details are shown in formula 1 and formula 2; (1) (2) in, are weight parameters, is the feature mapping function, ~ The scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training.

[0032] Further, in S103, the sampling method of the region proposal network is improved, and a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination is constructed. By calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, the region proposal network is used to simultaneously generate a head candidate frame and an overall candidate frame set for all pedestrian instances in the scene, including: The non-uniform difficult sample mining strategy based on occlusion overlap ratio judgment introduces a judgment threshold , the sample set is divided into two groups according to the average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories, the probability of each sample being drawn is defined as , the specific calculation method is shown in formula 3; (3) in, Indicates The occlusion coefficient of the sample is used to reflect the The degree to which each sample is occluded; For The occlusion coefficient of samples , further expressed as: (4) in, Indicates the number of samples required for testing. represents the total number of candidate samples, and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set are used to determine the threshold Different values ​​can be set according to the overall occlusion level of the data set; For the sample set The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , further expressed as: (5) (6) Furthermore, in S105, the obtained pedestrian head detection frame and the pedestrian overall detection frame are post-processed, and redundant detection frames generated in the joint detection process are filtered out to obtain a final pedestrian detection result, including: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively. By gradually reducing the confidence score of the overlapping detection frame, the competition between the overlapping frames is reduced without excessively suppressing them. The score of the overall pedestrian detection box and The score of head detection boxes It is expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set; The confidence scores of the head frame and the overall frame are weighted summed to obtain the joint confidence score. , specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or the overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

[0033] Furthermore, in S106, in order to further suppress the false detection and missed detection in the post-processing link, a joint loss function is constructed in the loss function part, and the joint loss function is used to punish the false detection that is not suppressed by the correct non-maximum value and the missed detection that is suppressed by the wrong non-maximum value, so that the final detection result is more accurate, including: The joint loss function Specifically expressed as: (12) Among them, the coefficient , , and are the weights of the balanced loss, and It is used to bring the falsely detected head frame and the overall frame closer for elimination. and It is used to push away the missed head frame and the overall frame that are suppressed by the erroneous non-maximum value, so that the head frame and the overall frame correctly correspond to the target frame; For the missed head frame and the overall frame that are used to suppress the erroneous non-maximum value, and The loss function is further expressed as: (13) (14) For the purpose of bringing the falsely detected head frame and the overall frame closer together for elimination and The loss function is further expressed as: (15) (16) in, and are the NMS thresholds corresponding to the head detection branch and the overall detection branch respectively. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

[0034] In an embodiment of the present invention, a multi-scale pedestrian detection method based on the combination of head and overall information is proposed. First, by densely connecting features at different levels, that is, making all scale features participate in the calculation of the final output features, each level of features finally obtained can cover semantic information and detailed texture information, thereby improving the network's sensitivity to multi-scale pedestrian targets; secondly, the present invention optimizes the sampling method of the region proposal network, by calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, thereby strengthening the learning of severely occluded samples in the difficult sample set during training, thereby improving the model's detection ability for occluded pedestrian targets; then, the present invention constructs a joint detection framework for pedestrian heads and overall information, aiming to use head detection to assist pedestrian detection, thereby reducing the adverse effects of pedestrian body occlusion on detection. In addition, the post-processing link and loss function module are optimized to reduce the interference of adjacent pedestrian targets on detection, and at the same time improve the intelligence and rationality of screening out redundant frames, so that the detection frames in dense areas are not overly suppressed while eliminating the false detection frames generated by the two detection branches, thereby further reducing the missed detection rate and false detection rate of pedestrian detection. Therefore, compared with previous algorithms, the pedestrian detection algorithm provided by the present invention has a stronger detection capability for multi-scale pedestrian targets and occluded pedestrian targets in complex crowd-dense scenes, and can reduce the missed detection rate of pedestrian targets.

[0035] Combination Figure 2 , 3, 4, and 5, the present invention aims at the problem that the existing target detection algorithm has a reduced detection accuracy and a high missed detection rate for small-scale pedestrian targets and occluded pedestrian targets in complex and dense scenes. In order to further facilitate the understanding of the scheme of the present invention, a multi-scale pedestrian detection method based on the combination of head and overall information under an embodiment of the present invention is described below. The overall method flow chart is shown in FIG. Figure 2 As shown, the specific steps include: Step 1: Build the Faster R-CNN network model; Step 2: Fuse the backbone network of the Faster R-CNN network model with the improved feature extraction network, and input the image to be detected into the Faster R-CNN network model fused with the improved feature extraction network for feature extraction to obtain the extracted feature map; Step 2.1: Use the ResNet50 network as the backbone network of the Faster R-CNN network model. The structure diagram of the ResNet50 network is as follows: Figure 3 As shown; Step 2.2: Fuse the improved feature extraction network with the Faster R-CNN network model. The overall structure of the improved feature extraction network is as follows: Figure 4 The specific steps are as follows: Step 2.2.1: Use the backbone network ResNet50 to learn the feature information of the image to be detected; Step 2.2.2: Feature fusion of the feature information obtained by the backbone network. Specifically, the feature splicing strategy of the Feature Pyramid Network (FPN) is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of the output feature information, thereby absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and the prediction value corresponding to each layer of scale features. ~ The details are shown in formula (1) and formula (2).

[0036] (1) (2) in, are weight parameters, is the feature mapping function, ~ It is the scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training. This method enables the model to learn which connections help the detection module to make predictions and which connections are invalid calculations, thereby improving the performance of the feature extraction network.

[0037] Step 3: Improve the sampling method of the Region Proposal Network (RPN) and construct a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination. By calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, the learning of severely occluded samples in the difficult sample set is strengthened during training, thereby improving the model's ability to detect occluded pedestrian targets. Then use the region proposal network to simultaneously generate head candidate boxes and overall candidate box sets for all pedestrian instances in the scene.

[0038] Step 3.1: For the non-uniform difficult sample mining strategy based on occlusion overlap ratio discrimination constructed in the present invention, by introducing the judgment threshold , the sample set is divided according to its average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories. Then the probability of each sample being drawn is defined as , the specific calculation method is shown in formula (3).

[0039] (3) in, Indicates The occlusion coefficient of the sample can reflect the The degree to which a sample is occluded.

[0040] Step 3.2: For the The occlusion coefficient of samples , which can be further expressed as: (4) in, Indicates the number of samples required for testing. Represents the total number of candidate samples. and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set, the threshold Different values ​​can be set depending on the overall occlusion level of the dataset.

[0041] Step 3.3: For the sample set in step 3.2 The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , which can be further expressed as: (5) (6) Step 4: Construct the pedestrian head detection branch and the pedestrian overall detection branch, and obtain preliminary target detection results through the pedestrian head detection branch module and the pedestrian overall detection branch module.

[0042] Step 5: Post-process the obtained pedestrian head detection frame and pedestrian overall detection frame to filter out redundant detection results generated during the joint detection process. The overall process pseudo code is as follows: Figure 5 The specific steps are as follows: Step 5.1: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively, and the confidence score of the overlapping detection frame is gradually reduced, thereby reducing its competitiveness without overly suppressing the overlapping frame. The score of the overall pedestrian detection box and The score of head detection boxes It can be expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set.

[0043] Step 5.2: Take the weighted sum of the confidence scores of the head box and the overall box obtained in step 5.1 to get the joint confidence score , which can be specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

[0044] Step 6: For the false detection and missed detection in the non-maximum suppression link, the present invention constructs a joint loss function in the loss function part , which aims to penalize false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima. Specifically expressed as: (12) Among them, the coefficient , , and are the weights of balancing losses. and It is used to “pull closer” the falsely detected head frame and the overall frame to facilitate its removal. and It is used to "push away" the missed head frame and the overall frame that have been suppressed by the erroneous non-maximum value so that they can correctly correspond to the target frame.

[0045] Step 6.1: Push away the missed head frame and the overall frame for suppressing the erroneous non-maximum value and The loss function can be further expressed as: (13) (14) Step 6.2: For the false detection of the head frame and the overall frame "pulling" closer to facilitate its removal and The loss function can be further expressed as: (15) (16) in, and They are the NMS thresholds corresponding to the head detection branch and the overall detection branch. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

[0046] The technical solution designed by the present invention can be well applied to the fields of intelligent monitoring, assisted safe driving, video monitoring of large venues, intelligent robots, etc.

[0047] The effectiveness of the present invention can be verified by the following experiments: 1. Experimental conditions and contents 1. The ImageNet dataset is used as the pre-training dataset for the pedestrian detection model, and then the pedestrian detection model is trained based on the CrowdHuman, CityPersons and TJU-DHD-pedestrian datasets. During the training process, some enhancement strategies are used for the dataset, including random cropping, random horizontal flipping, and random interference of image brightness. In the actual training process, the stochastic gradient descent optimizer is used for 35 epochs of training, the momentum factor momentum is set to 0.9, the learning rate adopts the warm up strategy, and the initial learning rate is set to a lower value. In this invention, it is set to , from the 1st iteration to the 800th iteration of each epoch, the learning rate increases linearly to , and then remain unchanged. By the 24th epoch, the learning rate was reduced to 10% of the original, and by the 28th epoch, the learning rate became 1% of the original.

[0048] 2. Use Average Precision (AP) and Logarithmic Mean Missed Detection Rate (MR) -2 ) is used as an evaluation index to evaluate the comprehensive performance of the method involved in the present invention. The specific calculation method of the average precision and the logarithmic average missed detection rate can be expressed as:

[0049]

[0050] in, It represents the accuracy, which is the ratio of the number of samples predicted as pedestrian targets to the number of samples actually marked as pedestrian targets in the image; Recall is the ratio of the number of correctly predicted pedestrian targets to all predicted results in the process of pedestrian target detection. represents the missed detection rate, Indicates the average number of false positives in each image. The higher the AP value, the higher the MR -2 The lower the value, the better the detection performance of the algorithm.

[0051] (II) Experimental results Depend on Figure 6 , 7As shown in Figures 8 and 8, on the CrowdHuman, CityPersons and TJU-DHD-pedestrian pedestrian detection datasets, compared with some popular pedestrian detection algorithms and the baseline Faster R-CNN algorithm before improvement, the average detection accuracy of the present invention is higher and the logarithmic average missed detection rate is lower. Therefore, the experimental results fully demonstrate that the present invention has better performance than some popular pedestrian detection algorithms and can better detect multi-scale pedestrian targets and severely occluded pedestrian targets.

[0052] Compared with the prior art, the multi-scale pedestrian detection method provided by the present invention has the following characteristics: 1. Compared with the existing YOLO series network models and R-CNN series network models, the method involved in the present invention uses a strategy of densely connecting features at different levels and allowing all scale features to participate in the calculation of the final output features in the feature extraction link, so that each level of features finally obtained can cover semantic information and detailed texture information, thereby improving the network's detection ability for multi-scale pedestrian targets. From the perspective of information flow, the feature extraction network constructed by the present invention can not only learn rich and detailed multi-scale features, but also integrate deep semantic information into shallow features, and shallow detailed texture information into deep features, which is more helpful to improve the network's sensitivity to multi-scale pedestrian targets. From the perspective of parameter optimization, when a dense connection strategy is used to fuse multi-scale features, in the actual training stage, the gradient can be back-propagated more efficiently, thereby accelerating the convergence speed of the network model without increasing the complexity of the network too much, which is more helpful to obtain a high-quality global optimal solution.

[0053] 2. Compared with the existing R-CNN series network models, the present invention optimizes the sampling method of the region proposal network. By calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, the learning of severely occluded samples in the difficult sample set is strengthened during training. As the training continues, the model will gradually improve its ability to detect occluded pedestrian targets, thereby showing higher performance in complex and crowded scenes.

[0054] 3. Compared with the existing YOLO series network models and R-CNN series network models, the method involved in the present invention designs a head and whole body dual detection branch in the core part of the model for joint detection, thereby making full use of head detection to assist pedestrian detection and reduce the impact of pedestrian body occlusion on detector performance.

[0055] 4. Compared with the existing YOLO series network models and R-CNN series network models, the method involved in the present invention optimizes the post-processing link and the loss function module, reduces the interference of adjacent pedestrian targets on detection, and improves the intelligence and rationality of screening out redundant frames, so that the detection frames in dense areas are not overly suppressed and the false detection frames generated by the two detection branches are eliminated, thereby further reducing the missed detection rate and false detection rate of pedestrian detection.

[0056] Accordingly, combined Fig. 9 As shown, in an embodiment of the present invention, a multi-scale pedestrian detection device based on the combination of head and overall information is provided, comprising: A construction unit 901 is used to construct a Faster R-CNN network model; An extraction unit 902 is used to fuse the backbone network of the Faster R-CNN network model with the improved feature extraction network, and input the image to be detected into the Faster R-CNN model fused with the improved feature extraction network to perform feature extraction, thereby obtaining an extracted feature map; A generating unit 903 is used to improve the sampling method of the region proposal network, construct a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination, calculate the occlusion overlap rate of each sample in the sample set, and assign a higher weight to the sample with a higher occlusion overlap rate, and use the region proposal network to simultaneously generate a head candidate frame and an overall candidate frame set for all pedestrian instances in the scene; A preliminary detection unit 904 is used to construct a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtain a preliminary target detection result through the pedestrian head detection branch module and the pedestrian overall detection branch module, wherein the preliminary target detection result includes a pedestrian head detection frame and a pedestrian overall detection frame; A post-processing unit 905 is used to post-process the obtained pedestrian head detection frame and the pedestrian whole detection frame, and filter out redundant detection frames generated in the joint detection process to obtain a final pedestrian detection result; The output unit 906 is used to construct a joint loss function in the loss function part to further suppress false detection and missed detection in the post-processing link. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate.

[0057] As a preferred solution, the extraction unit 902 is specifically used for: Use the ResNet50 network as the backbone network of the Faster R-CNN network model; Using the backbone network to learn feature information of the image to be detected; The feature information obtained by the backbone network is fused, and the feature splicing strategy of the feature pyramid network FPN is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of output feature information, absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and then the prediction value corresponding to each layer of scale features is ~ Specifically as shown in formula 1 and formula 2; (1) (2) in, are weight parameters, is the feature mapping function, ~ The scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training.

[0058] As a preferred solution, the generating unit 903 is specifically used for: The non-uniform difficult sample mining strategy based on occlusion overlap ratio judgment introduces a judgment threshold , the sample set is divided into two groups according to the average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories, the probability of each sample being drawn is defined as , the specific calculation method is shown in formula 3; (3) in, Indicates The occlusion coefficient of the sample is used to reflect the The degree to which each sample is occluded; For The occlusion coefficient of samples , further expressed as: (4) in, Indicates the number of samples required for testing. represents the total number of candidate samples, and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set are used to determine the threshold Different values ​​can be set according to the overall occlusion level of the data set; For the sample set The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , further expressed as: (5) (6) As a preferred solution, the post-processing unit 905 is specifically used for: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively. By gradually reducing the confidence score of the overlapping detection frame, the competition between the overlapping frames is reduced without excessively suppressing them. The score of the overall pedestrian detection box and The score of head detection boxes It is expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set; The confidence scores of the head frame and the overall frame are weighted summed to obtain the joint confidence score. , specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or the overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

[0059] As a preferred solution, the output unit 906 is specifically used for: The joint loss function Specifically expressed as: (12) Among them, the coefficient , , and are the weights of the balanced loss, and It is used to bring the falsely detected head frame and the overall frame closer for elimination. and It is used to push away the missed head frame and the overall frame that are suppressed by the erroneous non-maximum value, so that the head frame and the overall frame correctly correspond to the target frame; For the missed head frame and the overall frame that are used to suppress the erroneous non-maximum value, and The loss function is further expressed as: (13) (14) For the purpose of bringing the falsely detected head frame and the overall frame closer together for elimination and The loss function is further expressed as: (15) (16) in, and are the NMS thresholds corresponding to the head detection branch and the overall detection branch respectively. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

[0060] In an embodiment of the present invention, a multi-scale pedestrian detection device based on the combination of head and overall information is proposed. First, by densely connecting features at different levels, that is, making all scale features participate in the calculation of the final output features, each level of features finally obtained can cover semantic information and detailed texture information, thereby improving the network's sensitivity to multi-scale pedestrian targets; secondly, the present invention optimizes the sampling method of the region proposal network, by calculating the occlusion overlap rate of each sample in the sample set and giving a higher weight to the sample with a higher occlusion overlap rate, thereby strengthening the learning of severely occluded samples in the difficult sample set during training, thereby improving the model's detection ability for occluded pedestrian targets; then, the present invention constructs a joint detection framework for pedestrian heads and overall information, aiming to use head detection to assist pedestrian detection, thereby reducing the adverse effects of pedestrian body occlusion on detection. In addition, the post-processing link and loss function module are optimized to reduce the interference of adjacent pedestrian targets on detection, and at the same time improve the intelligence and rationality of screening out redundant frames, so that the detection frames in dense areas are not overly suppressed while eliminating the false detection frames generated by the two detection branches, thereby further reducing the missed detection rate and false detection rate of pedestrian detection. Therefore, compared with previous algorithms, the pedestrian detection algorithm provided by the present invention has a stronger detection capability for multi-scale pedestrian targets and occluded pedestrian targets in complex crowd-dense scenes, and can reduce the missed detection rate of pedestrian targets.

[0061] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0062] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-scale pedestrian detection method based on the combination of head and overall information, characterized in that: include: Build the Faster R-CNN network model; The backbone network of the Faster R-CNN network model is integrated with the improved feature extraction network, and the image to be detected is input into the Faster R-CNN model integrated with the improved feature extraction network to extract features, thereby obtaining an extracted feature map; The sampling method of the region proposal network is improved, and a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination is constructed. By calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, the region proposal network is used to simultaneously generate head candidate boxes and overall candidate box sets for all pedestrian instances in the scene. Constructing a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtaining a preliminary target detection result through the pedestrian head detection branch module and the pedestrian overall detection branch module, wherein the preliminary target detection result includes a pedestrian head detection frame and a pedestrian overall detection frame; Post-processing the obtained pedestrian head detection frame and the pedestrian overall detection frame, and filtering out redundant detection frames generated in the joint detection process to obtain a final pedestrian detection result; In order to further suppress false detection and missed detection in the post-processing stage, a joint loss function is constructed in the loss function part. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate.

2. The multi-scale pedestrian detection method based on the combination of head and overall information as claimed in claim 1, characterized in that: The backbone network of the Faster R-CNN network model is integrated with the improved feature extraction network, and the image to be detected is input into the Faster R-CNN model integrated with the improved feature extraction network for feature extraction to obtain an extracted feature map, including: Use the ResNet50 network as the backbone network of the Faster R-CNN network model; Using the backbone network to learn feature information of the image to be detected; The feature information obtained by the backbone network is fused, and the feature splicing strategy of the feature pyramid network FPN is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of output feature information, absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and then the prediction value corresponding to each layer of scale features is ~ The details are shown in formula 1 and formula 2; (1) (2) in, are weight parameters, is the feature mapping function, ~ The scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training.

3. The multi-scale pedestrian detection method based on the combination of head and overall information as claimed in claim 2, characterized in that: The sampling method of the region proposal network is improved, and a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination is constructed. By calculating the occlusion overlap rate of each sample in the sample set and assigning a higher weight to samples with a higher occlusion overlap rate, the region proposal network is used to simultaneously generate head candidate frames and overall candidate frame sets for all pedestrian instances in the scene, including: The non-uniform difficult sample mining strategy based on occlusion overlap ratio judgment introduces a judgment threshold , the sample set is divided into two groups according to the average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories, the probability of each sample being drawn is defined as , the specific calculation method is shown in formula 3; (3) in, Indicates The occlusion coefficient of the sample is used to reflect the The degree to which each sample is occluded; For The occlusion coefficient of samples , further expressed as: (4) in, Indicates the number of samples required for testing. represents the total number of candidate samples, and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set are used to determine the threshold Different values ​​can be set according to the overall occlusion level of the data set; For the sample set The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , further expressed as: (5) (6)。 4. The multi-scale pedestrian detection method based on the combination of head and overall information as claimed in claim 3, characterized in that: The obtained pedestrian head detection frame and the pedestrian overall detection frame are post-processed, and redundant detection frames generated in the joint detection process are screened out to obtain the final pedestrian detection result, including: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively. By gradually reducing the confidence score of the overlapping detection frame, the competition between the overlapping frames is reduced without excessively suppressing them. The score of the overall pedestrian detection box and The score of head detection boxes It is expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set; The confidence scores of the head frame and the overall frame are weighted summed to obtain the joint confidence score. , specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or the overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

5. The multi-scale pedestrian detection method based on the combination of head and overall information as claimed in claim 4, characterized in that: In order to further suppress false detection and missed detection in the post-processing link, a joint loss function is constructed in the loss function part. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate, including: The joint loss function Specifically expressed as: (12) Among them, the coefficient , , and are the weights of the balanced loss, and It is used to bring the falsely detected head frame and the overall frame closer for elimination. and It is used to push away the missed head frame and the overall frame that are suppressed by the erroneous non-maximum value, so that the head frame and the overall frame correctly correspond to the target frame; For the missed head frame and the overall frame that are used to suppress the erroneous non-maximum value, and The loss function is further expressed as: (13) (14) For the purpose of bringing the falsely detected head frame and the overall frame closer together for elimination and The loss function is further expressed as: (15) (16) in, and are the NMS thresholds corresponding to the head detection branch and the overall detection branch respectively. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

6. A multi-scale pedestrian detection device based on the combination of head and overall information, characterized in that: include: Construction unit, used to build Faster R-CNN network model; An extraction unit is used to fuse the backbone network of the Faster R-CNN network model with the improved feature extraction network, and input the image to be detected into the Faster R-CNN model fused with the improved feature extraction network to perform feature extraction, so as to obtain an extracted feature map; A generation unit is used to improve the sampling method of the region proposal network, construct a non-uniform difficult sample mining strategy based on occlusion overlap rate discrimination, calculate the occlusion overlap rate of each sample in the sample set, and give a higher weight to the sample with a higher occlusion overlap rate, and use the region proposal network to simultaneously generate a head candidate box and a whole candidate box set for all pedestrian instances in the scene; A preliminary detection unit, used to construct a pedestrian head detection branch module and a pedestrian overall detection branch module, and obtain preliminary target detection results through the pedestrian head detection branch module and the pedestrian overall detection branch module, wherein the preliminary target detection results include a pedestrian head detection frame and a pedestrian overall detection frame; A post-processing unit, used to post-process the obtained pedestrian head detection frame and the pedestrian whole detection frame, and filter out redundant detection frames generated in the joint detection process to obtain a final pedestrian detection result; The output unit is used to further suppress false detection and missed detection in the post-processing link. A joint loss function is constructed in the loss function part. The joint loss function is used to punish false detections that are not suppressed by correct non-maxima and missed detections that are suppressed by incorrect non-maxima, so that the final detection result is more accurate.

7. The multi-scale pedestrian detection device based on the combination of head and overall information as claimed in claim 6, characterized in that: The extraction unit is specifically used for: Use the ResNet50 network as the backbone network of the Faster R-CNN network model; Using the backbone network to learn feature information of the image to be detected; The feature information obtained by the backbone network is fused, and the feature splicing strategy of the feature pyramid network FPN is improved by combining the idea of ​​dense connection, that is, all scale features are involved in the calculation of output feature information, absorbing the advantages of large-scale feature maps expressing high-level semantic information and small-scale feature maps expressing underlying texture and other detail information, enhancing the network's fusion of multi-scale features, and then the prediction value corresponding to each layer of scale features is ~ The details are shown in formula 1 and formula 2; (1) (2) in, are weight parameters, is the feature mapping function, ~ The scale features of each layer obtained by the feature extraction network. In the specific training process, the parameters Participate in the back propagation of gradients and learn to update to the most appropriate value through model training.

8. The multi-scale pedestrian detection device according to claim 7, characterized in that: The generating unit is specifically used for: The non-uniform difficult sample mining strategy based on occlusion overlap ratio judgment introduces a judgment threshold , the sample set is divided into two groups according to the average occlusion overlap rate Divided into difficult sets ( ) and the ordinary set ( ) two categories, the probability of each sample being drawn is defined as , the specific calculation method is shown in formula 3; (3) in, Indicates The occlusion coefficient of the sample is used to reflect the The degree to which each sample is occluded; For The occlusion coefficient of samples , further expressed as: (4) in, Indicates the number of samples required for testing. represents the total number of candidate samples, and Respectively represent the first The occlusion overlap rate of samples and the average occlusion overlap rate of the entire sample set are used to determine the threshold Different values ​​can be set according to the overall occlusion level of the data set; For the sample set The occlusion overlap ratio of samples and the average occlusion overlap ratio of the entire sample set , further expressed as: (5) (6)。 9. The multi-scale pedestrian detection device based on the combination of head and overall information as claimed in claim 8, characterized in that: The post-processing unit is specifically used for: For the head detection frame and pedestrian overall detection frame obtained by the joint detection framework, a penalty factor of decreasing confidence is introduced respectively. By gradually reducing the confidence score of the overlapping detection frame, the competition between the overlapping frames is reduced without excessively suppressing them. The score of the overall pedestrian detection box and The score of head detection boxes It is expressed as: (7) (8) (9) (10) in, and are the overall frame and head frame with the highest scores in the previous iteration, and They are the NMS thresholds corresponding to overall detection and head detection, A small fixed value that is initially set; The confidence scores of the head frame and the overall frame are weighted summed to obtain the joint confidence score. , specifically expressed as: (11) in, Indicates the weight of the head detection score. If the head overlap or the overall overlap is greater than the preset threshold, the detection box with lower confidence will be suppressed.

10. The multi-scale pedestrian detection device based on the combination of head and overall information as claimed in claim 9, characterized in that: The output unit is specifically used for: The joint loss function Specifically expressed as: (12) Among them, the coefficient , , and are the weights of the balanced loss, and It is used to bring the falsely detected head frame and the overall frame closer for elimination. and It is used to push away the missed head frame and the overall frame that are suppressed by the erroneous non-maximum value, so that the head frame and the overall frame correctly correspond to the target frame; For the missed head frame and the overall frame that are used to suppress the erroneous non-maximum value, and The loss function is further expressed as: (13) (14) For the purpose of bringing the falsely detected head frame and the overall frame closer together for elimination and The loss function is further expressed as: (15) (16) in, and are the NMS thresholds corresponding to the head detection branch and the overall detection branch respectively. After removing the redundant frames generated by the head detection branch and the overall detection branch, the final pedestrian detection result is obtained.

Images