A human body detection method and device based on a top-mounted camera and a storage medium

Abstract

The present application belongs to the field of Internet of Things, and particularly relates to a human body detection method and device based on a top-mounted camera and a storage medium, the method comprising: acquiring a scene video stream collected by a top-mounted camera; inputting an image frame into a human body detection model to extract a plurality of basic feature maps of different scales; performing feature fusion through a feature pyramid network based on the plurality of basic feature maps to obtain a first intermediate feature map, a second intermediate feature map and a third intermediate feature map; generating a multi-scale first detection result through a first detection branch based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map; splicing and fusing the up-sampled first intermediate feature map with a shallow layer feature map in the plurality of basic feature maps, and generating a second detection result through a second detection branch; and fusing the first detection result and the second detection result to output a final human body detection frame. The method improves the recall rate and accuracy of human body detection.

Description

Technical Field

[0001] This invention relates to the field of the Internet of Things, and specifically to a method, device, and storage medium for human body detection based on a top-mounted camera. Background Technology

[0002] Currently, common security camera algorithms are mainly designed for side-mounted or downward-facing installations. From this traditional perspective, the camera can capture relatively complete and feature-rich images of the human body from the side or at an angle, which is beneficial for AI algorithms to identify and detect.

[0003] However, ceiling-mounted binocular cameras are installed indoors from a completely vertical downward angle, typically at heights between 6 and 15 meters. In this specific scenario, existing technologies exhibit significant shortcomings and defects, including: sparse human imaging features. Due to the vertical downward angle, the camera can only capture the top of the body (such as the head and shoulders), failing to acquire highly discriminative features like the face and torso. As the installation height increases (e.g., 12-15 meters), the human body appears as a small target in the image, with an extremely low pixel ratio, resulting in very limited visual information available for recognition. Furthermore, in crowded places (such as shopping malls and train stations), severe occlusion and adhesion between people occur. Existing algorithms struggle to cope with extreme cases where some features are further obscured, given the already limited human features, leading to a significantly higher false negative rate.

[0004] Given the challenges mentioned above, directly applying existing general-purpose target detection algorithms designed for side-mounted views (such as the standard YOLO series algorithms) makes it difficult to simultaneously guarantee high recall and accuracy in top-mounted, high-view scenarios, and thus fails to meet the commercial needs for precise statistics and attribute analysis.

[0005] Therefore, there is an urgent need for a human detection method that can effectively handle complex situations such as high installation, top-down viewing angle, small target, and multiple obstructions. Summary of the Invention

[0006] To improve the recall and accuracy of human body detection, in a first aspect, the present invention provides a human body detection method based on a top-mounted camera, the method comprising:

[0007] Acquire scene video streams captured by top-mounted cameras;

[0008] Input image frames from the scene video stream into the human detection model to extract basic feature maps at multiple different scales;

[0009] Based on multiple basic feature maps, feature fusion is performed through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map have successively decreasing resolutions and successively increasing number of channels;

[0010] Based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map, a multi-scale first detection result is generated through the first detection branch. The first detection result includes a first human body detection box based on the anchor box and its confidence.

[0011] The first intermediate feature map after upsampling is spliced ​​and fused with the shallow feature map in the multiple basic feature maps, and a second detection result is generated through the second detection branch. The second detection result includes a heat map containing the confidence of the human body center point.

[0012] The first detection result and the second detection result are combined to output the final human body detection box.

[0013] In one possible implementation, the step of fusing features based on multiple base feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map includes:

[0014] The fifth basic feature map is selected from multiple basic feature maps. This fifth basic feature map is then processed by spatial pyramid pooling, convolution, batch normalization, and nonlinear activation to obtain the third intermediate feature map. The upsampled third intermediate feature map is then concatenated with the second-to-last fourth basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention, convolution, batch normalization, and nonlinear activation to obtain the second intermediate feature map. Finally, the upsampled second intermediate feature map is concatenated with the third-to-last third basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention and convolution to obtain the first intermediate feature map.

[0015] In one possible implementation, the step of concatenating and fusing the upsampled first intermediate feature map with shallow feature maps from multiple base feature maps, and generating a second detection result through a second detection branch, includes:

[0016] The first intermediate feature map after upsampling is concatenated with the second basic feature map obtained from the second of multiple basic feature maps, and the concatenation result is subjected to feature extraction to obtain a heat map containing the confidence of the human body center point.

[0017] The step of generating a multi-scale first detection result based on the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map through a first detection branch includes:

[0018] After convolution and activation of the first intermediate feature map, the first feature map is obtained;

[0019] The first intermediate feature map after feature extraction is concatenated with the second intermediate feature map, and the concatenation result is subjected to linear spatial attention and feature extraction to obtain the second preliminary feature map. The second preliminary feature map is then convolved and activated to obtain the second feature map.

[0020] The second preliminary feature map after feature extraction is concatenated with the third intermediate feature map, and the concatenation result is subjected to linear spatial attention and feature extraction to obtain the third preliminary feature map. The third preliminary feature map is then convolved and activated to obtain the third feature map.

[0021] The first feature map, the second feature map, and the third feature map are merged, and the location of the first human detection box and its confidence score at multiple scales are output based on the merged result.

[0022] In one possible implementation, the heatmap containing the confidence level of the human body center point is generated using the following two-dimensional Gaussian kernel function formula:

[0023] , ;

[0024] Where (x, y) are the coordinates of a point on the heatmap, (c x c y () represents the coordinates of the center point of the actual human detection frame. is the standard deviation, which controls the width of the distribution; R is the Gaussian kernel radius.

[0025] R = min(r1, r2, r3);

[0026] Where r1, r2, and r3 are obtained by solving the following equations:

[0027] ;

[0028] ;

[0029] ;

[0030] Where min overlap To set the threshold hyperparameter, w is the width of the real human detection box, and h is the height of the real human detection box.

[0031] In one possible implementation, the method further includes:

[0032] The local maximum value of the heatmap containing the confidence of the human body center point is found by using a two-dimensional plane search algorithm;

[0033] The confidence level of the spread from the center point corresponding to the local maximum value to the center point of the human body is less than the preset confidence level.

[0034] The boundary of the second human detection frame is constructed using the diffusion boundary.

[0035] In one possible implementation, the step of fusing the first detection result and the second detection result to output the final human detection box includes:

[0036] The local maximum value is used as the confidence level of the second human detection box;

[0037] Based on the confidence scores, the first human detection box and the second human detection box are sorted in descending order to obtain a candidate list;

[0038] Select the first human detection box in the candidate list as the final human detection box, and calculate the intersection-union ratio (IUU) of the final human detection box with each human detection box in the candidate list except itself.

[0039] Human detection boxes with an intersection-union ratio exceeding a preset intersection-union ratio are removed from the candidate list;

[0040] Return to the steps of selecting the first human detection box in the candidate list as the final human detection box, and calculating the intersection-union ratio (IUU) of the final human detection box with each human detection box in the candidate list except itself, until the candidate list is empty.

[0041] In one possible implementation, the training process of the human detection model includes:

[0042] Based on the center point positions of every two ground truth bounding boxes on the original training image, calculate the Euclidean distance and intersection-union ratio of every two ground truth bounding boxes;

[0043] When the intersection-union ratio is greater than 0 or the Euclidean distance is less than the preset distance value, any one of the real annotation boxes will be blacked out and deleted.

[0044] The human detection model is trained using the original training images after blacking out and deleting the ground truth bounding boxes.

[0045] Secondly, the present invention provides a human body detection device based on a top-mounted camera, the device comprising:

[0046] The acquisition module is used to acquire scene video streams captured by the top-mounted camera;

[0047] The basic feature map extraction module is used to input image frames from the scene video stream into the human detection model and extract basic feature maps at multiple different scales.

[0048] The intermediate feature map acquisition module is used to perform feature fusion based on multiple basic feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map have successively decreasing resolutions and successively increasing number of channels;

[0049] The first detection result acquisition module is used to generate a multi-scale first detection result based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map through the first detection branch. The first detection result includes a first human body detection box based on the anchor box and its confidence.

[0050] The second detection result acquisition module is used to splice and fuse the upsampled first intermediate feature map with the shallow feature map in the multiple basic feature maps, and generate a second detection result through the second detection branch. The second detection result includes a heat map containing the confidence of the human body center point.

[0051] The final human detection box acquisition module is used to fuse the first detection result and the second detection result to output the final human detection box.

[0052] In one possible implementation, the intermediate feature map acquisition module is specifically used for:

[0053] The fifth basic feature map is selected from multiple basic feature maps. This fifth basic feature map is then processed by spatial pyramid pooling, convolution, batch normalization, and nonlinear activation to obtain the third intermediate feature map. The upsampled third intermediate feature map is then concatenated with the second-to-last fourth basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention, convolution, batch normalization, and nonlinear activation to obtain the second intermediate feature map. Finally, the upsampled second intermediate feature map is concatenated with the third-to-last third basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention and convolution to obtain the first intermediate feature map.

[0054] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described human detection methods based on a top-mounted camera.

[0055] The human detection method based on a top-mounted camera provided in this application includes: acquiring a scene video stream captured by the top-mounted camera; inputting image frames of the scene video stream into a human detection model to extract multiple basic feature maps at different scales; fusing features based on the multiple basic feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map have progressively decreasing resolutions and progressively increasing channel numbers; generating a multi-scale first detection result based on the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map through a first detection branch, the first detection result including a first human detection box based on anchor boxes and its confidence; concatenating and fusing the upsampled first intermediate feature map with shallow feature maps from the multiple basic feature maps, and generating a second detection result through a second detection branch, the second detection result including a heatmap containing the confidence of the human center point; fusing the first detection result and the second detection result to output the final human detection box. This method improves the recall and accuracy of human detection. Attached Figure Description

[0056] Figure 1 A schematic flowchart of a human body detection method based on a top-mounted camera provided in an embodiment of this application;

[0057] Figure 2 A schematic diagram of a heat map provided in an embodiment of this application;

[0058] Figure 3 The original training image provided in this application embodiment has been blackened and the ground truth bounding boxes have been removed;

[0059] Figure 4 This is a schematic diagram of the network structure of the human detection model provided in an embodiment of this application. Detailed Implementation

[0060] The present invention will be described in detail below through embodiments.

[0061] Currently, common security camera algorithms are mainly designed for side-mounted or downward-facing installations. From this traditional perspective, the camera can capture relatively complete and feature-rich images of the human body from the side or at an angle, which is beneficial for AI algorithms to identify and detect.

[0062] However, ceiling-mounted binocular cameras are installed indoors from a completely vertical downward angle, typically at heights between 6 and 15 meters. In this specific scenario, existing technologies exhibit significant shortcomings and defects, including: sparse human imaging features. Due to the vertical downward angle, the camera can only capture the top of the body (such as the head and shoulders), failing to acquire highly discriminative features like the face and torso. As the installation height increases (e.g., 12-15 meters), the human body appears as a small target in the image, with an extremely low pixel ratio, resulting in very limited visual information available for recognition. Furthermore, in crowded places (such as shopping malls and train stations), severe occlusion and adhesion between people occur. Existing algorithms struggle to cope with extreme cases where some features are further obscured, given the already limited human features, leading to a significantly higher false negative rate.

[0063] Given the challenges mentioned above, directly applying existing general-purpose target detection algorithms designed for side-mounted views (such as the standard YOLO series algorithms) makes it difficult to simultaneously guarantee high recall and accuracy in top-mounted, high-view scenarios, and thus fails to meet the commercial needs for precise statistics and attribute analysis.

[0064] Therefore, there is an urgent need for a human detection method that can effectively handle complex situations such as high installation, top-down viewing angle, small target, and multiple obstructions.

[0065] Firstly, see [the following] Figure 1 This application provides a method for human detection based on a top-mounted camera, the method comprising:

[0066] S101, acquire the scene video stream captured by the top-mounted camera.

[0067] Top-mounted cameras are generally top-mounted binocular cameras, and their mounting height needs to be between 6m and 15m.

[0068] S102, input the image frames of the scene video stream into the human detection model, and extract multiple basic feature maps at different scales.

[0069] After inputting image frames from the scene video stream into the human detection model, multiple convolutional processes, multiple batch normalization processes, and multiple non-linear activations are performed, resulting in five basic feature maps of different scales: the first, second, third, fourth, and fifth basic feature maps. These five basic feature maps have progressively decreasing resolutions and progressively increasing channel numbers.

[0070] S103, based on multiple basic feature maps, performs feature fusion through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map.

[0071] The first intermediate feature map, the second intermediate feature map, and the third intermediate feature map have progressively decreasing resolutions and progressively increasing number of channels.

[0072] The fifth basic feature map is selected from multiple basic feature maps. This fifth basic feature map is then processed by spatial pyramid pooling, convolution, batch normalization, and nonlinear activation to obtain the third intermediate feature map. The upsampled third intermediate feature map is then concatenated with the second-to-last fourth basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention, convolution, batch normalization, and nonlinear activation to obtain the second intermediate feature map. Finally, the upsampled second intermediate feature map is concatenated with the third-to-last third basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention and convolution to obtain the first intermediate feature map.

[0073] By fusing the extracted basic feature maps at multiple different scales through the feature pyramid network, we can obtain the first, second, and third intermediate feature maps with decreasing resolution and increasing number of channels. Since the first, second, and third intermediate feature maps represent features at different scales, we can capture human objects at the corresponding scales based on these three intermediate feature maps.

[0074] S104, based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map, generate a multi-scale first detection result through the first detection branch.

[0075] The first detection result includes a first human body detection box based on the anchor frame and its confidence level.

[0076] After convolution and activation of the first intermediate feature map, the first feature map is obtained;

[0077] The first intermediate feature map after feature extraction is concatenated with the second intermediate feature map, and the concatenation result is subjected to linear spatial attention and feature extraction to obtain the second preliminary feature map. The second preliminary feature map is then convolved and activated to obtain the second feature map.

[0078] The second preliminary feature map after feature extraction is concatenated with the third intermediate feature map, and the concatenation result is subjected to linear spatial attention and feature extraction to obtain the third preliminary feature map. The third preliminary feature map is then convolved and activated to obtain the third feature map.

[0079] The first feature map, the second feature map, and the third feature map are merged, and the location of the first human detection box and its confidence score at multiple scales are output based on the merged result.

[0080] The first feature map is generated based on the first intermediate feature map and has a relatively high resolution; the second feature map is generated based on the second intermediate feature map and has a medium resolution; the third feature map is generated based on the third intermediate feature map and has a relatively low resolution. The high-resolution receptive field is smaller, suitable for detecting small targets. At a height of 15m, the human body image is very small, and the first feature map can capture the detailed features of these tiny targets. The low-resolution receptive field is larger, suitable for detecting large targets. At a height of 6m, the human body image is larger, and the third feature map can see more global contextual information. The second feature map strikes a balance between the two.

[0081] S105, the upsampled first intermediate feature map is spliced ​​and fused with the shallow feature map in the multiple basic feature maps, and a second detection result is generated through the second detection branch.

[0082] The second detection result includes a heatmap containing confidence levels for the human body's center point.

[0083] The first intermediate feature map after upsampling is concatenated with the second basic feature map obtained from the second of the multiple basic feature maps, and the concatenation result is subjected to feature extraction to obtain a heat map containing the confidence of the human body center point.

[0084] The second basic feature map has a high resolution. By generating a heatmap containing the confidence of the human body center point through the second basic feature map, the best balance is achieved between spatial resolution and semantic information. It is particularly suitable for solving the detection and accurate localization of small targets at high altitudes, while maintaining reasonable computational efficiency.

[0085] See Figure 2 This is a schematic diagram of a heatmap. The heatmap has the same size as the second basic feature map. Each pixel value represents the confidence that the location is the center point of the target. The range is between [0,1]. The farther away from the center point, the closer the value is to 0, and the closer to the center point, the closer the value is to 1.

[0086] S106, merge the first detection result and the second detection result to output the final human body detection box.

[0087] After obtaining the heatmap containing the confidence scores of the human body center point, the second human detection box needs to be obtained in the following way:

[0088] The local maximum value of the heatmap containing the confidence of the human body center point is found by using a two-dimensional plane search algorithm;

[0089] The confidence level of the spread from the center point corresponding to the local maximum value to the center point of the human body is less than the preset confidence level.

[0090] The boundary of the second human detection frame is constructed using the diffusion boundary.

[0091] This application achieves a relatively accurate second human detection frame by diffusing outwards from the center point with the highest confidence level and reasonably pre-setting the confidence level to control the diffusion boundary.

[0092] In one example, the local maximum value can be used as the confidence level of the second human detection box;

[0093] Based on the confidence scores, the first human detection box and the second human detection box are sorted in descending order to obtain a candidate list;

[0094] Select the first human detection box in the candidate list as the final human detection box, and calculate the intersection-union ratio (IUU) of the final human detection box with each human detection box in the candidate list except itself.

[0095] Human detection boxes with an intersection-union ratio exceeding a preset intersection-union ratio are removed from the candidate list;

[0096] Return to the steps of selecting the first human detection box in the candidate list as the final human detection box, and calculating the intersection-union ratio (IUU) of the final human detection box with each human detection box in the candidate list except itself, until the candidate list is empty.

[0097] The first human detection box selected in the candidate list each time is the human detection box with the highest confidence score at that time. Based on it, the intersection-union ratio (IUR) with each human detection box in the candidate list except itself is calculated. If the IUR value is high (e.g., reaching 0.8 or 0.9), it means that the two human detection boxes are likely to define the same target. Therefore, the human detection boxes with low confidence scores need to be deleted, and the human detection boxes with high confidence scores need to be retained.

[0098] By selecting the final human detection box from the first and second human detection boxes through the above process, it is possible to ensure that for the same human body, the detection box with high confidence is retained, thereby improving the accuracy of human body detection.

[0099] The human detection method based on a top-mounted camera provided in this application includes: acquiring a scene video stream captured by the top-mounted camera; inputting image frames of the scene video stream into a human detection model to extract multiple basic feature maps at different scales; fusing features based on the multiple basic feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map have progressively decreasing resolutions and progressively increasing channel numbers; generating a multi-scale first detection result based on the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map through a first detection branch, the first detection result including a first human detection box based on anchor boxes and its confidence; concatenating and fusing the upsampled first intermediate feature map with shallow feature maps from the multiple basic feature maps, and generating a second detection result through a second detection branch, the second detection result including a heatmap containing the confidence of the human center point; fusing the first detection result and the second detection result to output the final human detection box. This method improves the recall and accuracy of human detection.

[0100] The heatmap above, which includes the confidence level of the human body's center point, is generated using the following two-dimensional Gaussian kernel function formula:

[0101] , ;

[0102] Where (x, y) are the coordinates of a point on the heatmap, (c x c y () represents the coordinates of the center point of the actual human detection frame. is the standard deviation, which controls the width of the distribution; R is the Gaussian kernel radius.

[0103] ;

[0104] (x1, y1, x2, y2) together constitute the boundary of the real human body detection box.

[0105] R = min(r1, r2, r3);

[0106] Where r1, r2, and r3 are obtained by solving the following equations:

[0107] ;

[0108] ;

[0109] ;

[0110] Where min overlapTo set the threshold hyperparameter (for example, it can be 0.7, indicating that the intersection-union ratio of the predicted Gaussian box and the real human detection box is not less than 0.7), w is the width of the real human detection box and h is the height of the real human detection box.

[0111] The three equations above simulate three relative positional relationships (external tangency, internal tangency, and one internal and one external) between a Gaussian distribution circle and the actual human detection bounding box. The goal is to find the minimum circle radius so that the overlap area between the circle and the actual human detection bounding box meets the requirements. The optimal circle radius differs for human detection bounding boxes with different aspect ratios. Minimizing R ensures that the required minimum overlap is achieved even in the worst-case scenario, avoiding inaccurate localization due to excessive Gaussian distribution diffusion. Small values ​​of w and h result in a small calculated R, while large values ​​of w and h result in a large calculated R. The solution in this application can automatically adapt to human targets of different sizes.

[0112] In one example, the training process of the human detection model includes:

[0113] Based on the center point positions of every two ground truth bounding boxes on the original training image, calculate the Euclidean distance and intersection-union ratio of every two ground truth bounding boxes;

[0114] When the intersection-union ratio is greater than 0 or the Euclidean distance is less than the preset distance value, any one of the real annotation boxes will be blacked out and deleted.

[0115] The human detection model is trained using the original training images after blacking out and deleting the ground truth bounding boxes.

[0116] The original training images include manually annotated ground truth bounding boxes, each representing a human body. The model frequently encounters crowded scenarios, such as two people with their arms around each other or a crowded shopping mall. In these scenarios, human targets can easily occlude each other, causing the loss of some human features. This results in the model failing to detect human targets during the inference phase, leading to missed detections. To simulate such crowded scenarios, this application calculates the Euclidean distance and Intersection over Union (IOU) based on the actual center points of each pair of ground truth bounding boxes. When the IOU is greater than 0 or the Euclidean distance is less than a preset value, any ground truth bounding box is blacked out and deleted. Thus, the original training image becomes an image consisting of partially blacked-out ground truth bounding boxes and black blocks. The blacked-out ground truth bounding boxes simulate the occlusion of human features. The human detection model is trained using the original training images with blacked-out and blacked-out ground truth bounding boxes, allowing the model to learn to predict unoccluded and blacked-out ground truth bounding boxes, thereby improving the robustness of the human detection model.

[0117] Suppose we have two true bounding boxes, box1 and box2, with coordinates of their bottom left and top right corners as follows:

[0118] The distance is calculated as follows:

[0119] ;

[0120] , ;

[0121] , ;

[0122] IOU calculates the intersection-union ratio of two frames:

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] Where w i and h i Indicates the width and height of the intersecting boxes:

[0129] , ;

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] When IOU > 0 or distance < 50 pixels, randomly select one of the two boxes with a probability of 0.5 to fill the real box with a black box, and then delete the real box. See [link to implementation]. Figure 3 .

[0135] In one example, see Figure 4This is a schematic diagram of the human detection model structure in this application. Input side: The model input is a single RGB-3 channel image with a resolution of 896*672. Model detection: The input image undergoes inference through the model architecture, which consists of multiple network layers, including ConvBNRelu6, C2f, SPPFM, UpSample, CBAM_LSA, Concat, Conv2d, Sigmoid, and a heatmap output. ConvBNReLU6 consists of a Conv2d convolutional layer with 3x3 kernels, a BatchNorm2d batch normalization layer, and a ReLU6 activation layer; C2f consists of a Conv2d convolutional layer with 1x1 kernels and a front-end convolutional layer with 1x1 width, multiple Conv2d convolutional layers with 3x3 kernels forming a depthwise convolutional layer, and a Conv2d convolutional layer with 1x1 kernels and a 1.5x width convolutional layer; SPPFM is a spatial pyramid pooling layer; UpSample is a 2x upsampling layer with nearest neighbor as its upsampling method; CBAM_LSA is a linear spatial attention mechanism module; Concat is an operator that merges two input feature maps by channel dimension; and Sigmoid is a normalized activation function.

[0136] First, the input image goes through two ConvBNRelu6 layers, then one C2f operator, one ConvBNRelu6 operator, and one C2f operator to output the P3 feature map;

[0137] P1 = ConvBNRelu6(input);

[0138] P2 = ConvBNRelu6(P1);

[0139] P3= C2f(ConvBNRelu6(C2f(P2)));

[0140] Wherein, P1 represents the intermediate output feature map, P2 is used to predict the heatmap, P3 represents the upper output feature map, P4 is the intermediate output feature map, and P5 is the final output feature map. P1 has a resolution of 448*336 and 32 channels; P2 has a resolution of 224*168 and 64 channels; P3 has a resolution of 112*84 and 128 channels; P4 has a resolution of 56*42 and 256 channels; and P5 has a resolution of 28*21 and 512 channels.

[0141] Next, the P5 feature map is processed by one layer of SPPFM operator and one layer of ConvBNRelu6 operator to output the N3 feature map (28*21*512).

[0142] N3 = ConvBNRelu6(SPPFM(P5));

[0143] The N3 feature map is processed by the UpSample operator and then concatted with the P4 feature map. After passing through the CBAM_LSA module and a ConvBNRelu6 operator layer, the N2 feature map (56*42*256) is obtained.

[0144] N2= ConvBNRelu6(CBAM_LSA(Concat(UpSample(N3),P4)));

[0145] The N2 feature map will be upsampled by the UpSample operator and then subjected to the Concat operator operation with the output P3 feature map. Finally, it will be processed by the CBAM-LSA module and the C2f operator to output the N1 feature map (112*84*128).

[0146] N1= C2f (CBAM_LSA(Concat(UpSample(N2),P3)));

[0147] The N1 feature map is processed by the UpSample operator and then concatted with the output P2 feature map. Finally, the output heatmap feature layer (224*168*1) is generated by the C2f operator.

[0148] Heatmap= C2f (Concat(UpSample(N1),P2)));

[0149] The N1 feature layer, after passing through Conv2d and Sigmoid layers, outputs an O1 feature map:

[0150] O1 = Sigmoid(Conv2d(N1));

[0151] Simultaneously, the N1 feature map will be copied, processed by the ConvBNRelu6 operator, and combined with the N2 feature map. Figure 1 The M1 feature map (56*42*384) is obtained by passing through the Concat operator, CBAM-LSA, and C2f operator, and then through the Conv2d and Sigmoid layers to output the O2 feature map.

[0152] M1=C2f(CBAM-LSA(Concat(ConvBNRelu6(N1),N2));

[0153] O2 = Sigmoid(Conv2d (M1));

[0154] Similarly, the M1 feature layer passes through the ConvBNRelu6 operator and is combined with the N3 feature layer. Figure 1The M2 feature map (28*21*768) is obtained by passing through the Concat operator, CBAM-LSA, and C2f operator, and then the O3 feature map is output by passing through the Conv2d and Sigmoid layers.

[0155] M2=C2f(CBAM-LSA(Concat(ConvBNRelu6(M1),N3));

[0156] O3 = Sigmoid(Conv2d (M2));

[0157] The outputs of the O1, O2, and O3 feature maps are merged, and the positions xywhs and scores of the human detection boxes are extracted from them. xywhs represents the x-coordinate, y-coordinate, width, and height of the center point of the human detection box; scores represents the confidence scores of the detection boxes containing human bodies.

[0158] Secondly, the present invention provides a human body detection device based on a top-mounted camera, the device comprising:

[0159] The acquisition module is used to acquire scene video streams captured by the top-mounted camera;

[0160] The basic feature map extraction module is used to input image frames from the scene video stream into the human detection model and extract basic feature maps at multiple different scales.

[0161] The intermediate feature map acquisition module is used to perform feature fusion based on multiple basic feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map, and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map, and the third intermediate feature map have successively decreasing resolutions and successively increasing number of channels;

[0162] The first detection result acquisition module is used to generate a multi-scale first detection result based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map through the first detection branch. The first detection result includes a first human body detection box based on the anchor box and its confidence.

[0163] The second detection result acquisition module is used to splice and fuse the upsampled first intermediate feature map with the shallow feature map in the multiple basic feature maps, and generate a second detection result through the second detection branch. The second detection result includes a heat map containing the confidence of the human body center point.

[0164] The final human detection box acquisition module is used to fuse the first detection result and the second detection result to output the final human detection box.

[0165] In one possible implementation, the intermediate feature map acquisition module is specifically used for:

[0166] The fifth basic feature map is selected from multiple basic feature maps. This fifth basic feature map is then processed by spatial pyramid pooling, convolution, batch normalization, and nonlinear activation to obtain the third intermediate feature map. The upsampled third intermediate feature map is then concatenated with the second-to-last fourth basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention, convolution, batch normalization, and nonlinear activation to obtain the second intermediate feature map. Finally, the upsampled second intermediate feature map is concatenated with the third-to-last third basic feature map from the multiple basic feature maps. The concatenated result is then processed by linear spatial attention and convolution to obtain the first intermediate feature map.

[0167] In one possible implementation, the second detection result acquisition module is specifically used for:

[0168] The first intermediate feature map after upsampling is concatenated with the second basic feature map obtained from the second of multiple basic feature maps, and the concatenation result is subjected to feature extraction to obtain a heat map containing the confidence of the human body center point.

[0169] The first detection result acquisition module is specifically used for:

[0170] After convolution and activation of the first intermediate feature map, the first feature map is obtained;

[0171] The first intermediate feature map after feature extraction is concatenated with the second intermediate feature map, and the concatenation result is subjected to linear spatial attention and feature extraction to obtain the second preliminary feature map. The second preliminary feature map is then convolved and activated to obtain the second feature map.

[0172] The second preliminary feature map after feature extraction is concatenated with the third intermediate feature map, and the concatenation result is subjected to linear spatial attention and feature extraction to obtain the third preliminary feature map. The third preliminary feature map is then convolved and activated to obtain the third feature map.

[0173] The first feature map, the second feature map, and the third feature map are merged, and the location of the first human detection box and its confidence score at multiple scales are output based on the merged result.

[0174] In one possible implementation, the device further includes a second human detection frame determination module, used for:

[0175] The local maximum value of the heatmap containing the confidence of the human body center point is found by using a two-dimensional plane search algorithm;

[0176] The confidence level of the spread from the center point corresponding to the local maximum value to the center point of the human body is less than the preset confidence level.

[0177] The boundary of the second human detection frame is constructed using the diffusion boundary.

[0178] In one possible implementation, the final human body detection box acquisition module is specifically used for:

[0179] The local maximum value is used as the confidence level of the second human detection box;

[0180] Based on the confidence scores, the first human detection box and the second human detection box are sorted in descending order to obtain a candidate list;

[0181] Select the first human detection box in the candidate list as the final human detection box, and calculate the intersection-union ratio (IUU) of the final human detection box with each human detection box in the candidate list except itself.

[0182] Human detection boxes with an intersection-union ratio exceeding a preset intersection-union ratio are removed from the candidate list;

[0183] Return to the steps of selecting the first human detection box in the candidate list as the final human detection box, and calculating the intersection-union ratio (IUU) of the final human detection box with each human detection box in the candidate list except itself, until the candidate list is empty.

[0184] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for detecting the direction of pedestrian gaze based on a top-mounted camera.

[0185] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0186] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are described simply because they are similar to the method embodiments; relevant parts can be referred to the descriptions of the method embodiments.

[0188] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A top camera based human detection method, characterized in that, The method comprises: acquiring a scene video stream collected by a top-mounted camera; inputting an image frame of the scene video stream into a human body detection model to extract a plurality of basic feature maps of different scales; based on the plurality of basic feature maps, performing feature fusion through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map and the third intermediate feature map have resolution decreasing and channel number increasing in turn; based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map, generating a multi-scale first detection result through a first detection branch, the first detection result comprising an anchor box-based first human body detection box and a confidence thereof; splicing and fusing the up-sampled first intermediate feature map and a shallow layer feature map in the plurality of basic feature maps, and generating a second detection result through a second detection branch, the second detection result comprising a heat map containing human body center point confidence; fusing the first detection result and the second detection result to output a final human body detection box; the splicing and fusing of the up-sampled first intermediate feature map and the shallow layer feature map in the plurality of basic feature maps, and the generation of the second detection result through the second detection branch, comprises: splicing the up-sampled first intermediate feature map and a second basic feature map obtained from the second of the plurality of basic feature maps, and obtaining a heat map containing human body center point confidence after feature extraction on the splicing result; the generation of the multi-scale first detection result through the first detection branch based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map, comprises: performing convolution and activation on the first intermediate feature map to obtain a first feature map; splicing the first intermediate feature map after feature extraction and the second intermediate feature map, and obtaining a second preliminary feature map after linear spatial attention and feature extraction on the splicing result, and obtaining a second feature map after convolution and activation on the second preliminary feature map; splicing the second preliminary feature map after feature extraction and the third intermediate feature map, and obtaining a third preliminary feature map after linear spatial attention and feature extraction on the splicing result, and obtaining a third feature map after convolution and activation on the third preliminary feature map; merging the first feature map, the second feature map and the third feature map, and outputting multi-scale first human body detection box positions and confidence thereof based on the merging result; the training process of the human body detection model comprises: calculating the Euclidean distance and the intersection over union value of each two real annotation boxes according to the center point positions of each two real annotation boxes on an original training image; when the intersection over union value is greater than 0 or the Euclidean distance is less than a preset distance value, blackening and deleting any one real annotation box; the blackened real annotation box is used to simulate the case that human body features are blocked; training the human body detection model through the original training image after blackening and deleting the real annotation boxes; the heat map containing human body center point confidence is generated through the following two-dimensional Gaussian kernel function formula: ， ； where (x, y) is the point coordinate on the heat map, (c x , c y ) is the center point coordinate of the real human detection frame, is the standard deviation, which controls the width of the distribution, and R is the Gaussian kernel radius; R = min(r1, r2, r3); wherein r1, r2, r3 are obtained from the following equations respectively: ； ； ； where min overlap is a set threshold hyperparameter, w is the width of the real human bounding box, and h is the height of the real human bounding box.

2. The method of claim 1, wherein, The feature fusion is performed on the plurality of base feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map and a third intermediate feature map, including: The fifth base feature map is obtained from the plurality of base feature maps, and after spatial pyramid pooling processing, convolution processing, batch normalization processing and non-linear activation are performed on the fifth base feature map, the third intermediate feature map is obtained; the third intermediate feature map after upsampling is spliced with the fourth base feature map obtained from the plurality of base feature maps in reverse order, and after linear spatial attention, convolution processing, batch normalization processing and non-linear activation are performed on the spliced result, the second intermediate feature map is obtained; the second intermediate feature map after upsampling is spliced with the third base feature map obtained from the plurality of base feature maps in reverse order, and after linear spatial attention and convolution processing are performed on the spliced result, the first intermediate feature map is obtained.

3. The method of claim 1, wherein, The method further includes: finding a local maximum value of the heat map containing the human center point confidence through a two-dimensional plane search algorithm; spreading from the center point corresponding to the local maximum value to the human center point confidence less than a pre-set confidence in all directions; using the diffusion boundary to constitute the boundary of the second human detection frame.

4. The method of claim 3, wherein, The first detection result and the second detection result are fused, and a final human detection frame is output, including: taking the local maximum value as the confidence of the second human detection frame; sorting the first human detection frame and the second human detection frame in descending order according to the confidence score to obtain a candidate list; selecting the first human detection frame in the candidate list as the final human detection frame, and calculating the intersection over union of the final human detection frame and each human detection frame in the candidate list except itself; removing the human detection frame with an intersection over union value exceeding a pre-set intersection over union from the candidate list; returning to the step of selecting the first human detection frame in the candidate list as the final human detection frame, and calculating the intersection over union of the final human detection frame and each human detection frame in the candidate list except itself until the candidate list is empty.

5. A top camera based human detection apparatus, characterized by, The device includes: an acquisition module configured to acquire a scene video stream captured by a top-mounted camera; a base feature map extraction module configured to input an image frame of the scene video stream into a human detection model, and extract a plurality of base feature maps of different scales; an intermediate feature map acquisition module configured to perform feature fusion on the plurality of base feature maps through a feature pyramid network to obtain a first intermediate feature map, a second intermediate feature map and a third intermediate feature map; the first intermediate feature map, the second intermediate feature map and the third intermediate feature map have resolution decreasing and channel number increasing in turn; a first detection result acquisition module configured to generate a multi-scale first detection result through a first detection branch based on the first intermediate feature map, the second intermediate feature map and the third intermediate feature map; the first detection result includes an anchor box-based first human detection frame and its confidence. The second detection result acquisition module is configured to splice and fuse the up-sampled first intermediate feature map and a shallow layer feature map in the plurality of base feature maps, and generate a second detection result through a second detection branch, the second detection result comprising a heat map containing a human center point confidence; The final human detection box acquisition module is configured to fuse the first detection result and the second detection result, and output a final human detection box. The second detection result acquisition module is specifically configured to splice the up-sampled first intermediate feature map and a second base feature map obtained from the plurality of base feature maps, and obtain a heat map containing a human center point confidence after feature extraction on the spliced result. The first detection result acquisition module is specifically configured to obtain a first feature map after convolution and activation of the first intermediate feature map. The first intermediate feature map after feature extraction and the second intermediate feature map are spliced, and a second prepared feature map is obtained after linear spatial attention and feature extraction on the spliced result; the second prepared feature map is subjected to convolution and activation to obtain a second feature map. The second prepared feature map after feature extraction and the third intermediate feature map are spliced, and a third prepared feature map is obtained after linear spatial attention and feature extraction on the spliced result; the third prepared feature map is subjected to convolution and activation to obtain a third feature map. The first feature map, the second feature map and the third feature map are merged, and a multi-scale first human detection box position and its confidence are output based on the merging result. The training process of the human detection model comprises: According to the center point positions of each two real annotation boxes on the original training image, the Euclidean distance and the intersection over union value of each two real annotation boxes are calculated; When the intersection over union value is greater than 0 or the Euclidean distance is less than a preset distance value, any one real annotation box is blackened and the real annotation box is deleted; the blackened real annotation box is used to simulate the case that the human feature is blocked; The human detection model is trained through the original training image after the real annotation boxes are blackened and deleted; The heat map containing the human center point confidence is generated through the following two-dimensional Gaussian kernel function formula: ， ； where (x, y) is the point coordinate on the heat map, (c x , c y ) is the center point coordinate of the real human detection frame, is the standard deviation, which controls the width of the distribution, and R is the Gaussian kernel radius; R = min(r1, r2, r3); Wherein, r1, r2, r3 are obtained from the following equations: ； ； ； where min overlap is a set threshold hyperparameter, w is the width of the real human bounding box, and h is the height of the real human bounding box.

6. The apparatus of claim 5, wherein, The intermediate feature map acquisition module is specifically configured to: The fifth base feature map obtained from the plurality of base feature maps is selected, and the fifth base feature map is subjected to spatial pyramid pooling processing, convolution processing, batch normalization processing and non-linear activation to obtain a third intermediate feature map; the up-sampled third intermediate feature map and a fourth base feature map obtained from the plurality of base feature maps are spliced, and the spliced result is subjected to linear spatial attention, convolution processing, batch normalization processing and non-linear activation to obtain a second intermediate feature map; the up-sampled second intermediate feature map and a third base feature map obtained from the plurality of base feature maps are spliced, and the spliced result is subjected to linear spatial attention and convolution processing to obtain a first intermediate feature map.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the human body detection method based on the top-mounted camera in any one of claims 1-4.

Images