Construction site safety detection method based on multi-scale feature fusion and key point positioning
By employing multi-scale feature fusion and key point localization methods, the problem of detecting large near-field targets and small far-field targets in the safety monitoring of large construction sites has been solved, improving the detection recall rate and environmental adaptability, and achieving stable violation identification in complex scenarios.
Patent Information
- Application Number
- CN202610923843.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-25
- Publication Date
- 2026-11-03
AI Technical Summary
Traditional visual detection methods based on single-scale feature extraction are difficult to simultaneously represent the effective features of large near-field targets and small far-field targets in the safety monitoring of large construction sites. This results in a high rate of missed detections of workers at a distance, and environmental changes affect the stability of the detection.
A multi-scale feature fusion and key point localization method is adopted. Multi-scale feature fusion is performed through an improved lightweight backbone network, shallow features are enhanced by frequency-aware enhancement branches, key point topology relationships are constrained by graph convolutional networks, and action classification is performed through spatiotemporal graph convolutional networks. Violation judgment thresholds and topology constraint weights are dynamically adjusted.
It effectively alleviates the problem of missed detection caused by scale changes, improves the detection recall level, reduces the false alarm rate, can maintain the stability and reliability of detection in complex environments, and can identify both static and dynamic violations.
Smart Images
Figure CN123033178A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of construction site safety inspection technology, and in particular relates to a construction site safety inspection method based on multi-scale feature fusion and key point localization. Background Technology
[0002] In the safety supervision of large construction sites, it is usually necessary to install high-altitude cameras on tower cranes or high points to obtain a wide monitoring field of view; however, this deployment method results in a huge difference in the size of the personnel targets in the monitoring screen. People in the foreground may occupy hundreds of pixels, while people in the background may appear as tiny targets of less than ten pixels.
[0003] Traditional visual detection methods based on single-scale feature extraction are mainly optimized for medium-sized targets when facing such scenarios. While shallow features retain high spatial resolution, they have weak semantic information, and deep features have rich semantic information but significantly reduced resolution. It is difficult to simultaneously take into account the effective feature representation of large targets in the foreground and small targets in the background, resulting in a high rate of missed detection of workers at a distance. This has become the core problem restricting the practical application of safety monitoring in large-scale construction sites. Therefore, we propose a construction site safety detection method based on multi-scale feature fusion and key point localization. Summary of the Invention
[0004] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:
[0005] This invention relates to a construction site safety inspection method based on multi-scale feature fusion and key point localization, comprising the following steps: Step S1: Acquire the real-time video stream from the high-altitude camera, and perform multi-scale feature fusion preprocessing on the image through an improved lightweight backbone network to generate a multi-scale feature map. The multi-scale feature map is used to extract features from both distant small targets and close-range large targets. Step S2: Perform cascaded detection on the multi-scale feature map. The first stage performs global human detection and outputs human bounding boxes. The second stage performs key point localization and accessory detection within each human bounding box. During the key point localization process, graph convolutional networks are used to constrain the geometric topological relationship between key points, and the corrected key point coordinates and accessory detection boxes are output. Step S3: Using the corrected key point coordinates and attachment detection boxes, perform violation determination by fusing key point semantics, and output the violation type and its confidence score, including helmet wearing detection and posture violation detection. Helmet wearing detection is completed by calculating the spatial relationship between head key points and helmet detection boxes. Posture violation detection is completed by extracting human key point sequences from consecutive frames and inputting them into a spatiotemporal graph convolutional network for action classification. Step S4: Output a structured alarm based on the violation type and confidence score, and upload the violation type, time, location, captured image and confidence information to the management platform.
[0006] Further, step S1 includes the following steps: Step S11: Extract shallow and deep features with progressively decreasing resolution from the improved lightweight backbone network. Upsample the deep features to the same spatial size as the shallow features and then stitch them together along the channel dimension to obtain the stitched features. Step S12: Adaptive channel weighting is applied to the concatenated features using a cross-layer feature recalibration and fusion module. Channel statistics are obtained through global average pooling, and local cross-channel dependencies are captured using one-dimensional convolution. Attention weights are generated to recalibrate the concatenated features, and then 1×1 convolution is used to reduce the dimensionality and generate the fused features of the current layer. The fusion process is represented as follows:
[0007] In the formula, This is a concatenated feature obtained by upsampling deep features and then concatenating them with shallow features along the channel dimension. For global average pooling, Indicates the kernel size as One-dimensional convolution operation, This represents the Sigmoid activation function. This represents a 1×1 convolution dimensionality reduction operation. This represents the fusion feature at the current level.
[0008] Step S13: Generate the highest resolution fused features Then, a frequency-aware enhancement branch is introduced to... Each channel independently performs a discrete cosine transform to convert the spatial domain features to the frequency domain, resulting in a spectrum. Step S14: Use a learnable mask matrix of the same size as the spectrogram. The spectrum is weighted, and the mask matrix is initialized during training to a numerical distribution that tends to preserve high-frequency coefficients. Then, the inverse discrete cosine transform is performed on the weighted spectrum to obtain high-frequency detail enhancement features in the spatial domain. Step S15: Combine high-frequency detail enhancement features with the original Element-wise addition yields the enhanced shallow features. This is used for subsequent small target detection; The frequency-aware enhancement branch strengthens the high-frequency details of shallow features to improve the representation of small targets at a distance. The enhancement process is represented as follows:
[0009] In the formula, For the highest resolution fusion features, For discrete cosine transform, A learnable mask matrix, It is the inverse discrete cosine transform. This is the enhanced shallow layer feature.
[0010] Furthermore, the specific method of cascaded detection in step S2 is as follows: the first stage uses the YOLOv5s detector to perform global human detection on multi-scale feature maps and outputs the human bounding box of each person; the second stage locates a preset number of human skeletal key points within each human bounding box through heatmap regression, and simultaneously runs the target detection branch to detect accessories such as safety helmets; step S2 also includes the following steps: Step S21: Regress the data obtained from the heatmap The initial coordinates of key points are used as graph nodes, and an adjacency matrix is defined based on the inherent connection relationships of the human skeleton. If node With nodes If there is a physical connection ,otherwise ; Step S22: Construct the projection adaptive matrix Elements of the projection adaptive matrix Based on the installation tilt angle of the high-altitude camera Calculations are performed for limb connections representing vertical relationships. For other connections, ,in For adjustment factors; Step S23: Adjust the adjacency matrix to After adding self-loops, a normalized graph Laplace matrix is constructed; Step S24: The keypoint coordinates are topologically corrected using a multi-layer graph convolutional network combined with a projection adaptive adjacency matrix, enabling occluded keypoints to infer their reasonable locations from connected visible nodes. The feature update method for each layer of the graph convolutional network is as follows:
[0011] In the formula, This is the adjacency matrix adjusted by the projection adaptive matrix. It is the identity matrix. for The degree matrix, For the first Layer node characteristics, For the first Layer-learnable weight matrix, For activation function, For the first Layer node characteristics.
[0012] Furthermore, in step S3, the helmet wearing detection includes the following steps: Step S31: Extract the coordinates of key points in the head Coordinates of the center point of the safety helmet detection frame Calculate the Euclidean distance between the head key points and the center point of the helmet detection frame. ; Step S32: Calculate a continuous wearing tendency score The determination of whether a safety helmet is actually worn on the head is based on a combination of distance and vertical relationship. If the wear is greater than the dynamic threshold, it is considered to be properly worn; if the human body is present but... If the threshold is not reached and the minimum Euclidean distance between the wrist key point and the center of the helmet detection frame is lower than the preset close-range threshold, it is determined that the helmet is not being worn. Wearing tendency score The calculation method is as follows:
[0013] The direction logic function used to determine the vertical relationship is:
[0014] In the formula, The distance is the Euclidean distance between the key point on the head and the center point of the safety helmet detection frame. For distance tolerance parameters, This is the vertical difference between the center point of the safety helmet detection frame and the key point on the head. The parameter controlling the steepness of the directional logic function is such that when the center of the helmet is higher than the key point of the head in the vertical direction, the value of the directional logic function approaches 1.
[0015] Furthermore, in step S3, the posture violation detection includes the following steps: Step S33: Continuous data acquisition Frame images, each frame extracted independently Each individual's key points form a spatiotemporal key point sequence; Step S34: with each frame Each key point is a spatial node. Temporal edges are established between the same key points in adjacent frames, and spatial edges are established between key points with physical connections within a single frame to construct a spatiotemporal graph. The weight of the spatial edge is dynamically determined based on the regression confidence of the heatmap of the two key points in the frame to adaptively adjust the strength of topological constraints caused by occlusion. Step S35: Input the spatiotemporal graph into the spatiotemporal graph convolutional network, extract the dynamic topological features of the action through alternating spatial graph convolution and temporal convolution operations, and classify and output the action category through a fully connected layer; The method for determining the weights of spatial edges is as follows:
[0016] In the formula, For the first Frame number The confidence level of the heatmap regression for each key point. For the first Frame number The confidence level of the heatmap regression for each key point. For the first Frame connection nodes With nodes Spatial edge weights.
[0017] Further, step S4 includes the following steps: Step S41: When a violation event is triggered, capture three frames: the current frame and the frames before and after it. Step S42: Draw the human body bounding box, the attachment detection box, and the key point connection diagram after graph convolutional network correction on the captured image to generate a key point annotation map; Step S43: Encapsulate the violation type, occurrence time, camera number, encoded data of the captured image, encoded data of the key point annotation map, and violation confidence score into a structured data body; Step S44: Upload the structured data to the management platform via the wireless communication module.
[0018] Furthermore, step S3 also includes obtaining current environmental parameters and camera installation parameters, and dynamically adjusting the judgment threshold and topological constraint weights in the graph convolutional network involved in the violation judgment in step S3, as follows: Calculate the average grayscale value for the current video frame. and local contrast , and combine them into a lighting description vector; The illumination description vector is compared with the pitch angle in the camera installation parameters. Encoding fusion is performed, and an adaptive adjustment factor is output through a miniature fully connected network. and weight adjustment value ; The decision threshold for helmet wearing detection and the topological constraint weights in the graph convolutional network are updated using adaptive adjustment factors and weight adjustment values. The environmental perception module automatically adjusts the judgment threshold based on lighting conditions and pitch angle. The judgment threshold is updated as follows:
[0019] The environment perception module automatically adjusts the vertical connection projection weights based on lighting conditions and pitch angle. The projection weights are updated as follows:
[0020] In the formula, The baseline threshold for helmet wearing detection. The updated judgment threshold, As an adaptive adjustment factor, As the initial adjustment factor, This is the weighted adjustment value. Install a tilt angle for the camera. This is the updated vertical connection projection weight.
[0021] A construction site safety detection device based on multi-scale feature fusion and key point localization is used to perform the method described in any of the above-mentioned methods, including: a high-altitude camera, a multi-scale feature fusion preprocessing module, a cascaded human body and component detection module, an environmental perception and adaptive adjustment module, a violation judgment module, a structured alarm output module, and a management platform. The high-altitude camera, multi-scale feature fusion preprocessing module, cascaded human body and component detection module, violation judgment module, structured alarm output module, and management platform are sequentially connected in a unidirectional manner; the output of the environmental perception and adaptive adjustment module is unidirectionally connected to the inputs of the cascaded human body and component detection module and the violation judgment module, respectively.
[0022] Furthermore, the high-altitude camera is deployed on a tower crane or at a high point to collect real-time video streams from the construction site; The multi-scale feature fusion preprocessing module is used to extract multi-scale features from video frames, fuse deep semantics and shallow details through cross-layer recalibration, and enhance high-frequency information using a frequency-aware enhancement branch to generate a multi-scale feature map that takes into account both near and far targets. The cascaded human body and component detection module first detects the human body bounding box, then locates key points and detects attachments within the box using a heatmap, and simultaneously uses a graph convolutional network combined with pitch angle constraints on the skeletal topology to correct the position of key points when occlusion occurs. The environmental perception and adaptive adjustment module is used to extract the average gray level and local contrast of the current frame to form a lighting description vector. After being fused with the camera pitch angle, it is output as an adjustment factor through a micro-network to dynamically correct the violation judgment threshold and the graph convolution vertical constraint weight.
[0023] Furthermore, the violation determination module is used to calculate the wearing tendency score based on the spatial distance and vertical relationship between the head key points and the safety helmet frame, and to distinguish between qualified wearing and hand-held / waist-mounted wearing based on the wrist position; a spatiotemporal graph is constructed for the continuous key point sequence, and the lying, climbing, and falling actions are classified by spatiotemporal graph convolution. The structured alarm output module is used to capture the current frame and the frames before and after when a violation is triggered, draw the human body frame, the attachment frame and the key point connection annotation diagram, and encapsulate the violation type, time, camera number, image encoding and confidence level before uploading; The management platform is used to receive alarm data, provide functions such as displaying violation information, viewing and statistical analysis of marked diagrams, and assist safety management personnel in verification and decision-making.
[0024] The present invention has the following beneficial effects: 1. This invention utilizes an improved lightweight backbone network for multi-scale feature fusion, adaptively integrating the strong semantic information contained in deep features with the high-resolution detail information preserved in shallow features. Simultaneously, frequency-aware enhancement is applied to the highest-resolution feature map to strengthen the edge and texture responses corresponding to small-sized targets. This mechanism enables the network to possess stable feature extraction capabilities for worker targets occupying only a few pixels in the distant region, effectively mitigating the false negative problem caused by scale variations and improving the overall detection recall level.
[0025] 2. By incorporating semantic information of key human body points, the system calculates the Euclidean distance and vertical relationship between key points on the head and the helmet detection frame to generate a continuous wearing tendency score for judgment, rather than relying on a simple spatial intersection-union threshold. When the helmet is not actually worn on the head, even if the helmet target is detected, the system can still identify the unworn state by matching the spatial relationship with the key points on the wrist, effectively reducing false alarms caused by holding the helmet by hand.
[0026] 3. By continuously acquiring multiple frames of human key point sequences to construct a spatiotemporal graph, and using a spatiotemporal graph convolutional network to jointly model the spatial topological relationships and temporal motion patterns of the key points, the system can automatically extract and classify the motion features of personnel. This mechanism enables the system to identify dynamic behaviors that pose safety hazards, such as lying down, climbing, and falling, extending violation detection from a single static check of wearing items to the level of dynamic behavior analysis.
[0027] 4. Factors such as changes in lighting, dust, shadows, and differences in camera installation angles in the construction site environment can adversely affect the stability of visual inspection. This method extracts the lighting description vector of the current frame in real time through the environmental perception module, combines it with the camera pitch angle parameters, and outputs an adaptive adjustment factor through a miniature fully connected network to dynamically adjust the safety helmet judgment threshold and the graph convolution topology constraint weights. This mechanism enables the system to autonomously adapt the judgment parameters under different time, weather, and installation conditions, maintaining the consistency and reliability of judgment performance and reducing the need for manual parameter adjustment due to environmental changes. Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 This is a flowchart illustrating the construction site safety inspection method based on multi-scale feature fusion and key point localization according to the present invention. Figure 2 This is a schematic diagram of the architecture of the construction site safety detection method based on multi-scale feature fusion and key point localization of the present invention. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] Please see Figure 1-2 As shown, this invention is a construction site safety inspection method based on multi-scale feature fusion and key point localization, comprising the following steps: Step S1: Acquire the real-time video stream from the high-altitude camera, and perform multi-scale feature fusion preprocessing on the image through an improved lightweight backbone network to generate multi-scale feature maps. The multi-scale feature maps are used to extract features from both distant small targets and close-range large targets. Step S2: Perform cascaded detection on multi-scale feature maps. The first stage performs global human detection and outputs human bounding boxes. The second stage performs key point localization and accessory detection within each human bounding box. During the key point localization process, graph convolutional networks are used to constrain the geometric topological relationship between key points, and the corrected key point coordinates and accessory detection boxes are output. Step S3: Using the corrected keypoint coordinates and attachment detection boxes, perform violation determination by fusing keypoint semantics, and output the violation type and its confidence score, including helmet wearing detection and posture violation detection. Helmet wearing detection is completed by calculating the spatial relationship between head keypoints and helmet detection boxes. Posture violation detection is completed by extracting human keypoint sequences from consecutive frames and inputting them into a spatiotemporal graph convolutional network for action classification. Step S4: Output structured alarms based on the violation type and confidence score, and upload the violation type, time, location, captured image and confidence information to the management platform.
[0032] Step S1 includes the following steps: Step S11: Extract shallow and deep features with progressively decreasing resolution from the improved lightweight backbone network. Upsample the deep features to the same spatial size as the shallow features and then stitch them together along the channel dimension to obtain the stitched features. Step S12: Adaptive channel weighting is applied to the concatenated features using a cross-layer feature recalibration and fusion module. Channel statistics are obtained through global average pooling, and local cross-channel dependencies are captured using one-dimensional convolution. Attention weights are generated to recalibrate the concatenated features, and then 1×1 convolution is used to reduce the dimensionality and generate the fused features of the current layer. The fusion process is represented as follows:
[0033] In the formula, This is a concatenated feature obtained by upsampling deep features and then concatenating them with shallow features along the channel dimension. For global average pooling, Indicates the kernel size as One-dimensional convolution operation, This represents the Sigmoid activation function. This represents a 1×1 convolution dimensionality reduction operation. This represents the fusion feature at the current level.
[0034] Step S13: Generate the highest resolution fused features Then, a frequency-aware enhancement branch is introduced to... Each channel independently performs a discrete cosine transform to convert the spatial domain features to the frequency domain, resulting in a spectrum. Step S14: Use a learnable mask matrix of the same size as the spectrogram. The spectrum is weighted, and the mask matrix is initialized during training to a numerical distribution that tends to preserve high-frequency coefficients. Then, the inverse discrete cosine transform is performed on the weighted spectrum to obtain high-frequency detail enhancement features in the spatial domain. Step S15: Combine high-frequency detail enhancement features with the original Element-wise addition yields the enhanced shallow features. This is used for subsequent small target detection; The frequency-aware enhancement branch strengthens the high-frequency details of shallow features to improve the representation of small targets at a distance. The enhancement process is represented as follows:
[0035] In the formula, For the highest resolution fusion features, For discrete cosine transform, A learnable mask matrix, It is the inverse discrete cosine transform. This is the enhanced shallow layer feature.
[0036] The specific method of cascaded detection in step S2 is as follows: The first stage uses the YOLOv5s detector to perform global human detection on multi-scale feature maps and outputs the human bounding box of each person; the second stage locates a preset number of human skeletal key points within each human bounding box through heatmap regression, and simultaneously runs the object detection branch to detect accessories such as safety helmets; step S2 also includes the following steps: Step S21: Obtain the data from heatmap regression The initial coordinates of key points are used as graph nodes, and an adjacency matrix is defined based on the inherent connection relationships of the human skeleton. If node With nodes If there is a physical connection ,otherwise ; Step S22: Construct the projection adaptive matrix Elements of the projection adaptive matrix Based on the installation tilt angle of the high-altitude camera Calculations are performed for limb connections representing vertical relationships. For other connections, ,in For adjustment factors; Step S23: Adjust the adjacency matrix to After adding self-loops, a normalized graph Laplace matrix is constructed; Step S24: The keypoint coordinates are topologically corrected using a multi-layer graph convolutional network combined with a projection adaptive adjacency matrix, enabling occluded keypoints to infer their reasonable locations from connected visible nodes. The feature update method for each layer of the graph convolutional network is as follows:
[0037] In the formula, This is the adjacency matrix adjusted by the projection adaptive matrix. It is the identity matrix. for The degree matrix, For the first Layer node characteristics, For the first Layer-learnable weight matrix, For activation function, For the first Layer node characteristics.
[0038] In step S3, the helmet wearing detection includes the following steps: Step S31: Extract the coordinates of key points in the head Coordinates of the center point of the safety helmet detection frame Calculate the Euclidean distance between the head key points and the center point of the helmet detection frame. ; Step S32: Calculate a continuous wearing tendency score The determination of whether a safety helmet is actually worn on the head is based on a combination of distance and vertical relationship. If the wear is greater than the dynamic threshold, it is considered to be properly worn; if the human body is present but... If the threshold is not reached and the minimum Euclidean distance between the wrist key point and the center of the helmet detection frame is lower than the preset close-range threshold, it is determined that the helmet is not being worn. Wearing tendency score The calculation method is as follows:
[0039] The direction logic function used to determine the vertical relationship is:
[0040] In the formula, The distance is the Euclidean distance between the key point on the head and the center point of the safety helmet detection frame. For distance tolerance parameters, This is the vertical difference between the center point of the safety helmet detection frame and the key point on the head. The parameter controlling the steepness of the directional logic function is such that when the center of the helmet is higher than the key point of the head in the vertical direction, the value of the directional logic function approaches 1.
[0041] In step S3, the detection of posture-related violations includes the following steps: Step S33: Continuous data acquisition Frame images, each frame extracted independently Each individual's key points form a spatiotemporal key point sequence; Step S34: with each frame Each key point is a spatial node. Temporal edges are established between the same key points in adjacent frames, and spatial edges are established between key points with physical connections within a single frame to construct a spatiotemporal graph. The weight of the spatial edge is dynamically determined based on the regression confidence of the heatmap of the two key points in the frame to adaptively adjust the strength of topological constraints caused by occlusion. Step S35: Input the spatiotemporal graph into the spatiotemporal graph convolutional network, extract the dynamic topological features of the action through alternating spatial graph convolution and temporal convolution operations, and classify and output the action category through a fully connected layer; The method for determining the weights of spatial edges is as follows:
[0042] In the formula, For the first Frame number The confidence level of the heatmap regression for each key point. For the first Frame number The confidence level of the heatmap regression for each key point. For the first Frame connection nodes With nodes Spatial edge weights.
[0043] Step S4 includes the following steps: Step S41: When a violation event is triggered, capture three frames: the current frame and the frames before and after it. Step S42: Draw the human body bounding box, the attachment detection box, and the key point connection diagram after graph convolutional network correction on the captured image to generate a key point annotation map; Step S43: Encapsulate the violation type, occurrence time, camera number, encoded data of the captured image, encoded data of the key point annotation map, and violation confidence score into a structured data body; Step S44: Upload the structured data to the management platform via the wireless communication module.
[0044] Step S3 also includes obtaining the current environmental parameters and camera installation parameters, and dynamically adjusting the judgment threshold and topological constraint weights in the graph convolutional network involved in the violation judgment in step S3. The steps are as follows: Calculate the average grayscale value for the current video frame. and local contrast , and combine them into a lighting description vector; The illumination description vector is compared with the pitch angle in the camera installation parameters. Encoding fusion is performed, and an adaptive adjustment factor is output through a miniature fully connected network. and weight adjustment value ; The decision threshold for helmet wearing detection and the topological constraint weights in the graph convolutional network are updated using adaptive adjustment factors and weight adjustment values. The environmental perception module automatically adjusts the judgment threshold based on lighting conditions and pitch angle. The judgment threshold is updated as follows:
[0045] The environment perception module automatically adjusts the vertical connection projection weights based on lighting conditions and pitch angle. The projection weights are updated as follows:
[0046] In the formula, The baseline threshold for helmet wearing detection. The updated judgment threshold, As an adaptive adjustment factor, As the initial adjustment factor, This is the weighted adjustment value. Install a tilt angle for the camera. This is the updated vertical connection projection weight.
[0047] A construction site safety detection device based on multi-scale feature fusion and key point localization is used to execute any of the above methods, including: a high-altitude camera, a multi-scale feature fusion preprocessing module, a cascaded human body and component detection module, an environmental perception and adaptive adjustment module, a violation judgment module, a structured alarm output module, and a management platform. The high-altitude camera, multi-scale feature fusion preprocessing module, cascaded human body and component detection module, violation judgment module, structured alarm output module, and management platform are sequentially connected in one direction; the output of the environmental perception and adaptive adjustment module is connected in one direction to the input of the cascaded human body and component detection module and the violation judgment module, respectively.
[0048] High-altitude cameras are deployed on tower cranes or at high points to collect real-time video streams from the construction site; The multi-scale feature fusion preprocessing module is used to extract multi-scale features from video frames. It fuses deep semantics and shallow details through cross-layer recalibration and strengthens high-frequency information using a frequency-aware enhancement branch to generate multi-scale feature maps that take into account both near and far targets. The cascaded human body and component detection module first detects the human body bounding box, then locates key points and detects attachments within the box using a heatmap, and at the same time uses a graph convolutional network combined with pitch angle constraints on the skeletal topology to correct the position of key points when occlusion occurs. The environmental perception and adaptive adjustment module is used to extract the average gray level and local contrast of the current frame to form a lighting description vector. After being fused with the camera pitch angle, it is output as an adjustment factor through a micro-network to dynamically correct the violation judgment threshold and the graph convolution vertical constraint weight. The violation determination module is used to calculate the wearing tendency score based on the spatial distance and vertical relationship between the head key points and the safety helmet frame, and to distinguish between qualified wearing and hand-held / waist-mounted wearing based on the wrist position; it constructs a spatiotemporal map for a continuous sequence of key points, and classifies lying down, climbing and falling actions through spatiotemporal map convolution. The structured alarm output module is used to capture the current frame and the frames before and after when a violation is triggered, draw the human body frame, the attachment frame and the key point connection annotation diagram, and encapsulate the violation type, time, camera number, image encoding and confidence level before uploading; The management platform is used to receive alarm data, provide functions such as displaying violation information, viewing and statistical analysis of marked diagrams, and assist safety management personnel in verification and decision-making.
[0049] One specific application of this embodiment is: This embodiment relies on a large-scale security monitoring scenario using high-altitude cameras deployed on tower cranes / high points at large construction sites. In this scenario, the imaging scale of personnel targets varies greatly, with large near-field targets coexisting with small distant targets less than 10 pixels. Simultaneously, there are complex environmental interferences such as drastic fluctuations in lighting, dust obstruction, and inconsistent camera installation angles. This invention effectively solves the problems of high false negative rates, numerous false alarms, and poor robustness of traditional methods in scenarios with large scale differences and complex environments. The specific implementation steps are as follows: Step 1: Multi-scale feature fusion preprocessing Real-time video streams from a high-altitude camera deployed on top of a tower crane are acquired. The video resolution is 1920×1080, and the frame rate is 25fps. The video stream is then input frame by frame into an improved lightweight backbone network (improved MobileNetV3). Shallow and deep features with progressively decreasing resolutions are extracted, and multi-scale feature fusion preprocessing is performed to generate multi-scale feature maps that take into account both small targets at long distances and large targets at close range. The specific process is as follows: 1.1 Cross-layer feature recalibration fusion Feature maps from layers 2, 3, and 4 were extracted from the backbone network, with resolutions of 480×270, 240×135, and 120×68, respectively. The deep features (layer 4) were upsampled to the same spatial size as the shallow features (layer 3) using bilinear interpolation, and then concatenated along the channel dimension to obtain the concatenated features. Similarly, the fused features of the third layer are upsampled and then concatenated with the features of the second layer.
[0050] A cross-layer feature recalibration fusion module is used to adaptively weight the concatenated features through channels. The fusion process is represented as follows:
[0051] In the formula, The spliced feature is obtained by upsampling deep features and concatenating them with shallow features along the channel dimension. This is a global average pooling operation used to obtain global statistics for each channel; Indicates the kernel size as (In this embodiment) One-dimensional convolution operations are used to capture local cross-channel dependencies; This represents the Sigmoid activation function, which maps the output to... Interval-based attention weight generation; This represents a 1×1 convolution dimensionality reduction operation, used to compress the number of channels; This represents the fusion feature at the current level.
[0052] 1.2 Frequency-Aware High-Frequency Detail Enhancement In generating the highest resolution fusion features After (resolution 480×270), a frequency-aware enhancement branch is introduced to strengthen the features of small targets. The specific steps are as follows: right Each channel independently performs a discrete cosine transform (DCT) to convert spatial domain features to the frequency domain, resulting in a spectrum. Through a learnable mask matrix of the same size as the spectrogram The spectrogram is weighted, and the mask matrix is initialized during training to tend to preserve the numerical distribution of high-frequency coefficients (initial values: low-frequency coefficients are set to 0.1, and high-frequency coefficients are set to 0.9). Performing inverse discrete cosine transform (IDCT) on the weighted spectrogram yields high-frequency detail enhancement features in the spatial domain; High-frequency detail enhancement features and the original Element-wise addition yields the enhanced shallow features. This is used for subsequent small target detection.
[0053] The enhancement process is represented as:
[0054] In the formula, This represents the highest resolution fusion feature; This is the discrete cosine transform; It is a learnable mask matrix; It is the inverse discrete cosine transform; This is the enhanced shallow layer feature.
[0055] Step 2: Cascaded Human Body and Component Inspection Two-stage cascaded detection is performed on the generated multi-scale feature maps. The first stage performs global human detection and outputs human bounding boxes. The second stage performs keypoint localization and accessory (helmet, seat belt) detection within each human bounding box, and uses graph convolutional networks to constrain the geometric topological relationships between keypoints and correct the position of occluded keypoints. The specific process is as follows: 2.1 Human bounding box detection The YOLOv5s detector is used to perform global human detection on multi-scale feature maps, outputting human bounding boxes with a confidence score greater than 0.5, and filtering out duplicate boxes with an overlap score greater than 0.7 to obtain the final human detection results.
[0056] 2.2 Preliminary location of key points and graph construction Within each human bounding box, preliminary coordinates of 17 human keypoints (defined in the COCO dataset) are obtained through heatmap regression, and these 17 keypoints are used as graph nodes. An adjacency matrix is defined based on the inherent connectivity of the human skeleton. If node With nodes If there is a physical connection between them (such as head-neck, shoulder-elbow), then ,otherwise .
[0057] 2.3 Construction of Projection Adaptive Adjacency Matrix Constructing the projection adaptive matrix Its elements Based on the installation tilt angle of the high-altitude camera (In this embodiment, the camera tilt angle is 60°) Calculation is performed for limb connections that represent vertical relationships (such as neck-waist, waist-knee). For other connections, .in As the initial adjustment factor, in this embodiment .
[0058] The original adjacency matrix is multiplied element-wise by the projected adaptive matrix to obtain the adjusted adjacency matrix. :
[0059] Towards Add identity matrix (Self-loop), and construct the normalized graph Laplace matrix. ,in for The degree matrix.
[0060] 2.4 Graph Convolution Topological Constraint Correction A three-layer graph convolutional network combined with a projection adaptive adjacency matrix is used to perform topological constraint correction on the keypoint coordinates, enabling occluded keypoints to infer their reasonable positions from connected visible nodes. The feature update method for each layer of the graph convolutional network is as follows:
[0061] In the formula, This is the adjacency matrix adjusted by the projection adaptive matrix; It is the identity matrix; for The degree matrix; For the first Layer node characteristics; For the first Layer-learnable weight matrix; It is the ReLU activation function; For the first Layer node characteristics.
[0062] Step 3: Violation determination by integrating key point semantics Using the corrected key point coordinates and the detection results of attachments, two types of violations are judged: helmet wearing detection and posture violation detection.
[0063] 3.1 Helmet Wearing Test Extracting the coordinates of key points (nose) on the head Coordinates of the center point of the safety helmet detection frame Calculate the Euclidean distance between the two. :
[0064] Calculate the continuous wearing tendency score The determination of whether a safety helmet is actually worn on the head is based on a combination of distance and vertical relationship:
[0065] The direction logic function used to determine the vertical relationship is:
[0066] In the formula, In this embodiment, the distance tolerance parameter is used. Pixel; The vertical difference between the center point of the safety helmet detection frame and the key point on the head; In this embodiment, parameters are used to control the steepness of the directional logic function. When the center of the safety helmet is higher than the critical points of the head in the vertical direction, A score close to 1 indicates a higher wearing tendency.
[0067] 3. When Greater than the dynamic judgment threshold If the helmet is worn correctly, it is considered properly worn; if a human body is detected but... If the threshold is not reached, and the minimum Euclidean distance between the wrist key point and the center of the helmet detection frame is lower than the preset close-range threshold (30 pixels in this embodiment), it is determined that the helmet is held / worn by hand, which is a violation of not wearing a helmet.
[0068] 3.2 Posture-related violation detection For three typical postural violations—lying down, climbing, and falling—a spatiotemporal graph convolutional network is used for action classification. The specific steps are as follows: Sixteen frames of images were continuously acquired, and 17 human body key points were independently extracted from each frame, forming a dimension of... The sequence of key spatiotemporal points; Constructing a spatiotemporal graph: Using 17 keypoints per frame as spatial nodes, temporal edges are established between the same keypoints in adjacent frames, and spatial edges are established between physically connected keypoints within a single frame. The weights of the spatial edges are dynamically determined based on the regression confidence of the heatmap of the two keypoints in that frame, to adaptively adjust the strength of topological constraints caused by occlusion.
[0069] In the formula, For the first Frame number Confidence level of heatmap regression for key points; For the first Frame number Confidence level of heatmap regression for key points; For the first Frame connection nodes With nodes Spatial edge weights.
[0070] 3. Input the spatiotemporal graph into a 2-layer spatiotemporal graph convolutional network. Extract the dynamic topological features of the action through alternating spatial graph convolution and temporal convolution operations. Classify and output the action category through a fully connected layer. Actions with a confidence score greater than 0.6 are judged as corresponding pose violations.
[0071] Step 4: Environmental Perception and Adaptive Parameter Adjustment The environmental perception module acquires real-time environmental parameters and camera installation parameters, and dynamically adjusts the violation judgment threshold and topological constraint weights in the graph convolutional network. The specific process is as follows: Calculate the average grayscale value for the current video frame. and local contrast , combined into a 2D lighting description vector; Combine the illumination description vector with the camera mounting pitch angle. Encoder fusion is performed, taking a mini fully connected network with two hidden layers (16 and 8 neurons in the hidden layers) as input, and outputting an adaptive regulator. and weight adjustment value ; Update the threshold for helmet wearing detection:
[0072] In the formula, As the baseline threshold, in this embodiment ; This is an adaptive adjustment factor, with a value range of [value range missing]. When the light is dim, Automatically increases and decreases the detection sensitivity to reduce false alarms; when there is sufficient light. Automatic reduction improves judgment sensitivity.
[0073] 4. Update the projection weights of vertical connections in the graph convolutional network:
[0074] In the formula, This is the initial adjustment factor; This is the weighted adjustment value, and its range is [value range missing]. The constraint strength of the vertical limb connection is dynamically adjusted according to lighting conditions and pitch angle to improve the positioning accuracy of key points under different installation angles.
[0075] Step 5: Structured Alarm Output When a violation event is detected, a structured alarm output process is executed, and the violation information is encapsulated and uploaded to the management platform: When a violation is triggered, immediately capture the current frame and one frame before and after it, for a total of three frames, as evidence of the violation. Draw human bounding boxes, safety helmet detection boxes, and key point connection diagrams after graph convolutional network correction on the captured images to generate key point annotation diagrams; The violation types are: not wearing a safety helmet, lying down, climbing, falling, time of occurrence (accurate to the second), camera number, Base64 encoded data of the captured image, Base64 encoded data of the key point annotation map, and violation confidence score, which are encapsulated into a structured data body in JSON format; The structured data is uploaded to the cloud management platform via a 4G wireless communication module. The management platform displays alarm information in real time and provides functions for querying historical violations and statistical analysis to assist safety management personnel in on-site verification and handling.
[0076] System Architecture of this Embodiment The system corresponding to the detection method described in this embodiment includes: a high-altitude camera, a multi-scale feature fusion preprocessing module, a cascaded human body and component detection module, an environmental perception and adaptive adjustment module, a violation determination module, a structured alarm output module, and a management platform.
[0077] The functions of each module are as follows: High-altitude camera: Deployed on tower cranes or high points to capture real-time video streams of the construction site at a resolution of 1920×1080 and a frame rate of 25fps; Multi-scale feature fusion preprocessing module: performs cross-layer feature recalibration fusion and frequency-aware high-frequency enhancement to generate multi-scale feature maps that take into account both near and far targets; Cascaded human body and component detection module: First, detect the human body bounding box, then locate key points within the box and detect attachments, and use graph convolutional networks combined with pitch angle to constrain the topological relationship of the skeleton; The environmental perception and adaptive adjustment module extracts the illumination description vector, combines it with the camera's pitch angle to output an adjustment factor, and dynamically corrects the judgment threshold and topological constraint weights. Violation determination module: Calculates helmet wearing tendency score to distinguish between proper wearing and handheld status; classifies postural violations through spatiotemporal volume integration; Structured alarm output module: captures images of frames before and after the violation, generates annotated images, encapsulates structured alarm data, and uploads it; Management platform: Receives and displays alarm information, and provides historical query and statistical analysis functions.
[0078] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0079] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. A construction site safety inspection method based on multi-scale feature fusion and key point localization, characterized in that, Includes the following steps: Step S1: Acquire the real-time video stream from the high-altitude camera, and perform multi-scale feature fusion preprocessing on the image through an improved lightweight backbone network to generate a multi-scale feature map. The multi-scale feature map is used to extract features from both distant small targets and close-range large targets. Step S2: Perform cascaded detection on the multi-scale feature map. The first stage performs global human detection and outputs human bounding boxes. The second stage performs key point localization and accessory detection within each human bounding box. During the key point localization process, graph convolutional networks are used to constrain the geometric topological relationship between key points, and the corrected key point coordinates and accessory detection boxes are output. Step S3: Using the corrected key point coordinates and attachment detection boxes, perform violation determination by fusing key point semantics, and output the violation type and its confidence score, including helmet wearing detection and posture violation detection. Helmet wearing detection is completed by calculating the spatial relationship between head key points and helmet detection boxes. Posture violation detection is completed by extracting human key point sequences from consecutive frames and inputting them into a spatiotemporal graph convolutional network for action classification. Step S4: Output a structured alarm based on the violation type and confidence score, and upload the violation type, time, location, captured image and confidence information to the management platform.
2. The construction site safety inspection method based on multi-scale feature fusion and key point localization according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Extract shallow and deep features with progressively decreasing resolution from the improved lightweight backbone network. Upsample the deep features to the same spatial size as the shallow features and then stitch them together along the channel dimension to obtain the stitched features. Step S12: Adaptive channel weighting is applied to the concatenated features using a cross-layer feature recalibration and fusion module. Channel statistics are obtained through global average pooling, and local cross-channel dependencies are captured using one-dimensional convolution. Attention weights are generated to recalibrate the concatenated features, and then 1×1 convolution is used to reduce the dimensionality and generate the fused features of the current layer. The fusion process is represented as follows: In the formula, This is a concatenated feature obtained by upsampling deep features and then concatenating them with shallow features along the channel dimension. For global average pooling, Indicates the kernel size as One-dimensional convolution operation, This represents the Sigmoid activation function. This represents a 1×1 convolution dimensionality reduction operation. This represents the fusion feature at the current level. Step S13: Generate the highest resolution fused features Then, a frequency-aware enhancement branch is introduced to... Each channel independently performs a discrete cosine transform to convert the spatial domain features to the frequency domain, resulting in a spectrum. Step S14: Use a learnable mask matrix of the same size as the spectrogram. The spectrum is weighted, and the mask matrix is initialized during training to a numerical distribution that tends to preserve high-frequency coefficients. Then, the inverse discrete cosine transform is performed on the weighted spectrum to obtain high-frequency detail enhancement features in the spatial domain. Step S15: Combine high-frequency detail enhancement features with the original Element-wise addition yields the enhanced shallow features. This is used for subsequent small target detection; The frequency-aware enhancement branch strengthens the high-frequency details of shallow features to improve the representation of small targets at a distance. The enhancement process is represented as follows: In the formula, For the highest resolution fusion features, For discrete cosine transform, A learnable mask matrix, It is the inverse discrete cosine transform. This is the enhanced shallow layer feature.
3. The construction site safety inspection method based on multi-scale feature fusion and key point localization according to claim 1, characterized in that, The specific method of cascaded detection in step S2 is as follows: The first stage uses the YOLOv5s detector to perform global human body detection on multi-scale feature maps and outputs the human body bounding box of each person; The second stage locates a preset number of human skeletal key points within each human body bounding box by heatmap regression, and at the same time runs the target detection branch to detect accessories such as safety helmets. Step S2 also includes the following steps: Step S21: Regress the data obtained from the heatmap The initial coordinates of key points are used as graph nodes, and an adjacency matrix is defined based on the inherent connection relationships of the human skeleton. If node With nodes If there is a physical connection ,otherwise ; Step S22: Construct the projection adaptive matrix Elements of the projection adaptive matrix Based on the installation tilt angle of the high-altitude camera Calculations are performed for limb connections representing vertical relationships. For other connections, ,in For adjustment factors; Step S23: Adjust the adjacency matrix to After adding self-loops, a normalized graph Laplace matrix is constructed; Step S24: The keypoint coordinates are topologically corrected using a multi-layer graph convolutional network combined with a projection adaptive adjacency matrix, enabling occluded keypoints to infer their reasonable locations from connected visible nodes. The feature update method for each layer of the graph convolutional network is as follows: In the formula, This is the adjacency matrix adjusted by the projection adaptive matrix. It is the identity matrix. for The degree matrix, For the first Layer node characteristics, For the first Layer-learnable weight matrix, For activation function, For the first Layer node characteristics.
4. The construction site safety inspection method based on multi-scale feature fusion and key point localization according to claim 1, characterized in that, In step S3, the helmet wearing detection includes the following steps: Step S31: Extract the coordinates of key points in the head Coordinates of the center point of the safety helmet detection frame Calculate the Euclidean distance between the head key points and the center point of the helmet detection frame. ; Step S32: Calculate a continuous wearing tendency score The determination of whether a safety helmet is actually worn on the head is based on a combination of distance and vertical relationship. If the wear is greater than the dynamic threshold, it is considered to be properly worn; if the human body is present but... If the threshold is not reached and the minimum Euclidean distance between the wrist key point and the center of the helmet detection frame is lower than the preset close-range threshold, it is determined that the helmet is not being worn. Wearing tendency score The calculation method is as follows: The direction logic function used to determine the vertical relationship is: In the formula, The distance is the Euclidean distance between the key point on the head and the center point of the safety helmet detection frame. For distance tolerance parameters, This is the vertical difference between the center point of the safety helmet detection frame and the key point on the head. The parameter controlling the steepness of the directional logic function is such that when the center of the helmet is higher than the key point of the head in the vertical direction, the value of the directional logic function approaches 1.
5. The construction site safety inspection method based on multi-scale feature fusion and key point localization according to claim 1, characterized in that, In step S3, the detection of posture-related violations includes the following steps: Step S33: Continuous data acquisition Frame images, each frame extracted independently Each individual's key points form a spatiotemporal key point sequence; Step S34: with each frame Each key point is a spatial node. Temporal edges are established between the same key points in adjacent frames, and spatial edges are established between key points with physical connections within a single frame to construct a spatiotemporal graph. The weight of the spatial edge is dynamically determined based on the regression confidence of the heatmap of the two key points in the frame, so as to adaptively adjust the topological constraint strength caused by occlusion. Step S35: Input the spatiotemporal graph into the spatiotemporal graph convolutional network, extract the dynamic topological features of the action through alternating spatial graph convolution and temporal convolution operations, and classify and output the action category through a fully connected layer; The method for determining the weights of spatial edges is as follows: In the formula, For the first Frame number The confidence level of the heatmap regression for each key point. For the first Frame number The confidence level of the heatmap regression for each key point. For the first Frame connection nodes With nodes Spatial edge weights.
6. The construction site safety inspection method based on multi-scale feature fusion and key point localization according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: When a violation event is triggered, capture three frames: the current frame and the frames before and after it. Step S42: Draw the human body bounding box, the attachment detection box, and the key point connection diagram after graph convolutional network correction on the captured image to generate a key point annotation map; Step S43: Encapsulate the violation type, occurrence time, camera number, encoded data of the captured image, encoded data of the key point annotation map, and violation confidence score into a structured data body; Step S44: Upload the structured data to the management platform via the wireless communication module.
7. The construction site safety inspection method based on multi-scale feature fusion and key point localization according to claim 1, characterized in that, Step S3 further includes obtaining current environmental parameters and camera installation parameters, and dynamically adjusting the judgment threshold and topological constraint weights in the graph convolutional network involved in the violation judgment in step S3, as follows: Calculate the average grayscale value for the current video frame. and local contrast , and combine them into a lighting description vector; The illumination description vector is compared with the pitch angle in the camera installation parameters. Encoding fusion is performed, and an adaptive adjustment factor is output through a miniature fully connected network. and weight adjustment value ; The decision threshold for helmet wearing detection and the topological constraint weights in the graph convolutional network are updated using adaptive adjustment factors and weight adjustment values. The environmental perception module automatically adjusts the judgment threshold based on lighting conditions and pitch angle. The judgment threshold is updated as follows: The environment perception module automatically adjusts the vertical connection projection weights based on lighting conditions and pitch angle. The projection weights are updated as follows: In the formula, The baseline threshold for helmet wearing detection. The updated judgment threshold, As an adaptive adjustment factor, As the initial adjustment factor, This is the weighted adjustment value. Install a tilt angle for the camera. This is the updated vertical connection projection weight.
8. A construction site safety detection device based on multi-scale feature fusion and key point localization, used to execute the method of any one of claims 1 to 7, characterized in that, include: High-altitude camera, multi-scale feature fusion preprocessing module, cascaded human body and component detection module, environmental perception and adaptive adjustment module, violation judgment module, structured alarm output module and management platform; The high-altitude camera, multi-scale feature fusion preprocessing module, cascaded human body and component detection module, violation judgment module, structured alarm output module, and management platform are sequentially connected in a unidirectional manner; the output of the environmental perception and adaptive adjustment module is unidirectionally connected to the inputs of the cascaded human body and component detection module and the violation judgment module, respectively.
9. The construction site safety detection device based on multi-scale feature fusion and key point localization according to claim 8, characterized in that, The high-altitude camera is deployed on a tower crane or at a high point to collect real-time video streams from the construction site; The multi-scale feature fusion preprocessing module is used to extract multi-scale features from video frames, fuse deep semantics and shallow details through cross-layer recalibration, and enhance high-frequency information using a frequency-aware enhancement branch to generate a multi-scale feature map that takes into account both near and far targets. The cascaded human body and component detection module first detects the human body bounding box, then locates key points and detects attachments within the box using a heatmap, and simultaneously uses a graph convolutional network combined with pitch angle constraints on the skeletal topology to correct the position of key points when occlusion occurs. The environmental perception and adaptive adjustment module is used to extract the average gray level and local contrast of the current frame to form a lighting description vector. After being fused with the camera pitch angle, it is output as an adjustment factor through a micro-network to dynamically correct the violation judgment threshold and the graph convolution vertical constraint weight.
10. The construction site safety detection device based on multi-scale feature fusion and key point localization according to claim 8, characterized in that, The violation determination module is used to calculate the wearing tendency score based on the spatial distance and vertical relationship between the key points of the head and the helmet frame, and to distinguish between qualified wearing and hand-held / waist-mounted wearing based on the wrist position; Construct a spatiotemporal graph for a continuous sequence of key points, and classify lying, climbing, and falling actions through spatiotemporal graph convolution. The structured alarm output module is used to capture the current frame and the frames before and after when a violation is triggered, draw the human body frame, the attachment frame and the key point connection annotation diagram, and encapsulate the violation type, time, camera number, image encoding and confidence level before uploading; The management platform is used to receive alarm data, provide functions such as displaying violation information, viewing and statistical analysis of marked diagrams, and assist safety management personnel in verification and decision-making.