A lightweight security hazard real-time detection method and system

Abstract

The application discloses a lightweight safety hazard real-time detection method and system, and belongs to the technical field of computer vision, deep learning and engineering safety management; the method comprises the following steps: multi-source image acquisition and preprocessing; a lightweight backbone network is used for multi-scale feature extraction of the image; cross-modal feature fusion of visible light and infrared images is realized based on an adaptive attention mechanism; a knowledge-guided detection head is constructed by using engineering safety specification knowledge, safety hazard detection is performed on the fused features, and a hazard category, a position and a confidence are output; hierarchical early warning is performed according to the hazard severity, and model adaptive optimization is performed through an edge-cloud collaborative mechanism; the application designs a lightweight network structure and a multi-modal fusion strategy in view of problems such as great light change, serious shielding and various hazard types in an engineering site, and realizes high-precision real-time detection on an edge device.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, deep learning and engineering safety management technology, and specifically relates to a lightweight real-time detection method and system for safety hazards. Background Technology

[0002] On-site safety management is of paramount importance in industries such as construction, rail transit, and energy and power. Traditional safety inspections mainly rely on manual patrols, which suffer from limited coverage, strong subjectivity, delayed response, and insufficient data utilization. Safety officers cannot monitor the construction site around the clock and in all aspects; inspection standards vary from person to person, making it easy to miss or misjudge, and there is often a significant time delay between the discovery and handling of potential hazards. In addition, historical inspection records are mostly stored in paper or spreadsheet format, making it difficult to systematically support risk prediction and management decisions.

[0003] In recent years, with the rapid development of deep learning technology, image recognition-based intelligent safety monitoring systems have gradually become a research and application hotspot. These systems deploy cameras at construction sites to automatically analyze video streams and identify common safety hazards such as personnel not wearing safety helmets, lack of edge protection, and intrusion into dangerous areas. However, existing technologies still have the following significant shortcomings: General object detection models have a huge number of parameters, such as detection networks based on ResNet-50 or VGG16, which are difficult to run in real time on edge devices with limited computing power. Engineering sites often do not have the conditions to deploy high-performance GPU servers. The lighting conditions at the construction site vary drastically, with strong light during the day, backlight at dusk, weak light at night, and even complete darkness alternating. In addition, severe weather such as rain, fog, and dust storms occur frequently. A single visible light image has extremely poor robustness under extreme lighting conditions, and most existing solutions rely solely on visible light sensors. Most existing detection models are data-driven black-box models, which lack effective integration with engineering safety specifications. There is a gap between their detection results and the actual needs of safety management. For example, they can only detect the distance between personnel and the edge, but cannot directly determine whether the specific requirements for edge protection in the "Technical Specification for Safety of High-Altitude Operations in Building Construction" are violated, making it difficult for the results to directly guide management decisions. Once deployed, the model is difficult to continuously optimize based on actual site conditions, cannot adapt to the dynamically changing environment and new types of hidden dangers at the engineering site, and lacks an effective feedback-learning loop.

[0004] Therefore, there is an urgent need for a lightweight real-time safety hazard detection solution that is tailored to the characteristics of complex engineering scenarios, balances detection accuracy and computational efficiency, integrates engineering safety standards, and has continuous self-learning capabilities. Summary of the Invention

[0005] This application aims to solve the aforementioned problems existing in the prior art and provide a lightweight method and system for real-time detection of safety hazards. To achieve the above objectives, the present invention provides a lightweight real-time safety hazard detection method, characterized by comprising the following steps: S1 Multi-Source Image Acquisition and Preprocessing: Simultaneously acquires construction site image data using visible light cameras and infrared thermal imaging equipment deployed at the construction site, and performs image enhancement and registration processing. S2 Multi-scale Feature Extraction: A lightweight backbone network is used to extract multi-scale features from the preprocessed image. The lightweight backbone network adopts a depthwise separable convolution and channel shuffling structure to reduce the amount of computation while maintaining the feature representation ability. S3 cross-modal feature fusion: Based on an adaptive attention mechanism, it dynamically adjusts the fusion weights of visible light image features and infrared image features to generate a fused feature map; S4 Knowledge-Guided Hazard Detection: Encodes engineering safety specifications into structured rules, constructs a knowledge-guided detection head, performs safety hazard target detection on the fused feature map, and outputs the hazard category, location, and confidence level; S5 graded early warning and feedback: Based on the severity of the hidden danger, it is graded to trigger the corresponding early warning mechanism, and the detection results are fed back to the edge for model adaptive optimization.

[0006] Furthermore, the image enhancement and registration processing in step S1 includes: A low-light enhancement algorithm based on Retinex theory is used to enhance the brightness of visible light images at night or under low-light conditions at engineering sites. A feature-point-based image registration algorithm is used to spatially align visible light and infrared images, eliminating pixel shifts caused by differences in device viewing angles.

[0007] Furthermore, the lightweight backbone network described in step S2 includes: The input layer receives the preprocessed multi-source images; Multiple stacked lightweight residual blocks, each containing depthwise separable convolutional layers, batch normalization layers, channel shuffling layers, and skip connections; The feature pyramid network layer is used to aggregate feature information at different scales.

[0008] Furthermore, the adaptive attention mechanism described in step S3 includes: Spatial attention branch: Calculates the importance weights of visible light and infrared feature maps in the spatial dimension; Channel attention branch: Evaluating the contribution of different characteristic channels to the hazard detection task; Dynamic fusion module: Adaptively adjusts the fusion ratio of two modal features based on scene lighting conditions.

[0009] Furthermore, the knowledge-guided detection head mentioned in step S4 includes: Hazard category coding layer: maps hazard types in engineering safety specifications into learnable semantic vectors; Rule constraint layer: Structured rules such as safe distance and wearing of protective equipment are introduced as detection constraints; Multi-task output layer: Simultaneously outputs the coordinates of the hazard boundary box, the category probability, and the violation severity score.

[0010] Furthermore, the tiered early warning system described in step S5 includes: Level 1 Warning: If a major safety hazard is detected, an audible and visual alarm will be triggered immediately, and safety management personnel will be notified. Level 2 warning: When a minor violation is detected, it is recorded and summarized into a report periodically; Level 3 warning: Minor anomalies detected, added to the daily inspection focus list.

[0011] Furthermore, it is characterized by including a model adaptive optimization step: Collect difficult samples during the edge detection process; Regularly upload difficult samples to the cloud for incremental model training; The updated model parameters are sent to the edge to enable continuous model optimization.

[0012] Furthermore, a lightweight real-time safety hazard detection system includes: Edge device layer: includes visible light cameras, infrared thermal imaging devices, and edge computing boxes, used for image acquisition and local inference; Network transport layer: 5G / 4G or industrial Ethernet are used to enable data communication between the edge and the cloud; Cloud-based analytics layer: This includes a model training platform, a knowledge base management system, and a big data analytics module, used for model optimization and knowledge updates; Application presentation layer: Provides real-time monitoring dashboards, mobile alert push notifications, and historical data query functions.

[0013] Furthermore, the edge computing box includes: Image preprocessing module: performs image enhancement and registration; Lightweight inference engine: Loads the optimized detection model and performs real-time inference; Local caching module: Stores recent detection data and difficult samples.

[0014] Furthermore, the knowledge base management system includes: Standards and regulations library: stores national and industry engineering safety standards; Hidden Danger Case Database: Collects historical engineering safety accident cases; Rule-based reasoning engine: Supports rule-based logic judgment of potential problems.

[0015] Beneficial effects Compared with the prior art, the outstanding advantages of the present invention are: By employing a depthwise separable convolution and channel shuffling structure, the number of model parameters is reduced by more than 70% compared to conventional residual networks, and the amount of floating-point operations is significantly reduced. It can achieve real-time inference of no less than 20 frames per second on edge computing boxes such as NVIDIA Jetson Xavier NX, meeting the stringent real-time requirements of engineering sites.

[0016] By fusing visible light and infrared thermal imaging information and using an adaptive attention mechanism to dynamically adjust the fusion weights of the two modalities, high detection accuracy can be maintained even in complex environments such as nighttime, heavy fog, and strong backlighting, significantly improving detection robustness.

[0017] By encoding engineering safety specifications into structured rules and learnable semantic vectors and integrating them into the detection head, the model's detection logic becomes closer to actual safety management needs. The output results not only include the target location and category, but also the violation degree score and rule compliance judgment, which can directly assist managers in making decisions.

[0018] Through the edge-cloud collaboration mechanism, the edge automatically collects difficult samples with low inference confidence, and the cloud periodically uses difficult samples and incremental data to perform incremental model training, and wirelessly distributes the updated model parameters to the edge, so as to achieve continuous adaptive evolution of the model without human intervention.

[0019] By classifying hazards into precise categories based on their severity, and by differentiating alarm triggers and response measures, we can avoid the "crying wolf" effect caused by a large number of low-level alarms, while ensuring that major hazards are dealt with in a timely manner, thereby improving the efficiency of safety management. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall system architecture provided in the embodiments of this application; Figure 2 A lightweight backbone network structure diagram provided in the embodiments of this application; Figure 3 A schematic diagram of the cross-modal feature fusion module provided in an embodiment of this application; Figure 4 This is a diagram of the knowledge-guided detection head structure provided in an embodiment of this application; Figure 5 This is a flowchart of edge-cloud collaborative optimization provided for embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application.

[0022] Example 1: like Figure 1 As shown in the figure, this embodiment provides a lightweight real-time safety hazard detection system, including an edge device layer, a network transmission layer, a cloud analysis layer, and an application display layer. The edge device layer is deployed at the engineering site and is the core of data acquisition and real-time inference. It includes: a visible light camera with a resolution of no less than 1920×1080 and a frame rate of 25fps, deployed in key construction areas such as entrances and exits, high-altitude work surfaces, and material storage areas, for acquiring high-definition color images; an infrared thermal imaging device with a resolution of 640×512, a temperature sensitivity of ≤0.05℃, and an operating wavelength of 8~14μm, used for target detection at night and in severe weather conditions, and can effectively capture human body heat sources and abnormal equipment heating; and an edge computing box equipped with an NVIDIA Jetson Xavier NX processor, whose GPU has 384 CUDA cores and 48 Tensor Cores, with an INT8 computing power of 21 TOPS, capable of processing 8 video streams simultaneously. The network transmission layer employs 5G / 4G wireless networks or industrial Ethernet to ensure reliable, low-latency data communication between the edge and the cloud. For high-bandwidth data such as video images, H.265 compression is used for transmission. The cloud analytics layer is deployed in a private or public cloud and includes a Kubernetes-based model training platform, a knowledge base management system, and a big data analytics module. The application presentation layer includes a real-time monitoring dashboard, a mobile alert push app, and a historical data query web page, supporting access from multiple terminals, including PCs and mobile devices.

[0023] Based on the above system, the method disclosed in this embodiment specifically includes steps S1 to S5, and more implementation details are provided in subsequent embodiments.

[0024] Step S1: Multi-source image acquisition and preprocessing. Visible light cameras and infrared thermal imaging devices acquire images synchronously, ensuring timestamp alignment accuracy better than 50ms via the NTP network time protocol. Image preprocessing mainly includes two sub-steps: visible light image enhancement and image registration. For visible light image enhancement, the Multi-Scale Retinex Band Color Restoration (MSRCR) algorithm based on Retinex theory is used. This algorithm estimates the image illuminance component through Gaussian filtering at multiple scales, separates the reflection component, and performs color restoration, effectively enhancing the brightness and detail in low-light areas. Specifically, the MSRCR processing formula can be expressed as: ,in Let F_k be the image of the i-th channel, and F_k be the Gaussian kernel with scale k. As the weight, K is set to 3, and the scales are set to 15, 80, and 250 pixels respectively. For color restoration, image registration employs a SIFT-based feature point matching algorithm. First, SIFT feature points are extracted from both the visible light and infrared images to generate descriptors. Nearest neighbor matching based on Euclidean distance is used, and mismatched points are eliminated using the RANSAC algorithm. The affine transformation matrix H is estimated, and the infrared image is mapped to the visible light image coordinate space, achieving pixel-level spatial alignment. After registration, the visible light image (RGB three channels) and the infrared image (single channel) are concatenated along the channel dimension to form a 4-channel input tensor. To further enrich the input information, the visible light image can also be converted to YCbCr space, and the luminance channel can be stacked with the infrared image to form a 2-channel input; the subsequent network structure can be adjusted accordingly.

[0025] Step S2: Multi-scale feature extraction. The preprocessed 4-channel image is input into a lightweight backbone network, the network structure of which is as follows: Figure 2 As shown, the specific implementation method will be detailed in Example 2. This network uses operations such as depthwise separable convolution and channel shuffling to quickly extract feature maps at 5 scales. It then performs top-down path aggregation through a feature pyramid network to output multi-scale fused feature map groups P3, P4, P5, P6, and P7, with downsampling ratios of 8, 16, 32, 64, and 128 times, respectively. The number of channels in each layer is uniformly set to 128 to balance feature representation and computational overhead.

[0026] Step S3: Cross-modal feature fusion. Since the network input in step S2 is a multimodal stitched image, the network implicitly learns the mixed features between modalities in both shallow and deep layers. However, to explicitly model the relationships between modalities and enhance the adaptability to the scene, this application integrates a cross-modal feature fusion module into one or more layers of the feature pyramid (typically layer P3). For example... Figure 3 As shown, this module receives two branches: visible light modal-specific feature maps extracted from the intermediate layers of the backbone network. and infrared modal-specific feature maps To decouple modal features, the initial features of each of the single-modal images before concatenation can be extracted using a lightweight convolutional head. However, to maintain efficiency, more details are given in Example 3. The adaptive attention mechanism... and Perform fusion to generate a fused feature map. Then it is sent back to the backbone network or directly transmitted to the detection head.

[0027] Step S4: Knowledge-guided vulnerability detection. The knowledge-guided detection head structure is as follows: Figure 4 As shown, its detailed implementation is described in Example 4. The detection head takes the fused feature map as input, uses the hazard category encoding layer to map typical hazard types in the engineering safety specifications into high-dimensional semantic vectors, and introduces structured rules such as safety distance thresholds and protective equipment wearing logic through the rule constraint layer. Finally, the multi-task output layer simultaneously predicts the target bounding box coordinates, category probability, and violation severity score. The violation severity score is a continuous value between 0 and 1, representing the severity level of the hazard; the closer to 1, the more severe the hazard.

[0028] Step S5: Tiered Early Warning and Feedback. Based on the hazard category and violation severity score output in Step S4, and combined with historical status, a tiered early warning is triggered. The early warning level classification and decision-making logic are detailed in Example 5. In addition to being displayed locally via an external audible and visual alarm connected to the edge computing box, the early warning information is also pushed to the cloud analysis layer and application display layer via the MQTT protocol, notifying safety management personnel immediately. Simultaneously, during the inference process, samples with confidence levels below a preset threshold (e.g., 0.6) or inconsistent with manual review results are automatically marked as difficult samples and stored in the local cache module as the data basis for model adaptive optimization. The relevant process is detailed in Example 5.

[0029] This embodiment describes the overall method and system coordination process from a macroscopic perspective. The technical details of this application will be further disclosed below with reference to more specific embodiments.

[0030] Example 2: This embodiment details the structure and specific parameters of the lightweight backbone network used in step S2. For example... Figure 2 As shown, the network is generally divided into three parts: an input convolutional layer, four stages of lightweight residual block groups, and a feature pyramid network. The input is a 4-channel image tensor with a resolution of 640×640. After passing through a standard convolutional layer with a stride of 2, a kernel size of 3×3, and an output channel size of 32, the feature map size becomes 320×320. Then it proceeds to stages one through four.

[0031] Each lightweight residual block's design incorporates ideas from ShuffleNet V2. The input feature map is divided into two groups along the channel dimension: one group undergoes a direct identity mapping; the other group is sequentially processed through a 1×1 pointwise convolution (output channels are half the input), a 3×3 depthwise separable convolution (preserving the number of channels), and another 1×1 pointwise convolution (the output channels are merged with the identity mapping portion to be consistent), using batch normalization and the ReLU6 activation function. After concatenating the two parts, a channel shuffling operation is performed to enhance information interaction between different groups. In this design, depthwise separable convolution significantly reduces computation, channel shuffling avoids information closure caused by grouped convolutions, and the identity mapping branch preserves the initial features, which is beneficial for gradient propagation. No additional SE attention modules are used within the block to maintain lightweight design.

[0032] The specific parameters for the four stages are shown in Table 1 below: Table 1 Parameters of each stage of the lightweight backbone network

[0033] The first residual block in each stage undergoes downsampling with a 3×3 depthwise separable convolution (stride of 2) added to the identity mapping branch, while the remaining blocks maintain the same feature map size. After four stages, four layers of feature maps, C2, C3, C4, and C5, are obtained, with 128, 256, 512, and 512 channels, respectively. To obtain high-level semantic features, a 3×3 convolution with a stride of 2 is added after C5 to obtain C6 (size 10×10, 512 channels), and then further processed by a 1×1 convolution with a stride of 2 to obtain C7 (size 5×5, 512 channels). {C3, C4, C5, C6, C7} are fed into the feature pyramid network for multi-scale fusion. The feature pyramid adopts a path aggregation network structure, transmitting semantic information from top to bottom through upsampling and lateral connections, and adding an additional path from bottom to top to transmit localization information. Finally, fused feature maps of five scales, P3 to P7, are generated, with all layers having a uniform channel count of 128. The P3 resolution is 80×80 and is used to detect small, potentially hazardous targets, such as improperly placed small tools; the P7 resolution is 5×5 and is used to detect large or full-area environmental risks, such as large areas of water accumulation or smoke.

[0034] The lightweight backbone network has approximately 7.2M parameters, while the ResNet-50 of the same depth has approximately 25.6M parameters. Using TensorRT for INT8 quantization inference on an NVIDIA Jetson Xavier NX, processing a single 640×640 image frame takes only about 12 milliseconds, meeting real-time requirements.

[0035] Example 3: This embodiment details the implementation of cross-modal feature fusion in step S3 and provides two variations of the adaptive attention mechanism. Overall, this module aims to dynamically integrate visible light feature maps. and infrared feature map Generate fused feature maps . and There are two ways to obtain the modality information: one is to concatenate the features into a 4-channel image during the network input stage. After passing through some backbone network layers, the resulting feature map naturally mixes the two modalities. However, for explicit guidance, the features of a certain intermediate layer can be separated into two feature tensors representing the modality through 1×1 convolution projection. The other is to extract the modality information using extremely shallow independent heads at the network front end. and Then, it is fed into the backbone network. To maintain overall lightweight design, this embodiment adopts the first approach and directly uses the feature map output from stage one of the backbone network as... and Based on the common foundation, each feature map is obtained by recalibrating through a learnable modal weight matrix, thus avoiding the introduction of too many parameters.

[0036] The first implementation of the adaptive attention mechanism is "dynamic fusion based on illumination intensity." For example... Figure 3 As shown, the global illumination intensity λ is first calculated using the average grayscale value of the visible light image. The visible light image is then converted to grayscale, and the mean grayscale value of all pixels is calculated and normalized to [0,1]. An illumination threshold θ = 0.3 is set (corresponding to low-light scenes). A spatial attention branch is then constructed: [The text abruptly ends here, so the translation stops as well.] and Dimensionality was reduced to a single channel using 1×1 convolutions, and a spatial weight map was generated using the sigmoid function. and Channel attention branch: for and Global average pooling is performed to obtain the channel description vector, which is then passed through two fully connected layers (dimensionality reduction ratio of 16) and a sigmoid function to generate the channel weights. and Then calculate the initial fusion features: The dynamic fusion weights are determined by the illumination intensity, and the formula is: .

[0037] Where α is the visible light feature fusion weight, and β is the infrared feature fusion weight. Under sufficient illumination, α approaches 1, and visible light features dominate; under low illumination, α approaches 0, and the weight of infrared thermal imaging features increases. Ultimately... This method is simple to calculate, responds directly to changes in global illumination, and is suitable for most day-night cycles.

[0038] The second implementation, "adaptive gating fusion based on a scene classifier," serves as a further optimization of this embodiment. A lightweight scene classifier (based on MobileNetV3-Small) is pre-trained to classify the current input image into three categories: "sunny daytime," "cloudy / foggy," and "nighttime." The classifier outputs a category probability vector p = [p1, p2, p3]. p is used as a gating signal, which is passed through a small network to generate modality fusion weights α and β, as well as adjustments to the internal parameters of the attention branch. For example, for the "nighttime" category, the spatial attention range of infrared features is increased because human body heat sources are more prominent in infrared images at night. This approach can perceive richer environmental semantics and improves the fusion robustness under extreme weather conditions. In actual deployment, different schemes can be selected based on the computing power of the edge devices. The first scheme has extremely low computational cost, while the second scheme slightly increases computation but achieves higher accuracy.

[0039] Regardless of the method used, the fused feature map F_fuse replaces the features of the corresponding layer in the original feature pyramid, or is added to the original features as a supplementary branch to further enhance the expression. Experiments show that this fusion method improves the average detection accuracy (mAP) by more than 8 percentage points in nighttime and hazy scenes.

[0040] Example 4: This embodiment describes in detail the structure of the knowledge-guided detection head in step S4, and how to integrate engineering safety regulations into the detection process. For example... Figure 4 As shown, the knowledge-guided detection head includes a hazard category encoding layer, a rule constraint layer, and a multi-task output layer. The detection head adopts an anchor-free design to further reduce computational load, but it is also compatible with anchor-based designs.

[0041] The hazard category coding layer organizes hazard types in engineering safety specifications into a hierarchical classification system. For example, based on standards such as the "Construction Safety Inspection Standard" (JGJ 59), 24 common hazards are identified, including: not wearing a safety helmet, not wearing a reflective vest, not wearing a safety belt while working at height, no guardrails at edges, no cover plates at openings, unauthorized hot work, personnel entering lifting and hoisting areas, and oxygen and acetylene cylinders being too close together. Each category is represented by a 128-dimensional learnable semantic vector. The initial values ​​of the vectors are pre-trained on a large number of safety specification texts using Word2Vec, ensuring that semantically similar hazards (such as "not wearing a safety helmet" and "not wearing a safety helmet correctly") are close in the vector space. In the detection head, the predicted category probability of each pixel on the feature map is calculated by similarity with this semantic vector matrix, rather than a simple fully connected classification, allowing the model to better understand the intrinsic relationships between different hazards.

[0042] The rule constraint layer is the core of knowledge guidance. It transforms structured safety rules into constraints that can be processed by neural networks, applying them to bounding box regression and violation severity scoring. Rule constraints are mainly divided into three categories: (1) Safety Distance Rules: According to construction safety regulations, workers at heights must maintain a safety distance of at least 1.5 meters from the edge, and it is strictly forbidden for anyone to stand under the crane arm. The distance between oxygen cylinders and acetylene cylinders must be no less than 5 meters. In implementation, firstly, using pre-calibrated camera intrinsic and extrinsic parameters and the assumed ground plane, the bottom center point of the detected target bounding box is back-projected onto the world coordinate system, and the Euclidean distance between the two is calculated. The camera calibration uses the Zhang calibration method to obtain the intrinsic parameter matrix K and distortion coefficients. Because the camera's perspective at the engineering site is approximately overhead, a homography transformation can be established. Map image coordinates to ground plane coordinates The coordinates were calculated by placing four markers with known world coordinates on the ground. The coordinates of the bottom center of the two-dimensional bounding box were then... After mapping, the world coordinates are obtained, and then the distance d is calculated. This distance is then compared with a normalized threshold. Compare and generate rule compliance scores. γ is the steepness coefficient, which causes the score to drop rapidly when the distance is less than a threshold. This score serves as an additional supervisory signal, and during training, the model is forced to learn distance-aware features through the loss function L_rule.

[0043] (2) Protective Equipment Rules: Safety helmets and safety belts must be worn when entering specific construction work areas. Based on the target detection results, if a personnel bounding box is detected, small equipment targets are detected within the bounding box, and a lightweight classifier is used to determine whether a safety helmet is present in the head area. If not, the violation score increases. This rule can be implemented by cascading an equipment verification subnetwork after the personnel detection head. This subnetwork also uses depthwise separable convolution, adding only a small amount of computation.

[0044] (3) Work process rules: such as requiring approval for high-risk nighttime operations and requiring fire extinguishers for hot work. These rules involve logical reasoning and are processed by the rule reasoning engine in the cloud-based knowledge base management system. After the edge detects a specific event (such as personnel using fire at night), it uploads the structured data to the cloud. The cloud-based reasoning engine combines the logical rules in the rule base (such as IF Night AND Hot Work THEN Check Approval Status) to perform reasoning and returns the results. The edge only executes visual rules with high real-time requirements, while complex logic is completed by the cloud, realizing edge-cloud division of labor.

[0045] The multi-task output layer simultaneously generates bounding box coordinates (composed of center point offset and width / height), class probability distribution, and violation severity score. The violation severity score s is output as a value between 0 and 1 via a sigmoid function. During training, a weighted combination of Smooth L1 loss and cross-entropy loss is used, along with rule constraint loss L_rule for joint optimization. This knowledge-guided design makes the model not only "visible" but also "understandable," allowing its output to be directly integrated with safety management processes.

[0046] Example 5: This embodiment details the tiered early warning mechanism in step S5 and the complete process of model adaptive optimization. The tiered early warning system dynamically generates response levels based on the severity of the hazard, avoiding excessive low-level alarms that could lead to managerial fatigue. The tiered decision-making logic comprehensively considers three dimensions: hazard category weight, violation severity score, and historical detection frequency. The specific tiering criteria are as follows: Level 1 Warning (Major Safety Hazard): This level addresses high-risk hazards that could lead to personal injury or significant property damage, such as fall risk, crane collision risk, fire and explosion risk, etc. The trigger condition is a confidence level of over 0.9 for the relevant hazard category and a violation severity score of over 0.8. Once triggered, the edge computing box immediately drives the on-site audible and visual alarms and broadcast system via the GPIO interface to issue a clear evacuation warning. Simultaneously, it pushes an emergency notification to the mobile apps of key personnel such as the safety director and project manager via the 5G network and automatically extracts 30-second video clips before and after the warning and uploads them to the cloud for permanent storage. If no manual confirmation is received within a preset time (e.g., 30 seconds), the system will automatically dial a pre-set emergency contact number.

[0047] Level 2 Warning (General Violation): This involves a violation of safe operating procedures that is not immediately fatal, such as not wearing a safety helmet or reflective vest. The trigger conditions are a category confidence score higher than 0.8 and a violation severity score higher than 0.5. The system records the event in both local and cloud databases, displays warning text on LED displays in the relevant area at the edge, and generates a violation statistics report every hour, which is then sent to the area safety officer.

[0048] Level 3 Warning (Minor Anomaly): This includes situations such as personnel briefly entering non-strictly restricted areas or slightly disorganized item placement. The trigger condition is a confidence level higher than 0.7. These alarms are only included in the daily inspection focus list and are displayed on the terminal to remind inspection personnel; they do not generate real-time alarm actions to avoid interfering with normal construction work.

[0049] In addition, the system supports security managers to confirm alarms or mark false alarms via a mobile app, and the confirmation results will be fed back into the model's hard sample mining process.

[0050] Model adaptive optimization is key to maintaining the long-term effectiveness of the system; its process is detailed below. Figure 5 The specific steps are as follows: (1) Hard Sample Collection: During local inference, the edge computing box automatically crops and saves the image regions corresponding to all candidate boxes with a detection confidence of less than 0.6 that are still retained after non-maximum suppression, as well as images marked as false positives or false negatives by the administrator, as hard samples. To protect privacy, sensitive areas such as faces can be blurred first. Storage uses H.265 intra-frame encoding compression, generating approximately 500 to 2000 samples per day.

[0051] (2) Uploading of difficult samples: During the off-peak construction period at the project site in the early morning (e.g., 3:00 AM), the edge device uses a 4G / 5G network to upload difficult samples in batches to the cloud analysis layer. Before uploading, hash verification and breakpoint resume are used to ensure data integrity. TLS encryption is used during transmission to meet the project data security requirements.

[0052] (3) Incremental training in the cloud: After receiving difficult samples, the cloud-based model training platform mixes them with the baseline dataset or uses only difficult samples for model fine-tuning. Incremental training uses a learning rate of 0.0001, momentum of 0.9, and batch size adjusted according to GPU memory. Training lasts for 5-10 epochs, and knowledge distillation loss is added. The original model is used as the teacher network to avoid catastrophic forgetting. After training, the model is validated on a new test set. If the mAP improvement exceeds 0.5%, it is considered an effective model.

[0053] (4) Model delivery and hot update: The INT8 quantized TensorRT engine file and model parameters are packaged into a differential update package and delivered to the edge via OTA. A model management daemon runs in the edge computing box. After receiving the update package, it verifies the signature, loads the new model in the background, and smoothly switches the inference thread to achieve hot updates without service interruption. After completion, a confirmation message is sent to the cloud.

[0054] (5) Federated learning extension: When this system is deployed at multiple engineering sites, the federated averaging algorithm can be used in the cloud to aggregate the model update gradients of each site, further improving the model's generalization ability while protecting the data privacy of each construction site.

[0055] Through this collaborative optimization mechanism, the model can continuously learn from new scenarios. After several months, the recognition rate of special hidden danger types that are difficult to identify in the early stages (such as the lack of new protective nets) has been greatly improved, truly achieving "the more you use it, the smarter it becomes".

[0056] Example 6: This embodiment further provides three application examples of typical complex engineering scenarios to demonstrate the adaptability of the method and the adjustment of the deployment scheme.

[0057] Scenario 1: Construction of a super high-rise building Construction sites of super high-rise buildings often involve frequent overlapping operations of tower cranes, numerous edges and openings, and complex vertical transportation of personnel and materials. Deployment plan: 24 visible light cameras and 12 infrared thermal imaging devices are deployed in key areas such as the core tube climbing formwork platform, the outer frame steel structure construction area, and the unloading platform. The infrared devices are primarily oriented against the light to overcome the effects of direct sunlight in the early morning and late afternoon. Edge computing boxes are deployed in floor-by-floor zones, with each unit connecting 4-6 video streams. For this scenario, the knowledge base is enhanced with safety distance rules: a spatiotemporal correlation rule is established between personnel detection within the tower crane boom's rotation range and the area below the suspended load. If personnel are detected below the suspended load, the violation score is immediately set to the maximum value. Visual detection rules for the integrity of floor edge guardrails are added, using edge detection and depth features to determine if guardrails are missing.

[0058] Scenario 2: Construction of Rail Transit Shield Tunnels The tunnel is extremely dark, relying mainly on artificial lighting. The air humidity is high, and dust is severe, making visible light cameras susceptible to light beam interference and scattering. Infrared thermal imaging is clearly superior in this environment due to the significant temperature difference between personnel and the environment. Deployment plan: One set of equipment is deployed every 50 meters behind the tunnel boring machine's operating room, in the segment assembly area, and along the transport track. Visible light and infrared cameras are used in pairs, equipped with explosion-proof protective covers. Due to the narrow space inside the tunnel, image distortion is significant. In step S1, wide-angle lens distortion correction is added, using Zhang's calibration method to estimate the distortion coefficient and perform distortion correction. A gas concentration threshold linkage is added to the knowledge rules: when the gas sensor value exceeds the threshold, the system automatically adjusts the sensitivity of suspected hot work and electrical spark detection to the highest level, lowering the violation severity scoring threshold to minimize the risk of explosion.

[0059] Scenario 3: Energy and Power Substation Substations contain numerous high-voltage devices, necessitating extremely strict safety distance controls. Infrared thermal imaging is used not only for personnel detection but also to monitor temperature anomalies in equipment joints and disconnectors. Deployment plan: Explosion-proof dual-light cameras (integrated visible light and infrared) are deployed in incoming and outgoing line bays, main transformer areas, and capacitor rooms. An edge computing box runs a specialized temperature analysis module, which, in addition to detecting personnel intrusion, performs real-time temperature segmentation of equipment areas. If the surface temperature of equipment exceeds a set threshold, an alarm for potential equipment malfunction is generated and integrated into a knowledge-guided detection framework. For example, if "transformer bushing overheating + personnel working at close range" is triggered simultaneously, it will be upgraded to a Level 1 warning. The rule base incorporates power safety work procedures, and the safety distance for live-line work is dynamically adjusted according to the voltage level. For example, the safety distance for 110kV is no less than 1.5 meters, and for 220kV, no less than 3 meters. These thresholds can be flexibly modified through configuration files. This application demonstrates the scalability and practicality of the proposed method in multiple scenarios.

[0060] Example 7: To verify the effectiveness of the technical solution presented in this application, a series of comparative experiments were conducted on real engineering sites and publicly available datasets. The test edge device was an NVIDIA Jetson Xavier NX with 16GB of memory. Dataset construction: A total of 120,000 images were collected from three different construction sites and one tunnel construction site, covering various conditions such as daytime, nighttime, foggy weather, and rainy weather. Sixteen types of hazards were labeled according to construction safety regulations, including not wearing a safety helmet, not wearing a safety belt, lack of edge protection, personnel intrusion, and unauthorized hot work. 100,000 images were used as the training set, 10,000 as the validation set, and 10,000 as the test set. Evaluation metrics included COCO-style mean accuracy (mAP@0.5:0.95), number of parameters, and single-frame inference time (ms).

[0061] Compared to baseline models such as YOLOv5s, YOLOX-s, and Faster R-CNN using ResNet-50+FPN, our proposed method is denoted as "Ours-Lite". Several ablation variants are also provided: a baseline using only visible light monomodality without knowledge guidance (Ours-Base), incorporating multimodal fusion (Ours-Multi), and further incorporating a knowledge-guided detection head (Ours-Lite). Quantization results are presented in Table 2 below.

[0062] Table 2 Performance Comparison of Test Sets

[0063] The results show that the lightweight backbone network of this application reduces the inference time to 12.4ms with a parameter count comparable to YOLOv5s (thanks to the efficiency of depthwise separable convolution and channel shuffling). After introducing multimodal fusion, the mAP is significantly improved to 46.3%, especially in the nighttime subset where accuracy increases from 32.1% to 44.8%, validating the benefit of infrared thermal imaging in low-light scenarios. Further addition of a knowledge-guided detection head achieves an mAP of 48.1%, and the introduction of violation severity scoring improves the NDCG index for hazard severity ranking by 7.2%. Furthermore, incremental training with difficult samples gradually improves the recognition rate of newly emerging hazard types after one month of continuous operation, without any accuracy degradation.

[0064] The experimental results above fully demonstrate that the method of this application achieves a good balance between lightweight, real-time performance and high precision, and significantly enhances its applicability in complex engineering scenarios through multimodal and knowledge-guided approaches, thus possessing practical engineering deployment value.

[0065] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Those skilled in the art, based on an understanding of the spirit and principles of this application, can make various improvements and modifications, which should also be considered within the scope of protection of this application.

Claims

1. A lightweight real-time safety hazard detection method, characterized in that, Includes the following steps: S1 Multi-Source Image Acquisition and Preprocessing: Simultaneously acquires construction site image data using visible light cameras and infrared thermal imaging equipment deployed at the construction site, and performs image enhancement and registration processing. S2 Multi-scale Feature Extraction: A lightweight backbone network is used to extract multi-scale features from the preprocessed image. The lightweight backbone network adopts a depthwise separable convolution and channel shuffling structure to reduce the amount of computation while maintaining the feature representation ability. S3 cross-modal feature fusion: Based on an adaptive attention mechanism, it dynamically adjusts the fusion weights of visible light image features and infrared image features to generate a fused feature map; S4 Knowledge-Guided Hazard Detection: Encodes engineering safety specifications into structured rules, constructs a knowledge-guided detection head, performs safety hazard target detection on the fused feature map, and outputs the hazard category, location, and confidence level; S5 graded early warning and feedback: Based on the severity of the hidden danger, it is graded to trigger the corresponding early warning mechanism, and the detection results are fed back to the edge for model adaptive optimization.

2. The method according to claim 1, characterized in that, The image enhancement and registration process in step S1 includes: A low-light enhancement algorithm based on Retinex theory is used to enhance the brightness of visible light images at night or under low-light conditions at engineering sites. A feature-point-based image registration algorithm is used to spatially align visible light and infrared images, eliminating pixel shifts caused by differences in device viewing angles.

3. The method according to claim 1, characterized in that, The lightweight backbone network mentioned in step S2 includes: The input layer receives the preprocessed multi-source images; Multiple stacked lightweight residual blocks, each containing depthwise separable convolutional layers, batch normalization layers, channel shuffling layers, and skip connections; The feature pyramid network layer is used to aggregate feature information at different scales.

4. The method according to claim 1, characterized in that, The adaptive attention mechanism described in step S3 includes: Spatial attention branch: Calculates the importance weights of visible light and infrared feature maps in the spatial dimension; Channel attention branch: Evaluating the contribution of different characteristic channels to the hazard detection task; Dynamic fusion module: Adaptively adjusts the fusion ratio of two modal features based on scene lighting conditions.

5. The method according to claim 1, characterized in that, The knowledge-guided detection head mentioned in step S4 includes: Hazard category coding layer: maps hazard types in engineering safety specifications into learnable semantic vectors; Rule constraint layer: Structured rules such as safe distance and wearing of protective equipment are introduced as detection constraints; Multi-task output layer: Simultaneously outputs the coordinates of the hazard boundary box, the category probability, and the violation severity score.

6. The method according to claim 1, characterized in that, The graded early warning mentioned in step S5 includes: Level 1 Warning: If a major safety hazard is detected, an audible and visual alarm will be triggered immediately, and safety management personnel will be notified. Level 2 warning: When a minor violation is detected, it is recorded and summarized into a report periodically; Level 3 warning: Minor anomalies detected, added to the daily inspection focus list.

7. The method according to claim 1, characterized in that, It also includes a model adaptive optimization step: Collect difficult samples during the edge detection process; Regularly upload difficult samples to the cloud for incremental model training; The updated model parameters are sent to the edge to enable continuous model optimization.

8. A lightweight real-time safety hazard detection system, used to implement the method described in any one of claims 1-7, characterized in that, include: Edge device layer: includes visible light cameras, infrared thermal imaging devices, and edge computing boxes, used for image acquisition and local inference; Network transport layer: 5G / 4G or industrial Ethernet are used to enable data communication between the edge and the cloud; Cloud-based analytics layer: This includes a model training platform, a knowledge base management system, and a big data analytics module, used for model optimization and knowledge updates; Application presentation layer: Provides real-time monitoring dashboards, mobile alert push notifications, and historical data query functions.

9. The system according to claim 8, characterized in that, The edge computing box includes: Image preprocessing module: performs image enhancement and registration; Lightweight inference engine: Loads the optimized detection model and performs real-time inference; Local caching module: Stores recent detection data and difficult samples.

10. The system according to claim 8, characterized in that, The knowledge base management system includes: Standards and regulations library: stores national and industry engineering safety standards; Hidden Danger Case Database: Collects historical engineering safety accident cases; Rule-based reasoning engine: Supports rule-based logic judgment of potential problems.

Images