A construction site violation detection system and method based on multi-stage feature refinement

Abstract

This invention discloses a construction site violation detection system and method based on multi-stage feature refinement, involving computer vision technology, and proposes this solution to address the problem of insufficient accuracy in existing technologies. The system consists of four modules working collaboratively: a parallel feature extraction backbone for training and optimizing the subsystem; a multi-stage training controller for dynamically switching data augmentation strategies according to preset rounds during training; a pose semantic association analyzer for detecting violations involving extremely small targets such as smoking, providing pose semantic auxiliary judgment functions in both the training and deployment phases; and a digital audit engine for transforming raw detection results into structured evidence that can be used by the regulatory platform and for adaptively optimizing the deployment threshold. Its advantage lies in achieving high-precision detection and structured audit output of construction site violations under complex working conditions by deeply integrating multi-stage feature refinement strategies and spatiotemporal semantic association mechanisms.

Description

Technical Field

[0001] This invention relates to computer vision and deep learning technologies, and in particular to a construction site violation detection system and method based on multi-stage feature refinement. Background Technology

[0002] With the continuous improvement of industrial safety standards, construction sites, as high-risk work environments, face real challenges in safety supervision, including complex environments, high personnel turnover, and the strong concealment of violations. Traditional video surveillance methods mainly rely on manual inspections, which suffer from insufficient real-time performance, poor continuity, susceptibility to subjective fatigue, and numerous blind spots. In recent years, target detection technology based on deep neural networks has been gradually applied to the field of industrial vision, but it still has the following shortcomings in complex construction site scenarios: First, the drastic changes in lighting and cluttered backgrounds, along with factors such as scaffolding, building materials, and equipment obstruction, can easily cause detection box shifts and false detections; Second, violations such as smoking often correspond to small targets with weak texture features, making it difficult to reliably identify them by relying solely on local pixel information; Third, while Mosaic data augmentation, commonly used during training, helps improve early generalization ability, it may introduce unnatural stitching boundaries in the later stages of convergence, thus affecting the model's localization accuracy on the real image distribution.

[0003] Therefore, there is an urgent need for a construction site violation detection system and method that can take into account both early generalization training and later real distribution refinement, and can combine spatiotemporal semantic relationships to identify hidden violations. Summary of the Invention

[0004] The purpose of this invention is to provide a construction site violation detection system and method based on multi-stage feature refinement, so as to solve the problems existing in the prior art.

[0005] The construction site violation detection system based on multi-stage feature refinement described in this invention consists of four modules working collaboratively: a parallel feature extraction backbone, a multi-stage training controller, a pose semantic association analyzer, and a digital audit engine.

[0006] The parallel feature extraction backbone is used to train and optimize the subsystem; The multi-stage training controller is used to dynamically switch data augmentation strategies according to preset rounds during the training process; The posture semantic association analyzer is used to detect violations of smoking-related small targets, and provides posture semantic auxiliary judgment function in both the training and deployment phases. The digital audit engine is used to transform raw detection results into structured evidence that can be used by the regulatory platform, and to perform adaptive optimization of deployment thresholds.

[0007] The construction site violation detection method based on multi-stage feature refinement described in this invention utilizes the construction site violation detection system to detect construction site violations.

[0008] The construction site violation detection system and method based on multi-stage feature refinement described in this invention have the advantage of achieving high-precision detection and structured audit output of construction site violations under complex working conditions by deeply integrating multi-stage feature refinement strategies with spatiotemporal semantic association mechanisms. This is mainly reflected in the following aspects: (1) To address the localization offset problem caused by the inconsistency between the augmented distribution and the real distribution in the later stage of training, the multi-stage feature refinement strategy turns off Mosaic data augmentation at the end of the model convergence period, so that the model can return to the real construction site image distribution to complete parameter fine-tuning, thereby reducing the spatial deviation of the prediction box and improving the localization accuracy of the bounding box.

[0009] (2) To address the problem that it is difficult to reliably identify extremely small target behaviors such as smoking by relying solely on local pixel features, a gesture semantic auxiliary judgment mechanism based on the topological association of "hand-mouth" was constructed. This mechanism combines local visual features with prior knowledge of human posture, thereby improving the reliability of identifying concealed high-risk behaviors and achieving an average accuracy of the smoking category. It reached 97.9%.

[0010] (3) To address the real-time deployment requirements at the edge of smart construction sites, this invention combines a lightweight YOLOv11n architecture with automatic mixed-precision inference technology, achieving low single-frame inference latency and high throughput on the RTX 4060 platform. This meets the real-time response requirements in scenarios with concurrent monitoring of multiple video streams. While ensuring system real-time performance, it improves the accuracy of violation detection and audit availability under dense and complex working conditions. It can be widely applied to automated safety supervision of smart construction sites and can provide technical reference for high-risk behavior recognition and video auditing in other industrial scenarios. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the process framework of the system described in this invention.

[0012] Figure 2 This is a precision-recall curve of the system described in this invention for each detection category. Detailed Implementation

[0013] The overall architecture of the construction site violation detection system based on multi-stage feature refinement described in this invention is as follows: Figure 1 As shown, the system consists of four modules working together: a parallel feature extraction backbone, a multi-stage training controller, a pose semantic association analyzer, and a digital audit engine. This enables a complete closed loop from raw video stream acquisition to automatic identification of violations, threshold filtering, and structured audit output.

[0014] The parallel feature extraction backbone is used to train and optimize the subsystem. The YOLOv11n object detection architecture is used as the base network, and enhanced feature extraction and cross-layer fusion structures improve the representation ability of targets at different scales. For the input 640×640 normalized image, multi-level features such as edges, textures, local shapes, human contours, clothing status, and scene semantics are extracted step-by-step to improve the comprehensive detection capability of targets such as safety helmets, reflective vests, and smoking objects.

[0015] The multi-stage training controller dynamically switches data augmentation strategies according to preset rounds during training to balance early-stage generalization and later-stage localization refinement. In the early to mid-stages of training, the Mosaic data augmentation strategy is enabled, generating composite samples by stitching together several images to increase the diversity of background composition, scale variation, and spatial distribution of the target. In the later stages of training, Mosaic data augmentation is disabled, allowing the model to re-fine-tune bounding box regression based on the real image distribution. By precisely controlling the lifecycle of the Mosaic data augmentation operator during training, a smooth transition from early-stage feature generalization to later-stage real-distribution refinement is achieved. The multi-stage feature refinement strategy effectively reduces bounding box drift under complex background interference, thereby improving bounding box regression quality and overall detection performance. The specific process is as follows: During the global search phase, i.e., Epochs 1-90, which is the early to mid-stage of model training, the system employs a Mosaic data augmentation strategy, which augments four randomly selected construction site images. The image is cropped and stitched together according to a random center point to generate an enhanced image. The expression is: ; in, Indicates the first Cropping and mapping operations on images under random center point constraints.

[0016] In the generalization learning phase during the early training stage, a stitching enhancement method is used to improve the scene complexity and target distribution richness of the training samples. Specifically, the stitched composite samples can introduce different background regions and lighting conditions, thereby simulating the actual situation of scaffolding occlusion, changes in ambient lighting, and the superposition of multiple construction interference factors at the training data level, thus improving the model's robustness to complex construction site scenarios. Simultaneously, the stitching enhancement method can also increase the target density in a single image, increasing the frequency of small-scale targets during training, allowing the model to more fully learn the edge information and local feature representation of small targets such as safety helmets and cigarette smoke, thereby enhancing the stability of small target detection. Furthermore, the training phase, combined with a preset initial learning rate and a linear warm-up mechanism, enables the model to smoothly complete parameter transfer based on pre-trained weights and converge more quickly to the feature space corresponding to the construction site violation recognition task.

[0017] During the distribution alignment phase, specifically Epochs 91-100, at the end of model training, the system executes the logic to disable the mosaic parameter control and augment the probabilities of the mosaic data. Setting it to 0 allows the model to regress from the augmented distribution to the true image distribution. At this stage, due to the elimination of non-physical texture and structural breaks caused by stitching boundaries, the predicted box center coordinates... With width and height The regression loss was further reduced, and the bounding box localization accuracy was improved accordingly. The control relationship is expressed as: ; ; This stage is used for fine-tuning and calibrating the model based on the real data distribution in the later stages of training. While using Mosaic data augmentation in the early stages can improve the model's generalization ability, it can also create unnatural edge structures and texture breaks in image stitching regions, introducing feature noise inconsistent with real surveillance scenes. By disabling the Mosaic data augmentation strategy in the later stages of training, the model can revert to the pixel distribution corresponding to real construction site camera images for parameter fine-tuning, thereby reducing the interference of non-physical edge features on the detection results. Simultaneously, the bounding box regression branch can perform further corrections based on the physical scale, aspect ratio, and positional relationship of the real targets, correcting the prediction box bias that may have accumulated in the early augmentation training, and ultimately improving the localization accuracy of target detection.

[0018] Under the above strategy, the model's loss function exhibits a clear two-stage convergence characteristic. The total loss function L is defined as: ; in, It is the bounding box regression loss. These are the bounding box weight coefficients. It is classification loss. It is the classification weight coefficient.

[0019] For bounding box localization loss The system adopts The loss measure is the overlap between the predicted bounding box and the ground truth bounding box, the distance between their center points, and the similarity in aspect ratio. To improve the boundary regression ability for small sample categories such as reflective clothing, this embodiment increases the weighting coefficients of the bounding boxes. The value was set to 7.5. Experimental results show that this strategy helps the train / box_loss and val / box_loss to converge stably, and improves the fit between the detection box and the target.

[0020] For classification loss The system employs Binary Cross-Entropy Loss (BCE Loss) to calculate the classification error for five target classes: helmet, not-helmet, reflective, not-reflective, and smoking. For extremely small targets such as smoking objects, this classification branch combines pose semantic association information to further enhance the recognition capability of scenarios where "hands are close to the mouth."

[0021] After the augmentation was disabled around round 90, both the model's loss descent rate and detection accuracy showed significant changes, indicating that the system entered a convergence phase primarily focused on refining the data based on the true distribution. Experimental data showed that after the augmentation was disabled in round 90, both train / box_loss and val / box_loss simultaneously triggered a second descent and steadily converged to below 0.6, demonstrating that the bounding box regression branch achieved higher localization accuracy under the true distribution.

[0022] Due to the high degree of fit between the predicted bounding box and the real target The model's average accuracy across all classes has been corrected. and The two indicators show a clear upward trend around the 90th round, indicating that the multi-stage feature refinement strategy has a significant effect on improving the overall detection performance.

[0023] To balance detection accuracy and training convergence efficiency, the system adopts the following training optimization parameter configuration.

[0024] AdamW Optimizer: The system employs the AdamW optimizer and uses weight decay to suppress overfitting risks on small sample classes. In this embodiment, the weight decay coefficient is set to 0.0005 to improve training stability.

[0025] Learning rate scheduling strategy: The initial learning rate is set to 0.01, combined with a linear warmup strategy of warmup_epochs=3.0 to reduce the risk of gradient oscillation in the early stage of training; the final learning rate multiplier is set to 0.01, so that the learning rate drops to 1% of the initial value at the end of training, which facilitates fine-grained adjustment of parameters near the optimal solution.

[0026] Batch configuration: On an NVIDIA GeForce RTX 4060 hardware environment, set the batch size to 16 to strike a balance between memory usage, training stability, and gradient update efficiency.

[0027] The described pose semantic association analyzer is used to detect smoking-related violations involving extremely small targets, providing pose semantic auxiliary judgment functions during both the training and deployment phases. Given that cigarette butt targets in construction site monitoring images are typically small in size, have weak features, and are easily affected by background interference, relying solely on local pixel information is insufficient to guarantee recognition stability. Therefore, by explicitly modeling the spatial topological relationship between the human hand region and the face or mouth region, prior constraints on human pose are introduced to form an auxiliary basis for identifying smoking behavior. During inference, the detected hand-adjacent mouth state is jointly judged with the cigarette butt candidate region to reduce false detections and improve the reliability of identifying concealed violations.

[0028] The digital audit engine transforms raw detection results into structured evidence that can be used by the regulatory platform and adaptively optimizes the deployment threshold. Through independent validation sets, the digital audit engine performs model.val evaluation and employs an F1 score maximization strategy to calculate the globally optimal confidence threshold. =0.393, which will serve as the basis for subsequent threshold screening and result publication. The F1 score is defined as: ; Where P represents precision and R represents recall.

[0029] The digital audit engine supports exporting data including category identifiers and bounding box center coordinates. ), bounding box width Boundary box height The system generates a structured violation metadata vector containing the target confidence score and timestamp t, which is used for subsequent event tracking, retrieval, and auditing. The final output includes the model's optimal weight file (best.pt), a violation data package in JSON / TXT format, and a real-time prediction image with visual labels, enabling integration with the smart construction site safety monitoring cloud backend and completing the business loop.

[0030] In this embodiment, the system performance is evaluated and threshold tuning is performed, such as... Figure 2As shown, the PR curves for each category reflect the relationship between precision and recall of the model under different confidence thresholds. Among them, the curve for the smoking category is closer to the upper right region, indicating that this category has better overall detection performance in this embodiment; the curve for the reflective clothing category is relatively far from the upper right region, indicating that this category is affected by the small number of samples and appearance changes, and its overall recognition performance is relatively low.

[0031] The system calculates different confidence thresholds based on the feedback from the validation set. recall rate With accuracy And solve Maximum score: ; when When the maximum value is obtained, the corresponding This refers to the globally optimal confidence threshold used during the deployment phase.

[0032] In this embodiment, the following experiment was conducted to verify the technical effectiveness of the system: A specialized dataset for smart construction site monitoring scenarios was constructed, covering complex backgrounds such as scaffolding obstruction, interference from building materials, and strong light projection. The dataset was divided into training, validation, and test sets to ensure an objective evaluation of the model's generalization ability.

[0033] The system exhibited excellent convergence characteristics during training. The training bounding box loss (train / box_loss) and validation bounding box loss (val / box_loss) converged steadily to below 0.6, demonstrating that the model's localization accuracy in complex backgrounds met the preset standard. Around the 90th epoch, due to the activation of the mosaic data augmentation disabling strategy, the model's... and The indicators showed a clear upward trend.

[0034] The construction site violation detection method based on multi-stage feature refinement described in this invention utilizes the construction site violation detection system to detect construction site violations.

[0035] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A construction site violation detection system based on multi-stage feature refinement, characterized in that, It consists of four modules working together: a parallel feature extraction backbone, a multi-stage training controller, a pose semantic association analyzer, and a digital audit engine. The parallel feature extraction backbone is used to train and optimize the subsystem; The multi-stage training controller is used to dynamically switch data augmentation strategies according to preset rounds during the training process; The posture semantic association analyzer is used to detect violations of smoking-related small targets, and provides posture semantic auxiliary judgment function in both the training and deployment phases. The digital audit engine is used to transform raw detection results into structured evidence that can be used by the regulatory platform, and to perform adaptive optimization of deployment thresholds.

2. The construction site violation detection system based on multi-stage feature refinement according to claim 1, characterized in that, The parallel feature extraction backbone uses the YOLOv11n target detection architecture as the base network, and enhances the representation ability of targets at different scales through enhanced feature extraction and cross-layer fusion structure; for the input normalized image, multi-level features are extracted step by step.

3. The construction site violation detection system based on multi-stage feature refinement according to claim 1, characterized in that, The multi-stage training controller enables the Mosaic data augmentation strategy during the early to mid-stages of training. This strategy generates composite samples by stitching together several images, thereby increasing the diversity of background combinations, scale variations, and spatial distribution of the target. During the later stages of training, the Mosaic data augmentation is turned off, allowing the model to re-fine-tune the bounding box regression based on the real image distribution.

4. The construction site violation detection system based on multi-stage feature refinement according to claim 3, characterized in that, The specific process of the multi-stage training controller is as follows: During the global search phase, the Mosaic data augmentation strategy was enabled, which involved combining four randomly selected construction site images. The image is cropped and stitched together according to a random center point to generate an enhanced image. The expression is: ； in, Indicates the first Cropping and mapping operations of images under random center point constraints; During the distribution alignment phase, the Mosaic data augmentation strategy is disabled, and the Mosaic data augmentation probability is increased. Set it to 0 to make the model regress from the augmented distribution to the true image distribution.

5. The construction site violation detection system based on multi-stage feature refinement according to claim 4, characterized in that, The loss function of the multi-stage training controller is defined as: ； Where L is the total loss function, It is the bounding box regression loss. These are the bounding box weight coefficients. It is classification loss. These are classification weight coefficients; For bounding box localization loss ,use The loss measures the degree of overlap between the predicted bounding box and the ground truth bounding box, the distance between their center points, and the similarity in aspect ratio. For classification loss The binary cross-entropy loss (BCE Loss) is used to calculate the classification error of each type of target.

6. The construction site violation detection system based on multi-stage feature refinement according to claim 1, characterized in that, The posture semantic association analyzer introduces prior constraints on human posture by explicitly modeling the spatial topological relationship between the human hand region and the face or mouth region, forming an auxiliary basis for identifying smoking behavior; during the reasoning process, the detected hand-adjacent mouth state is jointly judged with the cigarette butt candidate region.

7. The construction site violation detection system based on multi-stage feature refinement according to claim 6, characterized in that, The digital audit engine performs model.val evaluation through an independent validation set and calculates the globally optimal confidence threshold using an F1 score maximization strategy. This serves as the basis for subsequent threshold screening and result publication.

8. The construction site violation detection system based on multi-stage feature refinement according to claim 7, characterized in that, The F1 score is defined as: ； Where P represents precision and R represents recall.

9. The construction site violation detection system based on multi-stage feature refinement according to claim 8, characterized in that, The digital audit engine supports exporting data including category identifiers and bounding box center coordinates. ), bounding box width Boundary box height The system outputs a structured violation metadata vector containing the target confidence score and timestamp t, which is used for subsequent event tracking, retrieval, and auditing. The final output includes the model's optimal weight file best.pt, a violation data package in JSON / TXT format, and a real-time prediction image with visualization labels.

10. A method for detecting construction site violations based on multi-stage feature refinement, characterized in that, The construction site violation detection system as described in any one of claims 1-9 is used to detect construction site violations.

Images