Intelligent construction site violation identification and alarm method and system based on double deduplication and consistency check

Abstract

The application discloses a smart construction site violation identification and alarm method and system based on double deduplication and consistency verification, relates to the technical field of smart construction site safety management, and comprises the following steps: acquiring a real-time video stream and performing preprocessing, inputting an image frame after preprocessing into a pre-constructed lightweight target detection model to perform real-time inference, outputting a preliminary detection result, and generating original violation structured alarm data when a triggering condition is met; performing double deduplication determination on the original violation structured alarm data, and counting valid alarm data remaining after double deduplication; performing secondary rejudgment on the valid alarm data through a cloud multi-modal large model, and outputting a structured rejudgment result; performing consistency verification on the rejudgment result and the preliminary detection result, confirming a violation according to a consistency verification result, and pushing an alarm. The application improves the accuracy of smart construction site violation identification.

Description

Technical Field

[0001] This invention relates to the field of smart construction site safety management technology, specifically to a smart construction site violation identification and alarm method and system based on dual deduplication and consistency verification. Background Technology

[0002] With the continuous expansion of infrastructure construction in my country, construction site safety management faces challenges such as a large number of workers, complex construction environments, and dispersed safety hazards. Traditional methods relying on on-site inspections by safety officers have inherent limitations, including limited manpower, insufficient inspection frequency, and difficulty in achieving 24 / 7 coverage. In recent years, with the rapid development of computer vision and deep learning technologies, smart construction site violation identification systems based on video analytics have gradually become an important tool for construction site safety management.

[0003] Currently, mainstream smart construction site violation detection systems generally adopt a solution of deploying lightweight target detection models on edge AI computing devices. This solution identifies unsafe behaviors such as not wearing a safety helmet, not wearing reflective clothing, and smoking by performing frame-by-frame inference on real-time video streams captured by cameras, and triggers alarms in real time. It is currently the most widely used technical approach in the field of industrial vision inspection. However, edge models are limited by the number of model parameters and the distribution of training data. In complex construction scenarios such as drastic changes in lighting, dense crowds obscuring objects, and distant targets, their feature extraction capabilities are significantly insufficient, easily generating a large number of false alarms. More importantly, each detection makes independent decisions and lacks a review mechanism. When the model detects a suspected violation but the confidence level is low, the image is blurry, or the target is obscured, the system cannot proactively obtain higher-resolution images or richer perspective information to assist in the judgment. This results in a large number of blurry alarms being directly pushed, increasing the review burden on safety officers.

[0004] To address the issue of insufficient accuracy in edge-based small models, the industry has proposed a collaborative solution that combines real-time detection with lightweight small models with in-depth analysis using multimodal large models. However, these solutions primarily focus on optimizing the detection capabilities of small models and the analytical capabilities of large models. There is a lack of structured comparison and decision-making mechanisms between the detection results of small models and the re-judgment results of large models. The same violation is detected multiple times in consecutive video frames within a short period of time, generating a large number of redundant alarms. This leads to alarm fatigue among security personnel, causing them to overlook truly important violations. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, this application proposes the following technical solution: In a first aspect, embodiments of this application provide a smart construction site violation identification and alarm method based on dual deduplication and consistency verification, including: The system acquires and preprocesses real-time video streams, inputs the preprocessed image frames into a pre-built lightweight target detection model for real-time inference, outputs preliminary detection results, and generates original structured alarm data for violations when triggering conditions are met. Perform double deduplication on the original structured violation alarm data and count the remaining valid alarm data after double deduplication. The valid alarm data is re-evaluated using a cloud-based multimodal large model, and a structured re-evaluation result is output. The re-judgment result is checked for consistency with the preliminary detection result. Based on the consistency check result, the violation is confirmed and an alarm is triggered.

[0006] In one possible implementation, the process of acquiring and preprocessing a real-time video stream, inputting the preprocessed image frames into a pre-built lightweight target detection model for real-time inference, outputting preliminary detection results, and generating original structured violation alarm data when triggering conditions are met includes: The system continuously pulls real-time video streams from construction site monitoring cameras using the RTSP / ONVIF protocol, and decodes them at a preset frame rate to obtain continuous image frames. After scaling the image frames to the input standard size, the image pixel values ​​are normalized, and color gamut conversion, random cropping, and flip noise reduction preprocessing are performed simultaneously. The calculation formula is as follows: in, These are the original image pixel values. These are the normalized pixel values; The preprocessed image frame is input into the lightweight object detection model, and the output includes the object detection box, the category of the illegal object, and the prediction confidence. The preliminary detection results are used to determine violations based on preset category confidence thresholds, generating original structured violation alarm data.

[0007] In one possible implementation, the step of inputting the preprocessed image frame into a lightweight object detection model and outputting preliminary detection results including object detection boxes, violation object categories, and prediction confidence scores includes: The preprocessed image frame is input into the lightweight target detection model, and multi-scale feature extraction is completed through the model backbone network and feature pyramid network. The detection box coordinates, category prediction scores and target categories of each target in the image are output. Non-maximum suppression is performed on all detection boxes output by the model. A cross-union ratio (CURRR) threshold is set, and the CURRR of any two detection boxes is calculated. Redundant detection boxes with an overlap exceeding the CURRR threshold are removed, and the optimal detection box is retained. The calculation formula is as follows: ; in, There are two detection boxes to be compared. The area of ​​the intersection of the detection boxes. To detect the area of ​​the union of the bounding boxes, Two detection boxes The intersection and union ratio; For each detection box retained after nonmaximum suppression, the Sigmoid activation function is used to normalize the category prediction score output by the model to obtain the prediction confidence of each violation category. The output includes preliminary detection results such as the coordinates of the target detection box, the category of the illegal target, and the corresponding prediction confidence.

[0008] In one possible implementation, the step of determining the violation based on the preliminary detection result according to a preset category confidence threshold and generating original structured violation alarm data includes: Set different category confidence thresholds, and compare the predicted confidence in the preliminary detection results with the preset category confidence thresholds; When the predicted confidence level is greater than or equal to the preset category confidence level threshold, it is determined to be a suspected violation; When the predicted confidence level is less than the preset category confidence level threshold, it is determined that there is no violation. The calculation formula is as follows: in, Predict scores for the output categories. It is the Sigmoid activation function. The prediction confidence level is for the corresponding violation category c. For category c, a pre-set confidence threshold is set. This indicates that the violation was suspected. This indicates that there was no violation. After identifying a suspected violation, the current valid video frame is captured as an alarm image, and original structured alarm data containing device IP, alarm timestamp, violation type, detection confidence, detection box coordinates, and construction site location is generated simultaneously.

[0009] In one possible implementation, the original structured violation alarm data undergoes double deduplication, and the remaining valid alarm data after double deduplication is counted, including: Obtain the structured fields of each alarm in the original structured violation alarm data, and construct a cache key based on the device identifier, violation type, and location information. The calculation formula is as follows: in, For the camera device's IP address, This is a traffic violation alarm type. This serves as a location identifier for the logical area of ​​the construction site. It is a hash digest algorithm. Use a unique key for caching; Within a preset time window, historical alarm records with the same cache key are retrieved. If a historical alarm record with the same cache key exists within the preset time window, it is determined as a physical space duplicate hit; otherwise, it is determined as a physical space miss. The calculation formula is as follows: in, This is the current alarm timestamp. The most recent alert timestamp in the history of the same key. For physical deduplication time window, For repeated hits in physical space, Physical space not hit; Extract the visual feature vector of the current alarm image only if the physical space is repeatedly hit; The cosine similarity is calculated based on the visual feature vector and the feature vector of historical alarm images within a preset time window; When the maximum similarity exceeds the preset similarity threshold, it is determined to be a semantic duplicate hit; The current alarm is judged as a duplicate alarm and filtered out only when both physical space duplicate hits and semantic duplicate hits are true; otherwise, the current alarm is judged as valid alarm data and retained.

[0010] In one possible implementation, the current alarm is judged as a duplicate alarm and filtered out only when both physical space duplicate hits and semantic duplicate hits are true; otherwise, the current alarm is judged as valid alarm data and retained using the following calculation formula: in, For physical space repeated hit markers, For image semantic repetition hit tagging, To mark repeated alarms as a total flag, These are the normalized feature vectors of the two alarm images. Normalize the eigenvectors using the L2 norm. This represents the maximum cosine similarity between the current alarm image and all historical alarm images within the time window. A threshold is preset for semantic similarity.

[0011] In one possible implementation, the secondary review of the valid alarm data using a cloud-based multimodal large model to output a structured review result includes: The alarm images and preset dynamic scene prompts corresponding to the violation type in the valid alarm data are encapsulated into a review request and sent to the cloud multimodal large model service interface through the side-side large model call proxy. After receiving the review request, the cloud-based multimodal large model extracts high-dimensional semantic features from the alarm image using a visual encoder, performs semantic encoding on the prompt words using a text encoder, and completes cross-modal alignment of the image and text. The calculation formula is as follows: ; in, For image semantic feature vectors, For the semantic feature vector of the prompt word, The function calculates cosine similarity. Score for image and text alignment; The cloud-based multimodal large model combines prior knowledge of the construction site scenario to perform multi-dimensional semantic reasoning of violations, outputting the re-judgment confidence and no-violation confidence for each violation category. The calculation formula is as follows: in, This represents the prediction confidence level for the corresponding violation category c; For the first The original inference score for each violation category, For the first Confidence level of large-scale model re-judgment for similar violations; The above output is organized into a structured review result according to a preset format.

[0012] In one possible implementation, the consistency verification between the re-judgment result and the preliminary detection result is performed, and the violation is confirmed and an alarm is pushed based on the consistency verification result, including: Extract the violation existence label, violation category, confidence level and semantic explanation text from the re-judgment result, and compare them one by one with the violation target category and prediction confidence level in the preliminary detection result in multiple dimensions. If all dimensions are consistent, the consistency check is considered to have passed, the current alarm is confirmed as a real violation, and an alarm is automatically pushed. If any dimension comparison result is inconsistent, it is determined that the consistency check has failed, and the current alarm and end-to-end decision data are transferred to the manual review queue for manual judgment.

[0013] In one possible implementation, if any dimension comparison result is inconsistent, the consistency check is determined to have failed, and the current alarm and end-to-end decision data are transferred to a manual review queue for manual judgment, including: Alarms and end-to-end decision data for failed consistency checks are pushed to the manual review interface for centralized display. The final judgment result is received by the safety officer through the manual review interface; When the final judgment result is a false alarm, the alarm evidence and end-to-end decision data of the corresponding alarm will be associated and stored in the difficult case library; When the accumulated amount of the difficult example library reaches the trigger condition, the training process is automatically started to incrementally update the lightweight target detection model; When the final determination result is a genuine violation, a corresponding alarm will be pushed and confirmed. When the final determination result is uncertain, the corresponding alarm will be marked as pending, and complete decision-making chain data will be retained for subsequent traceability and review.

[0014] Secondly, embodiments of this application provide a smart construction site violation identification and alarm system based on dual deduplication and consistency verification, comprising: The acquisition module is used to acquire real-time video streams and preprocess them. The preprocessed image frames are input into a pre-built lightweight target detection model for real-time inference, output preliminary detection results, and generate original structured alarm data for violations when trigger conditions are met. The dual deduplication determination module is used to perform dual deduplication determination on the original structured violation alarm data and count the remaining valid alarm data after dual deduplication. The re-judgment module is used to perform a second re-judgment on the valid alarm data through a cloud-based multimodal large model and output a structured re-judgment result; The violation confirmation and alarm module is used to verify the consistency between the re-judgment result and the preliminary detection result, and to confirm the violation and push an alarm based on the consistency verification result.

[0015] Compared with the prior art, the beneficial effects of this application are as follows: This application constructs a four-level progressive judgment architecture, introducing new information dimensions at each level to progressively reduce the number of alarms. The first level tolerates false alarms to achieve zero false negatives; the second level eliminates spatiotemporal redundancy without introducing false negatives; the third level utilizes semantic understanding to improve accuracy; and the fourth level uses human experience as a fallback. The information gain at each level ensures that the certainty of the judgment increases progressively. Simultaneously, it employs a strict AND coupling logic of physical deduplication and semantic deduplication, determining duplication only when both are true, effectively avoiding false filtering or omissions caused by a single dimension. The duplicate alarm filtering rate reaches 30%–50%, significantly reducing the frequency of large-scale cloud model calls. In the consistency verification stage, this application expands model comparison from simple Boolean judgment to a four-dimensional structured verification, automatically confirming alarms and directly pushing them, transferring inconsistent alarms to manual review, and automatically aggregating false alarm data into a difficult case library, maximizing the efficiency of automated processing. Ultimately, after the difficult example library accumulates and triggers incremental training, the model completes iteration through grayscale verification and lossless hot loading, amplifying the optimization benefits step by step along the cascade propagation formula, causing the amount of manual review to continuously decrease, forming a self-enhancing closed loop that simultaneously improves detection accuracy and automation rate. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a smart construction site violation identification and alarm method based on dual deduplication and consistency verification, provided for an embodiment of this application; Figure 2 This is a schematic diagram of a smart construction site violation identification and alarm system based on dual deduplication and consistency verification, provided as an embodiment of this application. Detailed Implementation

[0017] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0018] Figure 1 A flowchart illustrating a smart construction site violation identification and alarm method based on dual deduplication and consistency verification, provided in this application embodiment, is shown below. Figure 1 This embodiment of a smart construction site violation identification and alarm method based on dual deduplication and consistency verification includes: S101: Acquire real-time video stream and preprocess it. Input the preprocessed image frames into a pre-built lightweight target detection model for real-time inference, output preliminary detection results, and generate original structured alarm data for violations when the triggering conditions are met.

[0019] In this embodiment, the edge device continuously pulls the real-time video stream from the construction site monitoring camera via the RTSP / ONVIF protocol, decodes it at a preset frame rate of 25fps to obtain continuous image frames, performs standardized preprocessing on the original image, uniformly scales it to the input standard size of 640×640, normalizes the image pixel values, and simultaneously performs color gamut conversion, random cropping, and flip noise reduction preprocessing to eliminate interference factors such as lighting and image distortion. The calculation formula for pixel normalization is: in, These are the original image pixel values. These are the normalized pixel values.

[0020] The preprocessed image frames are input into the lightweight YOLOv11n object detection model. Multi-scale feature extraction is performed through the model's backbone network and feature pyramid network. The model outputs the bounding box coordinates, category prediction scores, and object categories for each object in the image. Non-maximum suppression is performed on all detection boxes output by the model. A cross-union ratio (CURRR) threshold is set, and the CURRR of any two detection boxes is calculated. Redundant detection boxes with an overlap exceeding the CURRR threshold are removed, and the optimal detection box is retained. The calculation formula is as follows: ; in, There are two detection boxes to be compared. The area of ​​the intersection of the detection boxes. To detect the area of ​​the union of the bounding boxes, Two detection boxes The intersection-union ratio (IUU) is used to normalize the category prediction scores of the model output by the Sigmoid activation function for each detection box retained after nonmaximum suppression, so as to obtain the prediction confidence of each violation category. The output includes the target detection box coordinates, the violation target category and the corresponding prediction confidence.

[0021] Based on preset confidence thresholds for different categories, the preliminary detection results are used to determine violations. To achieve a high recall rate, the base threshold is... Set to 0.5, which can be lowered to 0.3 depending on the scenario requirements. For more difficult smoking violation identification, set the specific threshold to 0.6.

[0022] The predicted confidence level in the preliminary detection results is compared with a preset category confidence threshold. When the predicted confidence level is greater than or equal to the preset category confidence threshold, it is determined as a suspected violation; when the predicted confidence level is less than the preset category confidence threshold, it is determined as no violation. The calculation formula is as follows: in, Predict scores for the output categories. It is the Sigmoid activation function. The prediction confidence level is for the corresponding violation category c. For category c, a pre-set confidence threshold is set. This indicates that the violation was suspected. If no violation is found, and a suspected violation is identified, the current valid video frame is captured as an alarm image. Simultaneously, original structured alarm data containing device IP, alarm timestamp, violation type, detection confidence, detection box coordinates, and construction site location is generated and uploaded to the edge-side collaboration platform in real time via the MQTT protocol.

[0023] S102, perform double deduplication on the original structured violation alarm data, and count the remaining valid alarm data after double deduplication.

[0024] In this embodiment, after the edge platform receives the original structured alarm data of violations from the end side, it starts a dual deduplication engine. It filters redundant alarms by following the core rules of physical coarse filtering, semantic fine filtering, and AND logical coupling. It accurately removes duplicate alarms from the two orthogonal dimensions of time and space and semantics, and only retains new and valid alarms to enter the subsequent review process, which greatly reduces the cloud computing power consumption.

[0025] When performing physical space deduplication, the structured fields of each alarm in the original structured violation alarm data are obtained. A cache key is constructed based on the device identifier, violation type, and location information. Fast deduplication is achieved through microsecond-level retrieval in Redis. The calculation formula is as follows: in, For the camera device's IP address, This is a traffic violation alarm type. This serves as a location identifier for the logical area of ​​the construction site. It is a hash digest algorithm. Use a unique key for caching.

[0026] Within a preset time window, historical alarm records with the same cache key are retrieved. If a historical alarm record with the same cache key exists within the preset time window, it is determined as a physical space duplicate hit; otherwise, it is determined as a physical space miss. The calculation formula is as follows: in, This is the current alarm timestamp. The most recent alert timestamp in the history of the same key. For physical deduplication time window, For repeated hits in physical space, Physical space not hit.

[0027] Semantic similarity deduplication is performed only when there is a duplicate hit in physical space, avoiding unnecessary computational waste. The visual feature vector of the current alarm image is extracted, and cosine similarity is calculated based on the visual feature vector and the feature vectors of historical alarm images within a preset time window. This achieves accurate duplicate matching at the content dimension. When the maximum similarity exceeds a preset similarity threshold, it is determined to be a semantic duplicate hit. Only when both physical and semantic duplicate hits are true is the current alarm classified as a duplicate and filtered out; otherwise, the current alarm is considered valid alarm data and retained. This effectively avoids the false filtering and missed filtering problems of single-dimensional deduplication. The calculation formula is as follows: in, For physical space repeated hit markers, For image semantic repetition hit tagging, To mark repeated alarms as a total flag, These are the normalized feature vectors of the two alarm images. Normalize the eigenvectors using the L2 norm. This represents the maximum cosine similarity between the current alarm image and all historical alarm images within the time window. A preset threshold is set for semantic similarity. When If the alarm is repeated, it is considered a duplicate alarm and discarded directly; otherwise, it is considered a new alarm and pushed to the cloud-based big data model for review.

[0028] This logic covers four types of judgment scenarios: First, if there is no physical match, a new alarm is directly determined. Second, if there is a physical match but no semantic match, it is determined as an independent violation event. If there are different types of violations in the same area, the alarm is retained. Third, if there is a double match, it is determined as a redundant alarm and is filtered and discarded. Fourth, if there is no physical match, semantic calculation is skipped to save computing power.

[0029] The formula for calculating the percentage of repeated alarms is: Where N1 is the total number of original alarms on the terminal side, and N2 is the total number of new alarms remaining after double deduplication. In actual operation of this solution... Maintaining a stable level of 30%-50% can significantly reduce the frequency of subsequent large model calls.

[0030] S103 performs secondary review of valid alarm data through a cloud-based multimodal large model and outputs structured review results.

[0031] In this embodiment, the alarm images in the valid alarm data and the preset dynamic scene prompts corresponding to the violation type are encapsulated into a review request. The request is sent to the cloud multimodal large model service interface through the side-side large model call agent. It includes the original high-definition alarm image, preprocessed thumbnail, device / location / time structured metadata, and dynamic scene-specific prompts adapted to different violation types. The large model is clearly required to output standardized structured results.

[0032] After receiving the review request, the cloud-based multimodal large model extracts high-dimensional semantic features from the alarm image using a visual encoder, performs semantic encoding on the prompt words using a text encoder, and completes cross-modal alignment of the image and text. The calculation formula is as follows: ; in, For image semantic feature vectors, For the semantic feature vector of the prompt word, The function calculates cosine similarity. To score for image-text alignment, The higher the confidence level, the better the semantic matching between the image and text, and the higher the accuracy of the re-judgment. The cloud-based multimodal large model combines prior knowledge of the construction site scene and image-text alignment features to complete refined semantic reasoning. Unlike the local visual judgment of smaller models, it can identify boundary cases in occluded, distant, and blurred scenes. Finally, it outputs the re-judgment confidence and no-violation confidence for each violation category, calculated using the following formula: in, This represents the prediction confidence level for the corresponding violation category c; For the first The original inference score for each violation category, Index for category traversal; molecule This is the index mapping value of the original score for the current target violation category, amplifying the weight of high-scoring categories; denominator The sum of all n category index scores is used as a global normalization term to ensure that the sum of the confidence scores of all categories equals 1, thus forming a valid probability distribution. For the first The confidence level of the large-scale model for re-judgment of traffic violations, with values ​​ranging from... All categories The sum equals 1, and the larger the value, the higher the probability that the violation has actually occurred according to the model.

[0033] The large model outputs four types of structured data: violation existence labels, precise violation categories, confidence scores for each category, and semantic explanation text for the scene. It also adds special markers for abnormal scenes such as blurred images, severe occlusion, and viewing angle deviations. After inference is completed, the results are sent back to the edge consistency verification module via HTTP, synchronously caching data such as inference time, call status, and confidence scores for computational power statistics and anomaly monitoring.

[0034] S104, perform consistency verification between the re-judgment result and the preliminary detection result, confirm the violation based on the consistency verification result and send an alarm.

[0035] In this embodiment, the re-judged violation existence label, re-judged violation category, re-judgment confidence level, and semantic explanation text are extracted from the re-judgment result and compared one by one with the violation target category and prediction confidence level in the preliminary detection result. If all dimensions are consistent, the consistency check is deemed successful, the current alarm is confirmed as a real violation, and an alarm is automatically pushed. Consistency comprehensive judgment formula: Among them, D1 represents consistency in the existence of violations, D2 represents consistency in the type of violations, D3 represents meeting the confidence level, and D4 represents semantic interpretation matching. For consistent results, The results are inconsistent.

[0036] Based on the consistency verification results, precise traffic allocation is achieved, forming an efficient handling mode where machines process routine cases and manual intervention is used to handle exceptions: when all four-dimensional verifications are consistent, and the confidence level of the small model is not lower than the automatic confirmation threshold. When a violation is detected, it is automatically identified as a genuine violation, and an alarm is directly pushed to the management platform and the safety officer's terminal without manual intervention. The total number of alarms is automatically confirmed and recorded. .

[0037] If any verification dimension is inconsistent, it is judged as a boundary difficulty case and automatically transferred to the manual review queue. The total number of alarms awaiting review is recorded as follows. The user interface centrally displays the original alarm image, detection data from different models, and comparison results of dimensional differences, assisting security personnel in making quick decisions.

[0038] In abnormal scenarios such as large model timeouts and crashes, semantic dimension verification is skipped, and simplified traffic distribution is completed only based on the confidence of small models to ensure uninterrupted system operation.

[0039] When quantifying and statistically analyzing the collaborative effect, the collaborative effect of the large and small models is quantified using the automatic confirmation rate. The calculation formula is as follows: The higher the value, the stronger the system's automated processing capability and the lower the cost of manual intervention.

[0040] Human final review serves as the last resort in the four-level judgment system. It introduces human experience to make the final decision on difficult alarms that machines cannot accurately judge. At the same time, the results of human judgment are transformed into the core data source for the model's self-evolution, thus connecting the decision-making process with optimization.

[0041] The consistency check determines that it is inconsistent. All alarms are collected and processed through a unified manual review interface, providing a complete view of the entire data chain: original alarm images, small model detection boxes and confidence levels, large model review conclusions, and detailed four-dimensional verification differences. This helps safety officers quickly locate and determine points of disagreement, improving review efficiency.

[0042] Safety officers, drawing on their experience in various scenarios, make the final judgment, categorizing results into three types: genuine violations, false alarms, and uncertainties. Alarm data manually confirmed as false alarms is automatically linked to the entire decision-making process, including initial screening results, deduplication parameters, re-judgment data, and verification discrepancies, and simultaneously stored in a structured difficult example library to provide accurate sample support for model iteration.

[0043] Unlike traditional methods that only store alarm images, this embodiment retains complete closed-loop data in its difficult case library, including the original alarm image, real human labels, output results of each level of the model, deduplication similarity parameters, and verification difference vectors. This allows for precise location of model misjudgment links and core defects, providing a basis for targeted optimization.

[0044] This step is the core of the system's closed-loop self-optimization. Relying on the difficult case data accumulated through manual final review, it achieves continuous iterative improvement in model performance through the entire process of automated batch annotation, sample optimization, incremental training, gray-scale distribution, and edge hot loading. It gradually reduces the false alarm rate of front-end detection and the amount of manual review, forming a positive optimization flywheel.

[0045] First, a dual-condition complementary triggering mode of "quantity + time" is adopted to ensure the timeliness and stability of iteration. One is quantity triggering, which automatically starts training when 200 new unlabeled samples are added to the difficult example library. The other is time triggering, which iterates and optimizes regularly after 7 days since the last training to avoid the backlog of difficult examples or the long-term solidification of the model.

[0046] Subsequently, batch labeling and construction of obfuscated negative samples were performed, and an automated cloud-based labeling pipeline was launched. Difficult samples were pre-labeled using the Qwen3-VL large model. Results with a confidence level greater than or equal to 0.9 were directly adopted, while samples below the threshold were transferred to manual sampling for verification, significantly reducing labeling costs. At the same time, based on historical misjudgment patterns, targeted visual obfuscation negative samples were constructed, such as scenarios of holding a safety helmet or holding a thin object to simulate smoking, to address the model's shortcomings in scene discrimination.

[0047] When performing incremental LoRA training with dual-objective constraints, the training set is constructed by integrating newly added false positive samples and confused negative samples. LoRA low-rank adaptation fine-tuning technology is employed, optimizing only a small number of model parameters to achieve scenario adaptation, resulting in higher training efficiency. Strict dual-objective constraints are enforced: the false positive rate of the new model is no higher than that of the old model, and the increase in the false negative rate does not exceed 0.5%, ensuring that false positives are reduced without sacrificing detection recall.

[0048] During the gray-scale verification and full rollout, the optimized new model adopts a gray-scale iteration strategy. It is first rolled out to 10% of edge devices for a 24-hour trial run. The false alarm rate and automatic confirmation rate of the gray-scale group and the control group are compared. After confirming that there is no performance rebound and the indicators are optimized, the full rollout to all devices is completed.

[0049] After receiving the new model file, the edge device verifies the integrity of the SHA256 and atomically completes the model replacement without interrupting the video stream or restarting the inference process, ensuring the system operates stably around the clock.

[0050] Quantify the iteration effect by optimizing the model iteration rate: in, This represents the number of newly added difficult cases in this round. The total number of samples in the training set. The false alarm rate improvement coefficient directly reflects the optimization gains of each iteration.

[0051] Model optimization exhibits cascading effects. A slight reduction in the false alarm rate of the front-end small model can progressively reduce redundant alarms, improve automatic confirmation rates, and reduce manual review workload. The core propagation formula is: After multiple iterations, the false alarm rate and manual review rate of the system have steadily decreased, eventually reaching steady-state optimal performance and achieving a self-evolving closed loop throughout the entire process.

[0052] Corresponding to the smart construction site violation identification and alarm method based on dual deduplication and consistency verification provided in the above embodiments, this application also provides an embodiment of a smart construction site violation identification and alarm system based on dual deduplication and consistency verification.

[0053] See Figure 2 This application provides a smart construction site violation identification and alarm system 20 based on dual deduplication and consistency verification, comprising: The acquisition module 201 is used to acquire real-time video streams and perform preprocessing. The preprocessed image frames are input into a pre-built lightweight target detection model for real-time inference, output preliminary detection results, and generate original structured alarm data for violations when triggering conditions are met.

[0054] Dual deduplication determination module 202, configured to perform dual deduplication determination on the original structured violation alarm data, and count the valid alarm data remaining after the dual deduplication.

[0055] Re-judgment module 203, configured to perform secondary re-judgment on the valid alarm data through the cloud-side multimodal large model, and output a structured re-judgment result.

[0056] Violation confirmation and alarm module 204, configured to perform consistency check between the re-judgment result and the preliminary detection result, confirm the violation and push an alarm according to the consistency check result.

[0057] To verify the method of the embodiments of the present application, a foundation pit area of a large construction site is selected, with an area of about 2000 square meters. It includes six core monitoring areas: foundation pit operation area, main structure construction area, scaffold operation area, material stacking area, construction passage, and temporary office area. A total of 32 high-definition network cameras are deployed, all of which support RTSP real-time video stream output, a constant frame rate of 25fps, and 1080P resolution, respectively covering the six operation areas of the construction site, with no monitoring blind spots and no uncovered areas. Each camera is connected to an AI industrial control terminal. The industrial control device is equipped with an Ubuntu embedded system, 8G of running memory, and a built-in AI inference acceleration chip, pre-installed with the edge-side inference, alarm generation, and model hot loading programs supporting this embodiment, stably deployed with a lightweight YOLOv11 detection model, which is accurately adapted to six types of target detection tasks on the construction site: construction site personnel, compliant safety helmets, non-compliant no safety helmets worn, compliant reflective vests, non-compliant no reflective vests worn, and smoking at the construction site. The system adopts an end-edge-cloud three-level collaborative architecture throughout the whole process. All front-end devices collect video data and perform local inference uninterruptedly 7×24 hours, the edge platform performs centralized aggregation processing, and the cloud is responsible for high-precision re-judgment and model iteration, which is fully adapted to the all-weather uninterrupted construction monitoring requirements of the construction site.

[0058] This embodiment features a backend management system developed based on a Spring Boot microservice architecture, equipped with a Redis high-speed caching database, a MySQL business database, and a MinIO object storage service. It can support high-concurrency synchronous processing of over 30 video streams. Asynchronous distribution of alarm data is achieved through a RabbitMQ message queue, avoiding data congestion and alarm loss issues caused by high frame rate video inference. The system's visual management interface displays real-time video feeds from each camera position, real-time alarm information, device online status, model version information, deduplication statistics, manual review records, and model iteration logs. Safety officers can retrieve and trace the source of violations by region, time, and violation type. The system also supports full-parameter visual configuration, including detection confidence thresholds, physical deduplication time windows, semantic similarity thresholds, automatic confirmation thresholds, and training trigger parameters. Parameters can be flexibly adjusted according to the day / night working conditions and operational complexity of the construction site. Furthermore, it supports model gray-scale deployment, version rollback, and lossless hot loading, allowing for model iteration and upgrades without device downtime or interruption of monitoring tasks, ensuring the continuity of construction site monitoring services.

[0059] Specific core configuration parameters: End-side detection parameters: Confidence threshold for detecting violations related to safety helmets and reflective vests. For scenarios with backlighting in the early morning and late evening, or shadows, the recall rate can be lowered to 0.3 to ensure a higher confidence threshold for detecting subtle smoking violations. The NMS-IoU deduplication threshold is 0.45, used to filter overlapping redundant detection boxes within the image; the model input image size is fixed at 640×640 to adapt to the inference computing power of edge devices.

[0060] Dual deduplication parameters: physical-temporal deduplication time window The threshold for deduplication based on semantic similarity of images can be adjusted as needed, ranging from 10s to 120s. Historical alarm feature vectors are cached for 5 minutes. Invalid caches are automatically cleared after the timeout to free up device storage and computing resources.

[0061] Consistency verification parameters: Automatic confirmation threshold for large and small model collaboration It adopts a four-dimensional integrated consistency verification rule that considers the existence of violations, the type of violations, the confidence level, and the semantic interpretation. None of these elements can be omitted to ensure the accuracy of automated judgment.

[0062] Self-evolutionary iteration parameter: Hard example training threshold Timed iteration cycle Batch annotation automatic adoption threshold The LoRA rank of the model was fine-tuned to 8; the gray-scale trial run of the model lasted for 24 hours, and the full rollout was carried out after the verification indicators were found to be correct.

[0063] Execution Process: 32 cameras collect video footage from various construction areas around the clock. Edge AI devices perform frame-by-frame image preprocessing, optimizing image quality through pixel normalization, color gamut correction, and detail enhancement algorithms to address common interference scenarios on construction sites. The preprocessed images are then input into a lightweight YOLOv11 model for frame-by-frame inference. When workers are found to be not wearing safety helmets, reflective clothing, or smoking in key areas such as the edge of the foundation pit, scaffolding working level, or main construction road, and the detection confidence level is met, the system immediately captures a clear alarm frame image and simultaneously generates structured alarm data containing camera device number, precise monitoring area, time of violation, violation type, detection confidence level, and target coordinates. This data is uploaded to the edge-side collaborative platform within seconds via the MQTT protocol, accumulating to generate an original alarm dataset N1.

[0064] The edge platform employs a dual deduplication mechanism coupled with AND. A unique MD5 cache key is generated based on device number, monitoring area, and violation type. This key retrieves historical alarm data from the Redis cache within a 30-second time window, performing spatiotemporal matching to quickly filter out suspected duplicate alarm data. For alarms matching the spatiotemporal data, the CLIP visual encoder is further invoked to extract a 512-dimensional normalized feature vector from the alarm image, calculating its cosine similarity to historical alarm images. Only when the spatiotemporal location completely overlaps and the image semantics are highly similar, simultaneously meeting both conditions, is the system classifying it as a redundant duplicate alarm and automatically filtering it. For violations by different individuals in the same area, different types of violations from the same camera position, and independent violations across different time periods, the system retains valid alarms, ultimately resulting in a valid alarm dataset N2.

[0065] Valid alarms, after double deduplication screening, trigger on-demand review using the cloud-based Qwen3-VL multimodal large model. For high-frequency and complex scenarios on construction sites, such as partial target loss due to crowd occlusion, scaffolding member outline obstruction, and workers misjudging smoking while holding tools, the system outputs accurate structured review results. Subsequently, the system initiates a four-dimensional consistency verification mechanism, comparing the existence of violations, violation category, confidence level, and scene semantic interpretation between the small-scale model on the endpoint and the large-scale model on the cloud. If all four dimensions are completely consistent, and the small-scale model's detection confidence level is greater than or equal to 0.6, the system automatically determines it as a genuine and valid violation, directly pushing alarms to the safety officer's terminal and the backend monitoring platform for automated handling and counting the number of automatically confirmed alarms. If any dimension deviates, it is judged as a boundary-difficult case and automatically transferred to the manual review queue, forming a queue of alarms awaiting manual handling.

[0066] Site safety officers centrally process complex alarm data distributed by the system through a web-based visual backend. The backend interface fully displays the original alarm image, small model detection boxes and confidence levels, large model re-judgment conclusions, four-dimensional verification difference details, and the location and time of the violation, assisting safety officers in locating points of disagreement. Combined with the actual construction site conditions, the safety officer makes the final ruling, categorizing results into three types: genuine violations, model false alarms, and indeterminate. For frequently occurring false alarm scenarios, the system automatically collects end-to-end data, including original video frames, inference results from each level of the model, deduplicated similarity parameters, and verification difference data, storing them uniformly in a structured difficult example sample library. This accumulates real and effective site scenario data for targeted iterative optimization of the model.

[0067] During operation, the difficult example library continuously accumulates misclassified samples from various complex scenarios. When 200 new valid difficult example samples are added, or 7 days have passed since the last model iteration, the system automatically starts the cloud training pipeline. First, the difficult example samples are pre-labeled in batches using a large model. High-confidence labels are directly adopted, while low-confidence samples are sampled and verified by technical personnel for correction. At the same time, targeted negative samples for visual confusion at construction sites are constructed to address the model's shortcomings in recognizing scenarios such as safety helmets being carried on the shoulder, tools simulating smoking, and stains obscuring protective clothing. Subsequently, LoRA efficient fine-tuning technology is used to complete incremental model training, strictly controlling the increase in the false negative rate to ensure that the optimized model does not reduce the recall rate. The newly trained model is first distributed to 10% of edge devices for a 24-hour real-world trial run. The false positive rate and automatic confirmation rate are compared to confirm that the performance optimization has no rebound. Then, the model is fully distributed to all monitoring devices, and the model replacement is completed through lossless hot loading without stopping work or restarting the equipment, continuously improving the accuracy of violation recognition across the entire construction site.

[0068] In this embodiment, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0069] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A smart construction site violation identification and alarm method based on dual deduplication and consistency verification, characterized in that, include: The system acquires and preprocesses real-time video streams, inputs the preprocessed image frames into a pre-built lightweight target detection model for real-time inference, outputs preliminary detection results, and generates original structured alarm data for violations when triggering conditions are met. Perform double deduplication on the original structured violation alarm data and count the remaining valid alarm data after double deduplication. The valid alarm data is re-evaluated using a cloud-based multimodal large model, and a structured re-evaluation result is output. The re-judgment result is checked for consistency with the preliminary detection result. Based on the consistency check result, the violation is confirmed and an alarm is triggered.

2. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 1, characterized in that, The process involves acquiring and preprocessing real-time video streams, inputting the preprocessed image frames into a pre-built lightweight target detection model for real-time inference, outputting preliminary detection results, and generating original structured violation alarm data when trigger conditions are met, including: The system continuously pulls real-time video streams from construction site monitoring cameras using the RTSP / ONVIF protocol, and decodes them at a preset frame rate to obtain continuous image frames. After scaling the image frames to the input standard size, the image pixel values ​​are normalized, and color gamut conversion, random cropping, and flip noise reduction preprocessing are performed simultaneously. The calculation formula is as follows: in, These are the original image pixel values. These are the normalized pixel values; The preprocessed image frame is input into the lightweight object detection model, and the output includes the object detection box, the category of the illegal object, and the prediction confidence. The preliminary detection results are used to determine violations based on preset category confidence thresholds, generating original structured violation alarm data.

3. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 2, characterized in that, The preprocessed image frame is input into the lightweight object detection model, and the output is a preliminary detection result containing the object detection box, the category of the illegal object, and the prediction confidence, including: The preprocessed image frame is input into the lightweight target detection model, and multi-scale feature extraction is completed through the model backbone network and feature pyramid network. The detection box coordinates, category prediction scores and target categories of each target in the image are output. Non-maximum suppression is performed on all detection boxes output by the model. A cross-union ratio (CURRR) threshold is set, and the CURRR of any two detection boxes is calculated. Redundant detection boxes with an overlap exceeding the CURRR threshold are removed, and the optimal detection box is retained. The calculation formula is as follows: ； in, There are two detection frames to be compared. The area of ​​the intersection of the detection boxes. To detect the area of ​​the union of the bounding boxes, Two detection boxes The intersection and union ratio; For each detection box retained after nonmaximum suppression, the Sigmoid activation function is used to normalize the category prediction score output by the model to obtain the prediction confidence of each violation category. The output includes preliminary detection results such as the coordinates of the target detection box, the category of the illegal target, and the corresponding prediction confidence.

4. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 2, characterized in that, The step of determining violations based on the preliminary detection results according to a preset category confidence threshold and generating original structured violation alarm data includes: Set different category confidence thresholds, and compare the predicted confidence in the preliminary detection results with the preset category confidence thresholds; When the predicted confidence level is greater than or equal to the preset category confidence level threshold, it is determined to be a suspected violation; When the predicted confidence level is less than the preset category confidence level threshold, it is determined that there is no violation. The calculation formula is as follows: in, Predict scores for the output categories. It is the Sigmoid activation function. For the prediction confidence level corresponding to violation category c, For category c, a pre-set confidence threshold is set. This indicates that the violation was suspected. This indicates that there was no violation. After identifying a suspected violation, the current valid video frame is captured as an alarm image, and original structured alarm data containing device IP, alarm timestamp, violation type, detection confidence, detection box coordinates, and construction site location is generated simultaneously.

5. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 1, characterized in that, Perform double deduplication on the original structured violation alarm data, and count the remaining valid alarm data after double deduplication, including: Obtain the structured fields of each alarm in the original structured violation alarm data, and construct a cache key based on the device identifier, violation type, and location information. The calculation formula is as follows: in, For the camera device's IP address, This is a traffic violation alarm type. This serves as a location identifier for the logical area of ​​the construction site. It is a hash digest algorithm. Use a unique key for caching; Within a preset time window, historical alarm records with the same cache key are retrieved. If a historical alarm record with the same cache key exists within the preset time window, it is determined as a physical space duplicate hit; otherwise, it is determined as a physical space miss. The calculation formula is as follows: in, This is the current alarm timestamp. The most recent alert timestamp in the history of the same key. For physical deduplication time window, For repeated hits in physical space, Physical space not hit; Extract the visual feature vector of the current alarm image only if the physical space is repeatedly hit; The cosine similarity is calculated based on the visual feature vector and the feature vector of historical alarm images within a preset time window; When the maximum similarity exceeds the preset similarity threshold, it is determined to be a semantic duplicate hit; The current alarm is judged as a duplicate alarm and filtered out only when both physical space duplicate hits and semantic duplicate hits are true; otherwise, the current alarm is judged as valid alarm data and retained.

6. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 5, characterized in that, The current alarm is classified as a duplicate alarm and filtered out only when both physical space duplicate hits and semantic duplicate hits are true; otherwise, the current alarm is classified as valid alarm data and retained. The calculation formula is as follows: in, For physical space repeated hit markers, For image semantic repetition hit tagging, To mark repeated alarms as a total flag, These are the normalized feature vectors of the two alarm images. Normalize the eigenvectors using the L2 norm. This represents the maximum cosine similarity between the current alarm image and all historical alarm images within the time window. A threshold is preset for semantic similarity.

7. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 1, characterized in that, The process of performing a secondary review of the valid alarm data using a cloud-based multimodal large model and outputting structured review results includes: The alarm images and preset dynamic scene prompts corresponding to the violation type in the valid alarm data are encapsulated into a review request and sent to the cloud multimodal large model service interface through the side-side large model call proxy. After receiving the review request, the cloud-based multimodal large model extracts high-dimensional semantic features from the alarm image using a visual encoder, performs semantic encoding on the prompt words using a text encoder, and completes cross-modal alignment of the image and text. The calculation formula is as follows: ； in, For image semantic feature vectors, For the semantic feature vector of the prompt word, The function calculates cosine similarity. Score for image and text alignment; The cloud-based multimodal large model combines prior knowledge of the construction site scenario to perform multi-dimensional semantic reasoning of violations, outputting the re-judgment confidence and no-violation confidence for each violation category. The calculation formula is as follows: in, This represents the prediction confidence level for the corresponding violation category c; For the first The original inference score for each violation category, For the first Confidence level of large-scale model re-judgment for similar violations; The above output is organized into a structured review result according to a preset format.

8. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 1, characterized in that, The consistency verification between the re-judgment result and the preliminary detection result is performed. Based on the consistency verification result, the violation is confirmed and an alarm is pushed, including: Extract the violation existence label, violation category, confidence level and semantic explanation text from the re-judgment result, and compare them one by one with the violation target category and prediction confidence level in the preliminary detection result in multiple dimensions. If all dimensions are consistent, the consistency check is considered to have passed, the current alarm is confirmed as a real violation, and an alarm is automatically pushed. If any dimension comparison result is inconsistent, it is determined that the consistency check has failed, and the current alarm and end-to-end decision data are transferred to the manual review queue for manual judgment.

9. The smart construction site violation identification and alarm method based on dual deduplication and consistency verification according to claim 8, characterized in that, If any dimension comparison result is inconsistent, the consistency check is deemed to have failed, and the current alarm and end-to-end decision data are transferred to the manual review queue for manual judgment, including: Alarms and end-to-end decision data for failed consistency checks are pushed to the manual review interface for centralized display. The final judgment result is received by the safety officer through the manual review interface; When the final judgment result is a false alarm, the alarm evidence and end-to-end decision data of the corresponding alarm will be associated and stored in the difficult case library; When the accumulated amount of the difficult example library reaches the trigger condition, the training process is automatically started to incrementally update the lightweight target detection model; When the final determination result is a genuine violation, a corresponding alarm will be pushed and confirmed. When the final determination result is uncertain, the corresponding alarm will be marked as pending, and complete decision-making chain data will be retained for subsequent traceability and review.

10. A smart construction site violation identification and alarm system based on dual deduplication and consistency verification, characterized in that, include: The acquisition module is used to acquire real-time video streams and preprocess them. The preprocessed image frames are input into a pre-built lightweight target detection model for real-time inference, output preliminary detection results, and generate original structured alarm data for violations when trigger conditions are met. The dual deduplication determination module is used to perform dual deduplication determination on the original structured violation alarm data and count the remaining valid alarm data after dual deduplication. The re-judgment module is used to perform a second re-judgment on the valid alarm data through a cloud-based multimodal large model and output a structured re-judgment result; The violation confirmation and alarm module is used to verify the consistency between the re-judgment result and the preliminary detection result, and to confirm the violation and push an alarm based on the consistency verification result.

Images