Job site compliance detection method and system based on multi-modal data fusion

Abstract

The application provides a kind of job site compliance detection method and system based on multi-modal data fusion.The method comprises: spatiotemporal correlation is carried out on multi-modal original data, and multi-modal site data is generated;Modal adaptation preprocessing is carried out on multi-modal site data, and compliance detection features are extracted;According to the target job scene, modal data quality and compliance detection task, the fusion weight corresponding to each compliance detection feature is determined, and each compliance detection feature is adaptively fused to generate site state fusion features;Site state fusion features are input into compliance rule base and compliance judgment model respectively, and rule judgment result and model judgment result are obtained, and job site compliance detection result is generated;According to job site compliance detection result, determine the violation location, violation object, violation type and risk level, and generate corresponding early warning information and compliance detection record.The application can improve the utilization rate of multi-source data, improve the accuracy of compliance judgment, and enhance the real-time early warning capability.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring technology for industrial sites, and in particular to a method and system for detecting compliance at work sites based on multimodal data fusion. Background Technology

[0002] With the increasing demand for digital supervision in industrial production, engineering construction, power operation and maintenance, and mining operations, compliance inspection at work sites is gradually shifting from manual inspection to intelligent monitoring. Current technologies typically use fixed cameras to collect on-site images or videos, combined with manual inspections, video playback, or single-vision recognition models to detect behaviors such as the wearing of personal protective equipment, entry into hazardous areas, and violations. Some scenarios also incorporate environmental sensors or equipment status acquisition devices to obtain on-site environmental parameters or equipment operating information.

[0003] However, existing methods for on-site compliance inspection still have significant shortcomings. On the one hand, single-vision inspection is easily affected by changes in lighting, occlusion, dust, personnel posture, and shooting angle, making it difficult to reliably identify complex work conditions and prone to false positives and false negatives. On the other hand, on-site image data, environmental data, equipment condition data, and personnel status data are usually independent of each other, lacking a unified spatiotemporal correlation and fusion analysis mechanism. This makes it difficult for the system to accurately determine complex violations such as personnel being on duty but equipment not being used, or continuing to work despite environmental exceeding standards. Furthermore, existing solutions largely rely on manual review or fixed rule judgments, lacking real-time performance, and compliance standards vary significantly across different work scenarios, making it difficult for a single model or rule to adapt quickly. Therefore, existing technologies still suffer from fragmented detection data, incomplete intelligent judgment, weak real-time warning capabilities, and poor cross-scenario adaptability. Summary of the Invention

[0004] In view of this, the embodiments of this application provide a method and system for on-site compliance detection based on multimodal data fusion, so as to solve the problems of fragmented detection data, one-sided judgment, and insufficient real-time early warning capability in the prior art.

[0005] A first aspect of this application provides a method for on-site compliance detection based on multimodal data fusion, comprising: acquiring multimodal raw data representing the work object, work environment, and work process at the work site; spatiotemporally associating the multimodal raw data based on a unified time reference and work space coordinates to generate multimodal on-site data corresponding to the target detection period and target work area; performing modal adaptation preprocessing on the multimodal on-site data to extract compliance detection features representing the on-site visual state, operational state, environmental state, and behavioral state; determining the fusion weights corresponding to each compliance detection feature according to the target work scenario, modal data quality, and compliance detection task, and adaptively fusing each compliance detection feature to generate on-site state fusion features; inputting the on-site state fusion features into a compliance rule base and a compliance judgment model respectively to obtain rule judgment results and model judgment results, and performing hierarchical compliance judgment based on the rule judgment results and model judgment results to generate on-site compliance detection results; determining the violation location, violation object, violation type, and risk level according to the on-site compliance detection results, and generating corresponding early warning information and compliance detection records.

[0006] A second aspect of this application provides a work site compliance detection system based on multimodal data fusion, comprising: an acquisition module for acquiring multimodal raw data representing the work object, work environment, and work process at the work site, and spatiotemporally correlating the multimodal raw data based on a unified time reference and work space coordinates to generate multimodal site data corresponding to the target detection period and target work area; an extraction module for performing modal adaptation preprocessing on the multimodal site data to extract compliance detection features representing the site's visual state, operational state, environmental state, and behavioral state; a fusion module for determining the fusion weights corresponding to each compliance detection feature based on the target work scenario, modal data quality, and compliance detection task, and adaptively fusing each compliance detection feature to generate site state fusion features; a judgment module for inputting the site state fusion features into a compliance rule base and a compliance judgment model respectively to obtain rule judgment results and model judgment results, and performing hierarchical compliance judgment based on the rule judgment results and model judgment results to generate work site compliance detection results; and a generation module for determining the violation location, violation object, violation type, and risk level based on the work site compliance detection results, and generating corresponding early warning information and compliance detection records.

[0007] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By acquiring multimodal raw data representing the work object, work environment, and work process at the work site, and based on a unified time base and work space coordinates, spatiotemporally correlated the multimodal raw data to generate multimodal field data corresponding to the target detection period and target work area; modal adaptation preprocessing is performed on the multimodal field data to extract compliance detection features representing the visible state, operational state, environmental state, and behavioral state of the site; based on the target work scenario, modal data quality, and compliance detection task, the fusion weights corresponding to each compliance detection feature are determined, and each compliance detection feature is adaptively fused to generate field state fusion features; the field state fusion features are input into the compliance rule base and compliance judgment model respectively to obtain rule judgment results and model judgment results, and hierarchical compliance judgment is performed based on the rule judgment results and model judgment results to generate work site compliance detection results; based on the work site compliance detection results, the violation location, violation object, violation type, and risk level are determined, and corresponding early warning information and compliance detection records are generated. This application can improve the utilization rate of multi-source data, enhance the accuracy of compliance judgment, and strengthen real-time early warning capabilities. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the on-site compliance detection method based on multimodal data fusion provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the on-site compliance detection system based on multimodal data fusion provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0010] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0011] In existing technologies, compliance inspections at work sites mainly rely on manual inspections, fixed camera monitoring, and post-event video review. Some systems use single visual recognition models to detect visual behaviors such as personnel attire, entry into hazardous areas, and violations. Other sites are equipped with separate environmental sensors, equipment status acquisition devices, or personnel positioning devices to collect localized work status information. However, this data is usually scattered across different systems, lacking a unified spatiotemporal correlation and fusion analysis mechanism, making it difficult to form a comprehensive judgment on the overall compliance status of the work site.

[0012] The main problems with existing technologies are that single-vision detection is easily affected by changes in lighting, occlusion, dust, shooting angle, and personnel posture, leading to a high number of false positives and false negatives; environmental data, equipment operating data, and personnel status data are isolated, making it impossible to accurately identify complex violation scenarios; manual inspection and post-event review methods lack real-time capability, making it difficult to promptly detect safety hazards during operations; at the same time, compliance standards vary significantly across different industries and operating scenarios, limiting the adaptability of fixed rules or single models. Therefore, existing technologies suffer from fragmented detection data, biased judgments, and insufficient real-time early warning capabilities.

[0013] To address the aforementioned technical challenges, this application provides an intelligent on-site compliance detection method based on multimodal data fusion. This method acquires multimodal raw data representing the work object, work environment, and work process. Based on a unified time reference and work space coordinates, it performs spatiotemporal correlation on the multimodal raw data to generate multimodal on-site data corresponding to the target detection period and target work area. The multimodal on-site data then undergoes modal adaptation preprocessing to extract compliance detection features representing the visible state, operational state, environmental state, and behavioral state of the site. Furthermore, based on the target work scenario, modal data quality, and compliance detection task, fusion weights are determined, and each compliance detection feature is adaptively fused to generate on-site state fusion features.

[0014] During the compliance assessment phase, this application inputs the fused features of the on-site status into the compliance rule base and the compliance assessment model, respectively. The compliance rule base performs rule matching for compliance constraints under explicit standards, while the compliance assessment model identifies and classifies risks in complex operational states that are difficult to directly cover with rules. A tiered compliance assessment is then performed by combining the rule assessment results and the model assessment results to generate on-site compliance detection results. Subsequently, based on the on-site compliance detection results, the location, object, type, and risk level of the violation are determined, generating corresponding early warning information and compliance detection records, thus achieving a closed-loop process of detection, assessment, early warning, and archiving.

[0015] Through the above technical solutions, this application can unify and integrate previously scattered on-site perception data, equipment operation data, and personnel operation data, thereby improving the utilization rate of multi-source data; reduce the risk of misjudgment caused by environmental interference in a single modality through adaptive weight fusion, thereby improving the accuracy of compliance judgment; improve the ability to identify complex and hidden violation scenarios through hierarchical judgment of rule base and compliance judgment model; and enhance the real-time early warning capability and subsequent traceability capability of the work site by outputting early warning information and compliance detection records according to risk level.

[0016] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] Figure 1 This is a flowchart illustrating the on-site compliance detection method based on multimodal data fusion provided in this application. Figure 1 As shown, the method may specifically include: S101: Acquire multimodal raw data representing the work object, work environment and work process in the work site, and perform spatiotemporal correlation on the multimodal raw data based on a unified time reference and work space coordinates to generate multimodal field data corresponding to the target detection period and target work area; S102, Modal adaptation preprocessing is performed on multimodal field data to extract compliance detection features that characterize the field's visual state, operational state, environmental state, and behavioral state; S103. Based on the target operation scenario, modal data quality, and compliance inspection task, determine the fusion weights corresponding to each compliance inspection feature, and perform adaptive fusion of each compliance inspection feature to generate on-site status fusion features. S104: Input the fusion features of the on-site status into the compliance rule base and the compliance judgment model respectively to obtain the rule judgment results and the model judgment results. Then, perform hierarchical compliance judgment based on the rule judgment results and the model judgment results to generate the on-site compliance detection results. S105 determines the location, object, type, and risk level of violations based on the compliance inspection results at the work site, and generates corresponding early warning information and compliance inspection records.

[0018] In some embodiments, acquiring multimodal raw data characterizing the work object, work environment, and work process at the work site includes: Configure multi-source acquisition nodes according to the area division, equipment layout and operation process of the work site, and assign acquisition object identifiers and spatial location identifiers to the multi-source acquisition nodes; Based on a unified sampling strategy, multi-source acquisition nodes collect on-site perception data, equipment operation data, and personnel operation data corresponding to the target detection period; The collected data is appended with a collection timestamp, collection object identifier, and spatial location identifier, and then encapsulated according to the data source and job association to generate multimodal raw data.

[0019] Specifically, in high-altitude operations, hot work operations, and material hoisting operations at construction sites, the site can be divided into work areas such as floor work areas, edge protection areas, hot work areas, material storage areas, equipment operation areas, and personnel passage areas based on the construction layout plan, work permits, and equipment ledgers. Multi-source acquisition nodes matching the detection tasks are then configured within each work area. These multi-source acquisition nodes can include fixed visual acquisition nodes, mobile inspection acquisition nodes, environmental perception acquisition nodes, equipment status acquisition nodes, and personnel status acquisition nodes. During the deployment phase, the system establishes a node file for each acquisition node. The node file records the node type, installation location, monitoring orientation, coverage area, corresponding work area, associated equipment, and the type of acquisition object. Each acquisition node is assigned an acquisition object identifier and a spatial location identifier. The acquisition object identifier indicates the personnel, equipment, work procedures, or environmental objects that the acquisition node primarily targets, while the spatial location identifier indicates the installation coordinates and coverage area of ​​the acquisition node within the spatial coordinate system of the work site.

[0020] During the data collection process, the edge computing gateway generates a unified sampling strategy based on the target detection period and compliance detection task. This unified sampling strategy can include sampling period, triggering conditions, acquisition window length, cache depth, and data reporting frequency. For example, during the target detection period after a hot work permit takes effect, the system can increase the frequency of keyframe acquisition in the hot work area video, while simultaneously driving environmental perception acquisition nodes to synchronously collect combustible gas concentration, dust concentration, and ambient temperature; driving equipment status acquisition nodes to collect welding machine start / stop status, power consumption, and grounding status; and driving personnel status acquisition nodes to collect personnel location information, on-duty status, and operation trajectory. For routine inspection scenarios, the system can collect on-site perception data at a fixed cycle; for events such as equipment startup, personnel entering restricted areas, or environmental parameters approaching thresholds, the system can switch to event-triggered acquisition mode, enabling multiple acquisition nodes within the relevant area to synchronously report data within the same detection time window.

[0021] To ensure the availability of subsequent multimodal fusion analysis, the system attaches uniformly formatted metadata to the collected data at the acquisition end or edge gateway. For field perception data, image frames, video clips, or initial target detection results can be bound to the acquisition timestamp, object identifier, spatial location identifier, and camera coverage area. For equipment operation data, equipment start / stop status, operating load, fault signals, and operating condition change records can be bound to the equipment identifier, work area identifier, and acquisition timestamp. For personnel operation data, personnel location points, attendance status, authorization information, and trajectory segments can be bound to the personnel identifier, task identifier, and spatial location identifier. The acquisition timestamp can be distributed by a unified clock service or generated by the edge gateway after calibrating the local time of each acquisition node, thereby avoiding data misalignment caused by clock drift between different devices.

[0022] During the data encapsulation phase, the system organizes the collected data according to its source and operational relationships. Specifically, the edge gateway can encapsulate data records related to the same task, equipment, or personnel within the same area into a multimodal data packet, using the target detection period as the time unit and the target work area as the area unit. This multimodal data packet includes data payload, data source type, collection object identifier, spatial location identifier, collection timestamp, collection quality identifier, and operational association identifier. The operational association identifier can be generated based on work permits, equipment ledgers, and personnel scheduling information, indicating whether a person, equipment, and area belong to the same operational process within the current detection period. For example, in hot work detection on a three-layer platform, the system can encapsulate camera images of the hot work area, welding machine operating status, operator positioning information, and environmental sensor readings into the same multimodal data packet, enabling subsequent steps to determine whether personnel are working within the authorized area, whether equipment is being used according to specifications, and whether environmental conditions allow for continued operation.

[0023] By employing the above methods, the correlation between multi-source data and work areas, work objects, and work processes can be established during the data acquisition stage. This ensures that the multimodal raw data not only includes single-point acquisition results but also carries temporal, spatial, and semantic information about the work. This reduces data mismatch and object confusion in subsequent fusion analysis, improves the data integrity and spatiotemporal consistency of compliance detection at the work site, and provides a reliable data foundation for subsequent modality adaptation preprocessing, feature fusion, and hierarchical compliance determination.

[0024] In some embodiments, based on a unified time reference and work space coordinates, spatiotemporal correlation is performed on the multimodal raw data to generate multimodal field data corresponding to the target detection time period and target work area, including: The multimodal raw data is time-calibrated based on the acquisition timestamp to determine the detection time window of each data record relative to a unified time reference. Based on the spatial location identifier, each data record is mapped to the work space coordinates to determine the work area and work object corresponding to each data record; Based on the detection time window, work area, and work object, data records whose time offset meets preset association conditions and whose spatial location meets preset matching conditions are aggregated to generate multimodal field data corresponding to the target detection time period and target work area.

[0025] Specifically, in compliance inspection scenarios for hot work and hoisting operations at construction sites, after data acquisition, the system first uses an edge computing gateway to uniformly calibrate the acquisition timestamps in the multimodal raw data. Since on-site cameras, environmental sensors, equipment controllers, and personnel positioning tags may use different local clocks, the system can distribute a reference time to each acquisition node through a unified clock service, or correct the acquisition timestamps at the edge gateway based on node heartbeat time, network transmission latency, and sampling period. For video frames, sensor readings, equipment status records, and personnel positioning points, the system calculates their time offset relative to the unified time reference and divides the detection time window according to a preset detection granularity.

[0026] For example, in the detection of hot work operations on a three-layer platform, a detection time window can be set every 5 seconds after the work permit takes effect. Video keyframes, welding machine start-stop records, combustible gas concentration readings, and personnel positioning records are all mapped to the corresponding window. For data records whose time offset exceeds the window's allowable range, the system can mark them as late data or invalid associated data to avoid incorrectly combining data from different work times due to time misalignment.

[0027] Furthermore, after completing time calibration, the system maps each data record to work space coordinates based on spatial location identifiers. Work space coordinates can be established based on construction plans, equipment layout diagrams, and on-site calibration points, including spatial information such as floors, areas, coordinate units, and equipment coverage. For visual acquisition nodes, the system back-projects personnel frames, equipment frames, or hazardous area frames from the image to the on-site spatial coordinates based on the camera's installation location, shooting direction, and calibration parameters, thus determining the corresponding work area and work object.

[0028] For environmental sensing nodes, the system determines the environmental impact range corresponding to the sensing data based on the sensor installation coordinates and effective sensing radius. For equipment operation data, the system determines the work area to which the equipment belongs based on the equipment installation location in the equipment ledger or the current positioning coordinates of the mobile device. For personnel work data, the system determines the area where the personnel are located and their associated work tasks during the target detection period based on positioning tags, access control check-in, or trajectory points.

[0029] During the data aggregation phase, the system uses the detection time window, work area, and work object as a joint index to perform correlation matching on the multimodal raw data. Specifically, the system first filters data records that are within the same detection time window, then determines whether the spatial coordinates of each data record fall within the same target work area or adjacent influence range, and further determines whether they belong to the same work object or the same work process based on personnel identification, equipment identification, work task identification, and area authorization information.

[0030] For example, when a personnel location record shows that a person has entered the hot work area on the third-floor platform, and within the same time window, a camera detects welding activity in that area, a welding machine status record shows that the equipment has been started, and an environmental sensor reports that the concentration of combustible gas is close to the warning threshold, the system aggregates these records into a set of multimodal on-site data corresponding to that hot work area. If there is a slight spatial deviation between the same personnel location record and the camera recognition result, but the deviation does not exceed the preset spatial matching range, the system can still complete the aggregation based on the area coverage relationship and the work task association relationship; if the deviation exceeds the matching range, the record is retained as data to be confirmed.

[0031] For hoisting operations, the system can aggregate the hoisting equipment's operating status, video recognition results within the hoisting radius, personnel positioning trajectories, and on-site audible and visual alarm signals according to the same detection time window. This allows the system to determine whether irrelevant personnel are present within the hoisting area, whether the equipment's operating status is consistent with the hoisting process, and whether the warning zone has been triggered. In this way, the system doesn't simply stack data according to its source; instead, it establishes on-site status data units around the target detection time period and the target work area. This ensures that each set of multimodal on-site data points to a specific time segment, spatial range, and work object.

[0032] Through the above spatiotemporal correlation processing, scattered image data, sensor data, equipment data, and personnel data can be unified under the same work site semantics, reducing data mismatch caused by time drift, coordinate deviation, and object inconsistency between different acquisition nodes, improving the correlation accuracy and completeness of multimodal field data, and providing a reliable basis for subsequent compliance detection feature extraction, adaptive fusion, and hierarchical compliance judgment.

[0033] In some embodiments, modal adaptation preprocessing is performed on multimodal field data to extract compliance detection features characterizing the field's visual state, operational state, environmental state, and behavioral state, including: Based on the data type and acquisition quality of the multimodal field data, the preprocessing strategy corresponding to each modality of data is determined. For image data, image quality correction and target area parsing are performed; for numerical data, abnormal data correction and unit unification processing are performed; and for event data, status field parsing and job semantic mapping are performed. The preprocessed modal data are converted into a unified feature representation, and the state features related to compliance judgment are extracted by combining the target detection period, target operation area and operation object to generate compliance detection features.

[0034] Specifically, in the scenario of compliance inspection of hot work operations on a three-story platform at a construction site, after obtaining multimodal on-site data corresponding to the target inspection period and target work area, the system first reads the data type, acquisition quality identifier, and work association identifier of each data record to determine the preprocessing strategies corresponding to image data, numerical data, and event data. For image data, the system can determine the image quality correction method based on the brightness, clarity, occlusion ratio, and dust interference level of the camera image. For example, when the camera in the hot work area is at night or when there is a lot of dust, the system performs brightness compensation, defogging enhancement, noise suppression, and keyframe filtering on the video frames, and combines the camera coverage area and work space coordinates to crop out the welding points, personnel positions, safety warning lines, and the area around the equipment to form an image block of the target area.

[0035] Subsequently, the target detection model identifies personnel, equipment, protective gear, open flame status, and warning area boundaries in the target area image blocks, and extracts on-site visual features such as personnel posture, clothing status, distance between personnel and hazard sources, and whether there is unauthorized approach.

[0036] For numerical data, the system determines the abnormal data correction method based on sensor type, sampling frequency, historical fluctuation range, and acquisition quality identifier. If there are non-operational spikes in combustible gas concentration, dust concentration, temperature, noise, or equipment power between adjacent sampling points, the system can combine sliding window statistics, equipment operation stage, and adjacent sensor readings to remove outliers or perform interpolation correction. If different acquisition nodes use different units of measurement or ranges, the system converts the numerical data to a unified dimension supported by the compliance rule base and performs normalization processing. For example, the system can uniformly convert the concentration readings of multiple gas detection nodes to the same concentration unit and generate operating status and environmental status characteristics such as environmental exceedance level, environmental change trend, equipment load status, and equipment start-up / shutdown consistency based on the safety thresholds configured in the hot work permit.

[0037] For event-based data, the system performs status field parsing and job semantic mapping on event records reported by equipment controllers, access control systems, personnel positioning tags, work ticket systems, and alarm systems. Equipment events can be parsed into status fields such as start-up, shutdown, fault, overload, and protection action; personnel events can be parsed into status fields such as entering area, leaving area, on-duty confirmation, authorization verification, and trajectory change. The system further matches these status fields with the target detection time period, target work area, and work object to determine the job semantics corresponding to the event record. For example, when the personnel positioning record shows that a welder has entered the hot work area of ​​the third-floor platform, the work ticket shows that the person has hot work authorization, and the welding machine status record shows that the equipment has been started, the system can map the relevant events as authorized personnel performing hot work in the target area; if unauthorized personnel enter the warning area within the same time window, it is mapped as an abnormal personnel management situation.

[0038] Furthermore, after completing the modal adaptation preprocessing, the system converts the data from each modality into a unified feature representation. Specifically, image recognition results can be converted into structured visual features consisting of target category, spatial coordinates, confidence level, and status label; sensor readings can be converted into numerical features consisting of current value, rate of change, threshold distance, and confidence level; and equipment and personnel events can be converted into event features consisting of status code, duration, number of triggers, and job semantic label. All of the above features are indexed according to target detection time period, target work area, and work object, enabling the system to establish a unified set of compliance detection features around the same hot work process.

[0039] For example, during the target detection period, the system extracts visual features from image data, such as workers not fully wearing protective masks and workers being less than a safe distance from the welding point; it extracts environmental features from numerical data, such as combustible gas concentrations approaching the threshold and showing an upward trend; it extracts operational features from equipment operation data, such as the welding machine being started but the ventilation equipment not being started simultaneously; and it extracts behavioral features from personnel event data, such as an unauthorized person entering the warning area. The system uses these features collectively as compliance detection features related to the current hot work compliance determination, rather than relying solely on a single video recognition result or a single sensor alarm result.

[0040] Through the above processing, adaptive preprocessing can be performed on the noise characteristics, expression forms and operational semantic differences of different modal data, reducing the interference caused by weak light, dust, sensor fluctuations and inconsistent event fields to subsequent identification, and converting heterogeneous data into unified, fusionable and judgmentable compliance detection features, thereby improving the accuracy and stability of compliance judgment at the work site.

[0041] In some embodiments, based on the target operation scenario, modal data quality, and compliance inspection task, the fusion weights corresponding to each compliance inspection feature are determined, and the compliance inspection features are adaptively fused to generate on-site state fusion features, including: Analyze the scenario constraint parameters corresponding to the target operation scenario, and determine the modal credibility of each compliance detection feature based on the quality of the modal data; Based on the compliance detection task, the task relevance of each compliance detection feature to the compliance matter to be judged is determined, and the fusion weight corresponding to each compliance detection feature is generated by combining scenario constraint parameters, modal credibility and task relevance. The compliant detection features are weighted and fused according to the fusion weight, and the fusion results are verified for cross-modal consistency to generate on-site status fusion features that characterize the comprehensive operation status during the target detection period.

[0042] Specifically, in the compliance detection scenario for hot work operations on a three-story platform at a construction site, after generating compliance detection features, the system first reads the scenario configuration file and work permit information corresponding to the target operation scenario and parses the scenario constraint parameters. Scenario constraint parameters may include operation type, risk level of the operation area, hot work permit time, warning range, ventilation requirements, equipment interlocking requirements, personnel authorization conditions, and environmental safety thresholds. For example, hot work operations on a three-story platform are high-risk open flame operations; the scenario constraint parameters can be configured with the warning radius around the hot work point, the upper limit of combustible gas concentration, ventilation equipment activation requirements, personnel protection requirements, and rules prohibiting unauthorized personnel from entering. Based on these scenario constraint parameters, the system determines the basic importance values ​​of different compliance detection features in the current scenario, ensuring that features related to open flame safety, equipment interlocking, and personnel authorization receive a higher initial weight.

[0043] Subsequently, the system determines the modal credibility of each compliance detection feature based on the quality of the modal data. For image-related features, the system can calculate the visual modal credibility based on image brightness, target sharpness, occlusion ratio, dust interference level, and target recognition confidence. For example, in hot work areas, the camera image is significantly affected by welding fumes, and when the confidence of personnel mask recognition decreases, the system reduces the modal credibility corresponding to protective equipment recognition features and personnel posture features; when the camera's field of view is clear and the target area is unobstructed, the credibility of the corresponding visual features increases.

[0044] For environmental features, the system determines modal reliability based on sensor online status, sampling stability, consistency between adjacent nodes, and historical drift. For equipment features, the system determines modal reliability based on equipment acquisition interface communication status, status field completeness, and operational feedback continuity. For personnel features, the system determines modal reliability based on positioning accuracy, trajectory continuity, and identity verification results.

[0045] After determining the modal reliability, the system further calculates the task relevance of each compliance detection feature to the compliance item to be judged based on the compliance detection task. When taking whether hot work is allowed to continue as the compliance item to be judged, combustible gas concentration, ventilation equipment activation status, welding machine operation status, operator authorization status, and personnel intrusion into the warning area have high task relevance, while ordinary pedestrian trajectories or noise data in remote areas, which are less related to the current task, have low task relevance.

[0046] In some examples, when compliance with personal protective equipment (PPE) wearing is considered as a compliance criterion, personnel image target features, PPE identification features, personnel identity, and job position information have high task relevance, while equipment power changes and remote environmental readings are only used as auxiliary features in the judgment. The system can jointly determine the task relevance using a task rule mapping table, historical detection sample statistics, and attention response results in the compliance judgment model.

[0047] When generating fusion weights, the system normalizes and combines the base values ​​of importance, modal credibility, and task relevance corresponding to the scene constraint parameters to obtain the fusion weight for each compliance detection feature. For example, in nighttime hot work operations with high dust levels, the credibility of the visual modality decreases, so the system reduces the weight of the feature identifying those not wearing masks, while increasing the weights of features such as personnel positioning, work authorization, welding machine start / stop, and combustible gas readings. When environmental sensors experience short-term communication fluctuations, the system reduces the weight of the corresponding environmental features and compensates for the on-site condition judgment by using data from adjacent sensors and equipment operating characteristics. Thus, the fusion weights can be dynamically adjusted according to the work scenario, acquisition quality, and detection task, rather than simply splicing together modal features with fixed weights.

[0048] Furthermore, after completing the weight calculation, the system performs weighted fusion of each compliance detection feature according to the fusion weight, generating an initial fusion result. This initial fusion result can include multiple sub-state representations such as personnel compliance status, equipment linkage status, environmental risk status, and operation process status. The system then performs cross-modal consistency verification to determine whether there are logical conflicts between different modal features. For example, if visual features show that welding is in progress, but equipment features show that the welding machine is not started, the system checks the video time window, equipment acquisition delay, and equipment identification matching relationship; if personnel positioning shows that the operator is not in the hot work area, but the video identifies authorized personnel operating in the area, the system checks positioning drift, camera coverage boundaries, and personnel identification confidence. For explainable minor deviations, the system corrects them according to modal confidence; for unexplainable conflicts, the system retains the conflict identifier and increases the risk focus of subsequent judgments.

[0049] Through the above processing, the system can fuse dispersed visual features, environmental features, equipment features, and personnel behavior features into a field state fusion feature that represents the comprehensive operational status during the target detection period. This field state fusion feature not only reflects the detection results of a single modality but also reflects the mutual support relationships and conflict states between the various modalities, thereby improving the stability of compliance judgments in complex operational scenarios and reducing false detections and missed detections caused by interference from a single modality.

[0050] In some embodiments, the fused features of the on-site state are input into the compliance rule base and the compliance determination model, respectively, to obtain the rule determination result and the model determination result, including: Based on the target operational scenario and compliance testing task, the target judgment rule corresponding to the compliance matter to be judged is called from the compliance rule library; The on-site state fusion features are mapped to rule input items of the target determination rule, and compliance constraint matching is performed on the rule input items based on the target determination rule to generate the rule determination result; By inputting the fusion features of the on-site conditions into the trained compliance judgment model, risk classification and identification are performed on complex operation states that are difficult to be directly covered by rule constraints, and the model judgment results are generated.

[0051] Specifically, in the compliance detection scenario of hot work operations on a three-story platform at a construction site, after obtaining the fusion characteristics of the site status, the system determines the compliance matters that need to be judged based on the target operation scenario and the compliance detection task. The target operation scenario can be jointly determined by the work permit, the area risk level, the type of operation, and the on-site supervision configuration. For example, if the current scenario is hot work operations in a restricted area of ​​a three-story platform, the compliance detection task includes whether the hot work environment meets safety conditions, whether the operators are authorized, whether the hot work equipment and auxiliary safety equipment are linked in accordance with regulations, and whether there are any unauthorized personnel entering the restricted area.

[0052] Based on this scenario information, the system retrieves target determination rules matching the current operation from the compliance rule base. Target determination rules can include hot work environment threshold rules, personnel access rules, equipment start-up and shutdown linkage rules, warning area control rules, and operation time period rules, etc. The rules in the rule base can be pre-configured by industry safety specifications, enterprise operation standards, and on-site management requirements, and rule indexes are established according to operation type, area attributes, and risk level.

[0053] During the rule-based determination process, the system maps the fused features of the on-site conditions into rule input items that can be recognized by the target determination rules. For example, the fused features of the on-site conditions include combustible gas concentration characteristics, dust concentration change trends, welding machine operating status, ventilation equipment activation status, worker authorization status, protective clothing identification status, and the distance between personnel location and the boundary of the warning area. Based on the field definitions corresponding to the target determination rules, the system converts the above fused features into environmental safety input items, equipment linkage input items, personnel access input items, and area control input items.

[0054] In some examples, for hot work environment determination, the system maps combustible gas concentration, dust concentration, and ventilation status to environmental safety input items and matches them with environmental thresholds configured in the rule base; for equipment linkage determination, the system maps welding machine start / stop status and exhaust equipment status to equipment linkage input items and determines whether auxiliary safety equipment is synchronously in an effective state when hot work equipment is started; for personnel access determination, the system maps personnel identity, work authorization, and location to personnel access input items and determines whether personnel have the authority to enter the target work area and perform hot work.

[0055] Compliance constraint matching can be performed using a hierarchical matching approach. The system first determines whether the applicable conditions of the target determination rule are met, such as the work type being hot work, the target detection period being within the valid work permit period, and the target work area being a configured high-risk area. When the applicable conditions are met, the system then matches the rule input items with the threshold conditions, status conditions, and associated conditions in the rule.

[0056] If the concentration of combustible gas exceeds the safety threshold, or if the welding machine is started but the ventilation equipment is not, the system generates a rule judgment result indicating an abnormal rule hit status. If the operator is not on the authorized list or unauthorized personnel enter the warning area, the system generates a corresponding personnel or area violation rule judgment result. The rule judgment result can record the hit rule identifier, violation constraint, rule judgment level, corresponding work object, and judgment basis, so that subsequent warnings and traceability can clearly identify the source of the violation.

[0057] Simultaneously, the system inputs the fusion features of the on-site conditions into a trained compliance assessment model to identify complex operational states that are difficult to directly cover by rule constraints. The compliance assessment model can be trained based on historical on-site data, manual review results, and accident hazard samples. The model input is comprehensive operational state features, and the model output is a composite risk category and risk confidence level. For example, within a certain detection time window, the environmental reading may not have exceeded the threshold, but the concentration of combustible gas continues to rise, the ventilation equipment is operating intermittently, personnel are still performing welding actions, and there are materials piled up nearby. A single rule may not have triggered a serious violation, but the compliance assessment model can identify it as a risk of deterioration of the hot work environment based on the combination of multi-dimensional states. As another example, the protective mask may not be consistently identified in the video, but the personnel's location, work position, posture changes, and equipment operating status all indicate that the personnel are operating the welding machine at close range. The model can combine multi-modal features to output a protection risk or operational risk category.

[0058] In another example, in a hoisting operation scenario, the rule base can directly determine whether personnel entering the hoisting warning area is a violation, while the compliance judgment model can further identify potential risks of personnel entering the hoisting path based on changes in the load of the hoisting equipment, the direction of boom rotation, personnel movement trends, and on-site alarm status. Thus, the rule-based judgment results provide deterministic judgments under clear standards, while the model-based judgment results provide risk identification results under complex conditions. Together, they form the data foundation for subsequent hierarchical compliance judgments.

[0059] Through the above methods, the system can transform the fusion features of the on-site status into executable constraint matching results from the rule base and risk classification results identifiable by the compliance judgment model. This ensures that fixed compliance standards can be accurately executed and improves the ability to identify hidden violations and complex risk scenarios, thereby enhancing the accuracy, completeness, and interpretability of on-site compliance detection.

[0060] In some embodiments, the on-site state fusion features are mapped to rule inputs of the target determination rule, and compliance constraint matching is performed on the rule inputs based on the target determination rule to generate a rule determination result, including: Analyze the constraint fields, applicable conditions and judgment thresholds corresponding to the target determination rules, and determine the feature mapping relationship that matches the fusion features of the on-site state. Based on the feature mapping relationship, the status parameters corresponding to the compliance matters to be judged are extracted from the fusion features of the on-site status, and the status parameters are converted into rule input items; When the applicable conditions are met, the rule input items are matched with the judgment threshold for compliance constraints, and a rule judgment result containing the rule hit status, violation constraints, and rule judgment level is generated.

[0061] Specifically, in the compliance detection scenario of hot work operations on a three-story platform at a construction site, after obtaining the fusion features of the site status, the system first determines the compliance items to be judged based on the current target operation scenario and the compliance detection task. For example, if the current detection task is to determine whether the hot work meets the conditions for continuing operations, the compliance items to be judged may include environmental safety conditions, equipment linkage conditions, personnel access conditions, and area control conditions. After retrieving the corresponding target judgment rule from the compliance rule library, the system performs rule structure parsing on the target judgment rule to obtain the constraint fields, applicable conditions, and judgment thresholds.

[0062] The constraint field is used to specify the types of status parameters that the rule needs to read, such as environmental risk parameters, equipment operating parameters, personnel authorization parameters, and area occupancy parameters; the applicable conditions are used to limit the type of operation, operation area, operation time period, and risk level for which the rule takes effect; the judgment threshold is used to represent the compliance boundary, such as the upper limit of combustible gas concentration, the upper limit of dust concentration, the lower limit of warning distance, equipment linkage status requirements, and personnel authorization status requirements.

[0063] During rule parsing, the system establishes a feature mapping relationship between the constraint fields of the target determination rule and the feature fields in the field status fusion features. The field status fusion features already include the comprehensive operational status formed through adaptive fusion, such as the environmental safety characterization of the hot work point, welding machine operation status, ventilation equipment status, worker identity status, protective gear status, personnel location status, and warning area occupancy status. Based on field semantics, work object identification, and the target work area, the system maps the combustible gas concentration characterization to the environmental safety constraint field, the welding machine operation status and ventilation equipment status to the equipment linkage constraint field, the personnel identity authorization and job status to the personnel access constraint field, and the spatial distance between personnel and the warning boundary to the area control constraint field. For cases where the same constraint field corresponds to multiple modal sources, the system prioritizes reading the comprehensive status parameters that have undergone weighted fusion and consistency verification, avoiding misjudgments caused by directly using single sensor readings or single image recognition results.

[0064] Based on feature mapping relationships, the system extracts state parameters corresponding to the compliance matters to be judged from the fusion features of on-site conditions, and converts these state parameters into rule input items that the target judgment rule can execute and match. For example, for environmental safety conditions, the system extracts the fusion values ​​of combustible gas concentration, dust concentration, environmental change trends, and ventilation status, and converts them into environmental safety input items; for equipment linkage conditions, the system extracts the start / stop status of welding machines, the start / stop status of exhaust equipment, the duration of equipment operation, and fault status, and converts them into equipment linkage input items; for personnel access conditions, the system extracts personnel identity, work permit matching status, on-duty status, and authorized area matching status, and converts them into personnel access input items; for area control conditions, the system extracts personnel spatial coordinates, warning range boundaries, stay duration, and personnel type, and converts them into area control input items. Rule input items can be stored in the form of structured key-value records, with accompanying detection time windows, work area identifiers, and work object identifiers, facilitating subsequent traceability.

[0065] Before performing compliance constraint matching, the system first determines whether the applicable conditions of the target determination rule are met. For example, whether the target detection period is within the valid time of the hot work permit, whether the target work area belongs to the hot work zone, whether the work risk level meets the rule requirements, and whether the relevant personnel or equipment are associated with the current work task. Only when the applicable conditions are met will the system perform compliance constraint matching between the rule input items and the determination threshold; if the applicable conditions are not met, the rule is marked as inapplicable to avoid misjudgment of irrelevant scenarios.

[0066] If the applicable conditions are met, and the fusion value of combustible gas concentration in the environmental safety input item exceeds the judgment threshold, the system will mark the environmental safety constraint item as hit; if the welding machine is in the start state but the exhaust equipment is not in the effective operation state, the system will mark the equipment linkage constraint item as hit; if the personnel entering the warning area do not match the work permit, the system will mark the personnel access or area control constraint item as hit.

[0067] When generating rule judgment results, the system determines the rule judgment level based on the number of violated constraints, the risk level of the constraints, and the duration. For example, a single low-risk warning can constitute a minor violation; exceeding environmental thresholds and simultaneous equipment malfunctions can constitute a serious violation; and failing to meet environmental safety conditions while personnel continue to perform hot work can constitute a major safety hazard. The rule judgment result records the rule hit status, the violated constraints, the rule judgment level, the hit rule identifier, the trigger time window, the associated work object, and the judgment basis.

[0068] Through the above methods, the target determination rules can be integrated with the characteristics of the on-site status to form a clear field mapping and constraint matching relationship, making the rule determination process executable, interpretable and traceable, and improving the accuracy of the execution of hard compliance standards in high-risk operation scenarios such as hot work and hoisting operations.

[0069] In some embodiments, hierarchical compliance assessment is performed based on rule-based assessment results and model-based assessment results to generate on-site compliance inspection results, including: Based on the rule-based judgment results, determine whether the compliance matter to be judged triggers mandatory compliance constraints, and generate the corresponding deterministic judgment identifier; The composite risk category and risk confidence level corresponding to the compliance matter to be judged are determined based on the model's judgment results; The judgment is fused based on deterministic judgment identifiers, composite risk categories, and risk confidence levels. When there is a conflict between the rule-based judgment results and the model-based judgment results, the conflict is adjudicated according to the preset risk priority, generating on-site compliance detection results that include compliance status, violation category, and risk level.

[0070] Specifically, in the scenario of compliance detection for hot work operations on a three-story platform at a construction site, after obtaining the rule-based judgment results and the model-based judgment results, the system first identifies mandatory compliance constraints on the rule-based judgment results. Mandatory compliance constraints can originate from industry safety regulations, enterprise hot work standards, and on-site work permit conditions, and are mainly used to limit hard compliance requirements that cannot be weakened by model inference results. For example, situations such as an invalid hot work permit, flammable gas concentration exceeding a safety threshold, welding machines being started without ventilation equipment, and unauthorized personnel entering the hot work warning area can all be configured as mandatory compliance constraints.

[0071] The system reads the rule hit status, violation constraints, and rule judgment level from the rule judgment results, determines whether the current compliance matter to be judged hits mandatory compliance constraints, and generates a deterministic judgment identifier accordingly. The deterministic judgment identifier can include statuses such as compliance passed, mandatory violation, mandatory work stoppage, and pending review, which are used to reflect the deterministic judgment of the operation process at the rule level.

[0072] At the model judgment level, the system analyzes the model judgment results output by the compliance judgment model to determine the composite risk category and risk confidence level corresponding to the compliance matter to be judged. The composite risk category is used to represent the risk type that is difficult for rules to directly cover but is reflected by a combination of multimodal states, such as the risk of deterioration of hot work environment, the risk of abnormal equipment linkage, the risk of deviation of personnel behavior, and the risk of failure of regional control. The risk confidence level is used to represent the reliability of the model's judgment on the composite risk category, and can be jointly determined by the model output probability, the completeness of input features, the modality confidence level, and the degree of matching of historical samples.

[0073] For example, if within a certain detection time window, the concentration of combustible gas has not yet exceeded the threshold but shows a continuous upward trend, the welding machine is running, the exhaust equipment is intermittently shut down, and the operator is still in a welding posture, the model can identify this state as a risk of deterioration of the hot work environment and output a high risk confidence level. As another example, if the camera is unable to stably identify protective equipment due to smoke interference, but the personnel location, welding machine operation, and posture changes collectively indicate that the personnel are performing close-range hot work, the model can output a personnel protection risk category with a corresponding confidence level.

[0074] Subsequently, the system performs a fusion of judgments based on deterministic judgment identifiers, composite risk categories, and risk confidence levels. During the fusion process, the system prioritizes retaining deterministic judgments that hit mandatory compliance constraints, while using composite risks identified by the model as the basis for risk level correction. If the deterministic judgment identifier indicates compliance, but the model judgment result shows a high-confidence composite risk, the system can adjust the detection result from compliance to risk warning or minor violation, and record the basis for the model judgment. If the deterministic judgment identifier indicates mandatory violation, the violation status is not canceled due to low model confidence; instead, risk category and risk explanation information are supplemented based on the model judgment result.

[0075] For example, if the rule layer determines that the concentration of combustible gas is not exceeded, but the model layer identifies that the environmental reading is continuously close to the threshold and the ventilation status is unstable, the system can mark the compliance status as having potential hazards and raise the risk level to the warning level. If the rule layer has already hit the rule that the exhaust equipment is not activated, the system will directly generate the equipment linkage violation result and determine whether to further upgrade it to a serious violation based on the model output.

[0076] When there is a conflict between the rule-based judgment and the model-based judgment, the system adjudicates the conflict according to a preset risk priority. Conflicts can include situations where the rule layer determines compliance while the model layer determines high risk, the rule layer determines minor violation while the model layer determines major risk, or the corresponding violation categories at the rule layer and model layer are inconsistent. The preset risk priority can be configured based on personnel safety, environmental hazard level, probability of equipment malfunction, and sensitivity of the work area. For high-risk matters involving open flames, excessive gas levels, or personnel entering hazardous areas, the system adopts a high-risk priority principle; for low-risk matters where the model confidence is insufficient and the rule is not matched, the system can generate a result pending review. After the adjudication is completed, the system determines the final compliance status, final violation category, and final risk level, and writes the rule basis, model basis, conflict type, and adjudication basis into the work site compliance inspection results.

[0077] Through the above-mentioned hierarchical compliance judgment method, the system can ensure that mandatory compliance rules are stably executed in high-risk operation scenarios, and can also use the compliance judgment model to identify hidden violations and complex risks. This avoids the problems of overly mechanical judgment by a single rule or the lack of deterministic basis for judgment by a single model, thereby improving the accuracy, reliability and interpretability of compliance detection results at the work site.

[0078] In some embodiments, when there is a conflict between the rule-based judgment result and the model-based judgment result, a conflict resolution is performed according to a preset risk priority to generate an on-site compliance inspection result that includes compliance status, violation category, and risk level, including: When the compliance status or risk level corresponding to the rule judgment result is inconsistent with the model judgment result, the conflict judgment item is determined, and the rule hit status, rule judgment level, composite risk category and risk confidence level corresponding to the conflict judgment item are extracted. Based on the preset risk priority, the rule judgment level and composite risk category are prioritized and compared, and the decision is determined in combination with the risk confidence level. Based on the ruling, the final compliance status, final violation category, and final risk level are determined, and on-site compliance inspection results are generated.

[0079] Specifically, in the compliance detection scenario for hot work operations on a three-story platform at a construction site, after completing rule-based and model-based judgments, the system first compares the consistency of their outputs. Using the target detection period, target work area, and work object as indexes, the system reads the rule-based and model-based judgment results corresponding to the same object and compares their differences in compliance status, violation category, and risk level. For example, within a 5-second detection window, the rule-based judgment result shows that the combustible gas concentration does not exceed the safety threshold, the work permit is valid, and the welding machine's operating status meets the work permit requirements, thus providing a compliance or minor warning level. However, the compliance judgment model, based on fusion characteristics such as continuously rising combustible gas concentration, intermittent shutdown of ventilation equipment, workers still maintaining welding postures, and close proximity of surrounding materials, outputs a risk of deterioration of the hot work environment with a high confidence level. In this case, the system determines that the rule-based judgment result and the model-based judgment result are inconsistent in risk level and identifies the environmental safety issue as a conflict judgment item.

[0080] Furthermore, after determining the conflict criteria, the system extracts the rule hit status, rule judgment level, composite risk category, and risk confidence level corresponding to the conflict criteria. The rule hit status indicates whether the target judgment rule triggers mandatory compliance constraints, such as exceeding environmental limits, equipment linkage failure, unlicensed personnel operation, and intrusion into a restricted area; the rule judgment level indicates the classification result of the severity of the violation at the rule level, such as compliance, minor violation, serious violation, and major hidden danger.

[0081] Composite risk categories are used to represent the types of risks identified by the compliance assessment model, such as abnormal environmental trends, unstable equipment linkages, deviations in personnel behavior, and failures in regional control. Risk confidence level indicates the reliability of the model's assessment results and can be determined by combining model output probability, input feature completeness, modal credibility, and historical sample similarity. The system combines the above information into conflict adjudication data units and retains the time window, area identifier, personnel identifier, equipment identifier, and associated on-site data of the conflict, facilitating the subsequent generation of traceable adjudication evidence.

[0082] The system pre-configures a risk priority table, which can be determined based on the type of operation, impact on personnel safety, environmental hazard level, likelihood of equipment malfunction, and area sensitivity. Taking hot work as an example, uncontrolled open flame environment, abnormal gas concentration, ventilation equipment failure, and unauthorized personnel entering the warning area can be configured as higher priorities; ordinary image obstruction, insufficient confidence in protective equipment identification, or short-term fluctuations in single-point sensors can be configured as lower priorities. For hoisting operations, personnel remaining within the boom's rotation path, hoisting equipment overload, and warning area failure can be configured as higher priorities. When executing conflict resolution, the system maps the rule judgment level and composite risk category to the risk priority table, obtaining rule-side risk priority values ​​and model-side risk priority values, and combines these with risk confidence levels to determine the final resolution tendency.

[0083] For example, if the rule judgment result is compliant, but the model judgment result is an abnormal environmental trend and the risk confidence level reaches the preset high confidence condition, the system can determine the adjudication result as a risk warning or minor violation according to the high-risk priority principle, and raise the risk level to the warning level; if the rule judgment result has hit the mandatory compliance constraint, such as the welding machine is started but the ventilation equipment is not started, even if the model outputs a low risk due to image occlusion, the system still takes the rule judgment level as the main factor and adjudicates it as a serious violation or major hidden danger; if the rule judgment result is a minor violation, but the model judgment result shows that there is deviation in personnel behavior and the confidence level is low, the system can maintain the rule-side level and write the model-side risk as an auxiliary explanation into the detection result; if the model judgment result is high risk but the corresponding input features have insufficient modal credibility, the system can generate an adjudication result to be reviewed, while maintaining the issuance of warnings to avoid potential high-risk scenarios being ignored.

[0084] Furthermore, after generating the ruling, the system determines the final compliance status, final violation category, and final risk level based on the ruling. The final compliance status can indicate whether the current operation is compliant, whether there are risk warnings, whether it constitutes a violation, and whether work stoppage is required. The final violation category can be determined based on the primary source of the ruling, such as environmental violation, equipment violation, personnel violation, operational violation, or a combination of violations. The final risk level can be determined by combining the rule judgment level, the combination of risk categories, the risk confidence level, and the duration.

[0085] For example, in hot work operations on a three-story platform, if the system detects that the ventilation equipment is not continuously operating and the concentration of combustible gas is close to the threshold, while personnel continue welding, the final compliance status can be determined as a violation. The final violation category is determined as a combined environmental and equipment violation, and the final risk level is determined as a serious violation or a major safety hazard. The system also writes the rule hits, model risk categories, conflict reasons, priority comparison results, and adjudication basis into the on-site compliance detection results.

[0086] By employing the aforementioned conflict resolution mechanism, simple averaging or random selection can be avoided when rule-based and model-based judgments differ, thus unifying and coordinating mandatory compliance rules, high-confidence composite risks, and the principle of prioritizing on-site safety. This approach ensures that hard safety standards are not weakened by model misjudgments and can identify trending and composite risks in advance before rules are triggered, thereby improving the robustness, accuracy, and safety early warning value of on-site compliance testing results.

[0087] In some embodiments, the location, object, type, and risk level of the violation are determined based on the compliance inspection results at the work site, and corresponding early warning information and compliance inspection records are generated, including: Based on the compliance inspection results at the work site, the spatial location markers, work object markers, and judgment criteria within the target inspection period are associated with the violation location, violation object, violation type, and risk level. Based on the risk level, a corresponding early warning strategy is matched to generate early warning information that includes an early warning identifier, a response level, and the target audience, and the corresponding early warning is issued. The early warning information is linked and stored with the compliance test results, judgment criteria, and related on-site data at the work site to generate traceable compliance test records.

[0088] Specifically, in the scenario of compliance inspection for hot work operations on a three-story platform at a construction site, after generating the compliance inspection results, the system first locates the violation and confirms its attribution based on the spatial location identifiers, work object identifiers, and judgment criteria within the target inspection period. Spatial location identifiers can point to floors, areas, coordinate units, camera coverage areas, and sensor operating ranges; work object identifiers can point to personnel, equipment, work tasks, and environmental monitoring points. Based on the final compliance status, final violation category, and final risk level in the compliance inspection results, the system backtracks the corresponding multimodal on-site data to determine the target work area, specific coordinate location, and scope of impact of the violation.

[0089] For example, during hot work on a three-story platform, if the system detects that the welding machine has been started but the ventilation equipment is not operating effectively, and the concentration of combustible gas is increasing while the operator is still in a welding posture, the system will identify the location of the violation as the welding point and the surrounding warning area within the hot work area of ​​the three-story platform, identify the objects of the violation as the corresponding welding machine, ventilation equipment and the current operator, identify the type of violation as a combination of equipment linkage abnormality and environmental risk violation, and identify the risk level as a serious violation or a major safety hazard.

[0090] When determining the location and object of a violation, the system can also perform coordinate conversion by combining the on-site electronic map and work space coordinates, transforming the detection results from internal coordinate units into area names and graphic annotations that can be recognized by supervisors. For example, the system can mark the hot work area on the east side of the three-story platform on the on-site monitoring interface and overlay personnel frames, equipment frames, warning boundaries, and risk area outlines on the video screen. For personnel violations, the system determines the identity or category of the violating personnel based on personnel location tags, access control check-in records, and visual recognition results; for equipment violations, the system determines the equipment name, equipment number, and abnormal operating condition type based on equipment ledgers, data collection node numbers, and equipment status records; for environmental violations, the system determines the exceeded area and its impact range based on sensor spatial location identifiers and sensing radius. The judgment criteria can include rule hits, model risk categories, risk confidence levels, sensor readings, equipment status, video snapshots, and conflict resolution criteria.

[0091] Subsequently, the system matches corresponding early warning strategies based on the risk level. Early warning strategies can be configured in tiers according to risk levels such as minor violations, serious violations, and major safety hazards, and are linked to on-site handling procedures, personnel responsibilities, and information dissemination channels. For minor violations, the system can generate on-site prompts, push them to the team leader's or on-site safety officer's terminal, and record them in the background log. For serious violations, the system can trigger pop-up prompts, mobile messages, and on-site audible and visual alarms, requiring responsible personnel to confirm and handle the situation within a preset time. For major safety hazards, the system can issue high-level early warnings, simultaneously pushing the warning information to the project safety manager, the duty dispatch platform, and the enterprise safety management backend, and, if necessary, linking the equipment control system to output shutdown or work suspension commands. For example, when the concentration of combustible gas in the hot work area approaches the threshold, the ventilation equipment is not activated, and personnel continue hot work, the system generates a high-level early warning indicator, with the handling level configured to immediately stop hot work and check the ventilation equipment. The notification recipients include the workers, safety officer, project manager, and monitoring center.

[0092] When an alert is generated, the system structurally encapsulates the location of the violation, the object of the violation, the type of violation, the risk level, the handling suggestions, the recipients, and the alert time. The alert identifier can be generated by combining the work area code, the detection time window, the violation type code, and the risk level code, facilitating subsequent querying and deduplication. The handling level can be determined based on the risk level, the duration of the violation, and whether mandatory compliance constraints are met. For the same violation event detected in multiple consecutive detection time windows, the system can merge and generate continuous alerts to avoid interference from repeated pushes; for events with escalating risk levels, the system can generate escalation alerts and retain records of the level changes. After an alert is issued, the system records the recipients, delivery status, confirmation status, and handling feedback, enabling closed-loop management of the violation handling process.

[0093] During the compliance inspection record generation phase, the system binds and stores early warning information, on-site compliance inspection results, rule-based judgment criteria, model-based judgment criteria, conflict resolution criteria, and related on-site data. Related on-site data can include video clips, key image frames, sensor reading sequences, equipment operation records, personnel location trajectories, and work permit information within the target inspection period. The system establishes an index according to work tasks, work areas, violation types, and risk levels, forming traceable compliance inspection records. For daily supervision, the system can automatically generate daily, weekly, and monthly compliance inspection reports based on these records, statistically analyzing the frequency of violations, high-risk areas, common violation types, and rectification closure status. For incident debriefing or regulatory audits, the system can quickly replay multimodal on-site data and judgment criteria within the corresponding time window based on early warning indicators.

[0094] Through the above processing, the system can transform compliance test results into locationable, pushable, actionable, and traceable regulatory data, extending the process of violation incidents from the identification stage to the early warning issuance, handling tracking, and evidence retention stages, thereby improving the real-time risk control capabilities and closed-loop management capabilities of compliance supervision at the work site.

[0095] The following are system embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0096] Figure 2 This is a schematic diagram of the structure of the on-site compliance detection system based on multimodal data fusion provided in this application embodiment. Figure 2 As shown, the system includes: The acquisition module 201 is used to acquire multimodal raw data representing the work object, work environment and work process in the work site, and perform spatiotemporal correlation on the multimodal raw data based on a unified time reference and work space coordinates to generate multimodal field data corresponding to the target detection period and target work area; The extraction module 202 is used to perform modal adaptation preprocessing on multimodal field data and extract compliance detection features to characterize the field's visual state, operational state, environmental state, and behavioral state. The fusion module 203 is used to determine the fusion weights corresponding to each compliance detection feature based on the target operation scenario, modal data quality and compliance detection task, and to adaptively fuse each compliance detection feature to generate on-site status fusion features. The judgment module 204 is used to input the fusion features of the on-site status into the compliance rule base and the compliance judgment model respectively, obtain the rule judgment results and the model judgment results, and perform hierarchical compliance judgment based on the rule judgment results and the model judgment results to generate the on-site compliance detection results. The generation module 205 is used to determine the location, object, type and risk level of the violation based on the compliance inspection results at the work site, and to generate corresponding early warning information and compliance inspection records.

[0097] In some embodiments, Figure 2 The acquisition module 201 configures multi-source acquisition nodes according to the regional division, equipment layout and operation process of the work site, and assigns acquisition object identifiers and spatial location identifiers to the multi-source acquisition nodes; based on the unified sampling strategy, the multi-source acquisition nodes are driven to collect on-site perception data, equipment operation data and personnel operation data corresponding to the target detection period; the acquired data is appended with acquisition timestamps, acquisition object identifiers and spatial location identifiers, and encapsulated according to data source and operation association to generate multimodal raw data.

[0098] In some embodiments, Figure 2 The acquisition module 201 performs time calibration on the multimodal raw data according to the acquisition timestamp to determine the detection time window of each data record relative to a unified time reference; it maps each data record to the work space coordinates according to the spatial location identifier to determine the work area and work object corresponding to each data record; based on the detection time window, work area and work object, it aggregates data records whose time offset meets the preset association conditions and whose spatial location meets the preset matching conditions to generate multimodal field data corresponding to the target detection period and target work area.

[0099] In some embodiments, Figure 2 The extraction module 202 determines the preprocessing strategy corresponding to each modality of data based on the data type and acquisition quality of the multimodal field data. It performs image quality correction and target area parsing on image data, abnormal data correction and unit unification processing on numerical data, and status field parsing and job semantic mapping on event data. The preprocessed modal data is converted into a unified feature representation, and status features related to compliance judgment are extracted by combining the target detection period, target operation area and operation object to generate compliance detection features.

[0100] In some embodiments, Figure 2 The fusion module 203 parses the scenario constraint parameters corresponding to the target operation scenario and determines the modal credibility of each compliance detection feature based on the quality of modal data; it determines the task relevance of each compliance detection feature to the compliance matter to be judged based on the compliance detection task, and generates the fusion weight corresponding to each compliance detection feature by combining the scenario constraint parameters, modal credibility, and task relevance; it performs weighted fusion of each compliance detection feature according to the fusion weight, and performs cross-modal consistency verification on the fusion result to generate the field status fusion feature used to characterize the comprehensive operation status within the target detection period.

[0101] In some embodiments, Figure 2 The judgment module 204, based on the target operation scenario and compliance detection task, calls the target judgment rule corresponding to the compliance matter to be judged from the compliance rule library; maps the on-site state fusion features to the rule input items of the target judgment rule, and performs compliance constraint matching on the rule input items based on the target judgment rule to generate the rule judgment result; inputs the on-site state fusion features into the trained compliance judgment model, performs risk classification and identification on the composite operation status that is difficult to be directly covered by the rule constraints, and generates the model judgment result.

[0102] In some embodiments, Figure 2 The judgment module 204 parses the constraint fields, applicable conditions, and judgment thresholds corresponding to the target judgment rule, and determines the feature mapping relationship that matches the fusion features of the on-site status. Based on the feature mapping relationship, it extracts the status parameters corresponding to the compliance matters to be judged from the fusion features of the on-site status, and converts the status parameters into rule input items. When the applicable conditions are met, it matches the rule input items with the judgment threshold for compliance constraints, and generates a rule judgment result that includes the rule hit status, violation constraint items, and rule judgment level.

[0103] In some embodiments, Figure 2 The judgment module 204 determines whether the compliance matter to be judged triggers mandatory compliance constraints based on the rule judgment result and generates a corresponding deterministic judgment identifier; it determines the composite risk category and risk confidence level corresponding to the compliance matter to be judged based on the model judgment result; it performs judgment fusion based on the deterministic judgment identifier, composite risk category and risk confidence level, and when there is a conflict between the rule judgment result and the model judgment result, it performs conflict adjudication according to the preset risk priority, and generates on-site compliance detection results including compliance status, violation category and risk level.

[0104] In some embodiments, Figure 2The generation module 205 determines the location, object, type, and risk level of violations by associating the spatial location identifier, object identifier, and judgment criteria within the target detection period with the compliance detection results at the work site. Based on the risk level, it matches the corresponding early warning strategy, generates early warning information containing early warning identifier, handling level, and target audience, and executes the corresponding early warning release. It then associates and stores the early warning information with the compliance detection results, judgment criteria, and associated on-site data to generate a traceable compliance detection record.

[0105] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0106] Figure 3 This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various system embodiments described above.

[0107] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0108] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0109] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0112] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for on-site compliance detection based on multimodal data fusion, characterized in that, include: Acquire multimodal raw data representing the work object, work environment and work process in the work site, and perform spatiotemporal correlation on the multimodal raw data based on a unified time reference and work space coordinates to generate multimodal field data corresponding to the target detection time period and target work area; Modal adaptation preprocessing is performed on the multimodal field data to extract compliance detection features that characterize the field's visual state, operational state, environmental state, and behavioral state. Based on the target operation scenario, modal data quality, and compliance inspection task, the fusion weights corresponding to each compliance inspection feature are determined, and each compliance inspection feature is adaptively fused to generate on-site status fusion features. The fused features of the on-site status are input into the compliance rule base and the compliance judgment model respectively to obtain the rule judgment results and the model judgment results. Based on the rule judgment results and the model judgment results, hierarchical compliance judgment is performed to generate on-site compliance detection results. Based on the compliance inspection results at the work site, the location, object, type, and risk level of the violation are determined, and corresponding early warning information and compliance inspection records are generated.

2. The method according to claim 1, characterized in that, The acquisition of multimodal raw data representing the work object, work environment, and work process at the work site includes: Configure multi-source acquisition nodes according to the area division, equipment layout and operation process of the work site, and assign acquisition object identifiers and spatial location identifiers to the multi-source acquisition nodes; Based on a unified sampling strategy, the multi-source acquisition nodes collect on-site perception data, equipment operation data, and personnel operation data corresponding to the target detection period. The collected data is appended with a collection timestamp, a collection object identifier, and a spatial location identifier, and then encapsulated according to the data source and job association to generate the multimodal raw data.

3. The method according to claim 2, characterized in that, The process involves spatiotemporally correlating the multimodal raw data based on a unified time reference and operational space coordinates to generate multimodal field data corresponding to the target detection time period and target operational area, including: The multimodal raw data is time-calibrated based on the acquisition timestamp to determine the detection time window of each data record relative to the unified time reference. Based on the spatial location identifier, each data record is mapped to the work space coordinates to determine the work area and work object corresponding to each data record; Based on the detection time window, the work area, and the work object, data records whose time offset meets preset association conditions and whose spatial location meets preset matching conditions are aggregated to generate the multimodal field data corresponding to the target detection time period and the target work area.

4. The method according to claim 1, characterized in that, The modal adaptation preprocessing of the multimodal field data, extracting compliance detection features to characterize the field's visual state, operational state, environmental state, and behavioral state, includes: Based on the data type and acquisition quality of the multimodal field data, the preprocessing strategy corresponding to each modality of data is determined, and image quality correction and target area parsing are performed on image data, abnormal data correction and unit unification processing are performed on numerical data, and status field parsing and job semantic mapping are performed on event data. The preprocessed modal data are converted into a unified feature representation, and the state features related to compliance determination are extracted by combining the target detection period, target operation area and operation object to generate the compliance detection features.

5. The method according to claim 1, characterized in that, The process involves determining the fusion weights corresponding to each compliance detection feature based on the target operation scenario, modal data quality, and compliance detection task, and then adaptively fusing these features to generate on-site state fusion features, including: The scenario constraint parameters corresponding to the target operation scenario are analyzed, and the modal credibility of each compliance detection feature is determined based on the modal data quality. Based on the compliance detection task, the task relevance of each compliance detection feature to the compliance matter to be judged is determined, and the fusion weight corresponding to each compliance detection feature is generated by combining the scenario constraint parameters, the modal credibility and the task relevance. The compliance detection features are weighted and fused according to the fusion weights, and the fusion results are cross-modal consistency verification to generate the field status fusion features used to characterize the comprehensive operation status during the target detection period.

6. The method according to claim 1, characterized in that, The step of inputting the fused features of the on-site state into the compliance rule base and the compliance judgment model respectively to obtain the rule judgment result and the model judgment result includes: Based on the target operation scenario and the compliance detection task, the target judgment rule corresponding to the compliance matter to be judged is called from the compliance rule library; The on-site state fusion features are mapped to rule input items of the target determination rule, and compliance constraint matching is performed on the rule input items based on the target determination rule to generate the rule determination result; The fused features of the on-site conditions are input into the trained compliance judgment model to perform risk classification and identification on complex operation states that are difficult to be directly covered by rule constraints, and generate the model judgment result.

7. The method according to claim 6, characterized in that, The step of mapping the fused features of the on-site state to rule input items of the target determination rule, and performing compliance constraint matching on the rule input items based on the target determination rule to generate the rule determination result includes: The constraint fields, applicable conditions, and judgment thresholds corresponding to the target determination rule are analyzed to determine the feature mapping relationship that matches the fusion features of the on-site state. Based on the feature mapping relationship, state parameters corresponding to the compliance matters to be judged are extracted from the on-site state fusion features, and the state parameters are converted into the rule input items; When the applicable conditions are met, the rule input items are matched with the judgment threshold for compliance constraints to generate the rule judgment result, which includes the rule hit status, violation constraint items, and rule judgment level.

8. The method according to claim 6, characterized in that, The hierarchical compliance assessment based on the rule-based judgment results and the model-based judgment results, generating on-site compliance inspection results, includes: Based on the results of the rule determination, it is determined whether the compliance matter to be judged triggers mandatory compliance constraints, and a corresponding deterministic judgment identifier is generated; Based on the model's judgment results, determine the composite risk category and risk confidence level corresponding to the compliance matter to be judged; The determination is fused based on the deterministic determination identifier, the composite risk category, and the risk confidence level. When there is a conflict between the rule determination result and the model determination result, the conflict is adjudicated according to the preset risk priority, and the on-site compliance detection result including compliance status, violation category, and risk level is generated.

9. The method according to claim 1, characterized in that, The process of determining the location, object, type, and risk level of the violation based on the compliance inspection results at the work site, and generating corresponding early warning information and compliance inspection records, includes: Based on the compliance inspection results at the work site, the spatial location identifiers, work object identifiers, and judgment criteria within the target inspection period are associated with the violation location, violation object, violation type, and risk level. Based on the risk level, a corresponding early warning strategy is matched to generate early warning information containing an early warning identifier, a handling level, and a target audience, and the corresponding early warning is issued. The warning information is associated with the compliance inspection results at the work site, the judgment criteria, and the related on-site data and stored together to generate a traceable compliance inspection record.

10. A work site compliance detection system based on multimodal data fusion, characterized in that, include: The acquisition module is used to acquire multimodal raw data representing the work object, work environment and work process in the work site, and perform spatiotemporal correlation on the multimodal raw data based on a unified time reference and work space coordinates to generate multimodal field data corresponding to the target detection time period and target work area; The extraction module is used to perform modal adaptation preprocessing on the multimodal field data and extract compliance detection features to characterize the field visual state, operating state, environmental state and behavioral state. The fusion module is used to determine the fusion weights corresponding to each compliance detection feature based on the target operation scenario, modal data quality, and compliance detection task, and to adaptively fuse each compliance detection feature to generate on-site status fusion features. The judgment module is used to input the fused features of the on-site status into the compliance rule base and the compliance judgment model respectively, obtain the rule judgment results and the model judgment results, and perform hierarchical compliance judgment based on the rule judgment results and the model judgment results to generate the on-site compliance detection results. The generation module is used to determine the location, object, type and risk level of the violation based on the compliance inspection results at the work site, and to generate corresponding early warning information and compliance inspection records.

Images