Multi-modal behavior recognition and compliance judgment method and system for high-risk operation

Abstract

The present application belongs to the technical field of safety monitoring of high-risk operations, and relates to a multi-modal behavior recognition and compliance judgment method and system for high-risk operations, comprising obtaining visible light images and safety specification data of a high-risk operation area, pre-processing the visible light images to obtain standard visible light images, and constructing a multi-dimensional structured compliance text rule library; extracting visible light behavior features in the standard visible light images and compliance text semantic features in the multi-dimensional structured compliance text rule library; dynamically weighting and fusing the two to obtain a multi-modal feature vector; converting the compliance text semantic features into a calculable judgment index, comparing the index with the multi-modal feature vector, triggering different levels of risk warnings according to the comparison results; based on the multi-dimensional structured compliance text rule library, performing a violation check on the multi-modal feature vector triggering the risk warning to obtain a violation reason, and realizing automatic recognition and full-process compliance judgment of high-risk operation behaviors.

Description

Technical Field

[0001] This invention belongs to the field of high-risk operation safety monitoring technology, specifically relating to a method and system for multimodal behavior recognition and compliance judgment for high-risk operations. Background Technology

[0002] High-risk operations (such as hot work, high-altitude work, confined space work, and high-voltage equipment operation) are characterized by complex environments, numerous safety hazards, and serious consequences of accidents. Problems such as personnel violations, abnormal equipment conditions, and exceeding environmental parameters during the operation process can easily lead to safety accidents such as fires, explosions, electric shocks, and personal injuries. Therefore, real-time monitoring and compliance assessment of high-risk operations are crucial.

[0003] Currently, safety monitoring of high-risk operations mainly relies on manual inspections and single-modal monitoring equipment, which has many limitations: First, manual inspections are inefficient, highly subjective, and easily affected by factors such as fatigue and negligence, making it impossible to achieve 24-hour uninterrupted monitoring and difficult to detect hidden safety hazards in a timely manner; Second, single-modal monitoring equipment has monitoring blind spots. For example, the recognition accuracy of visible light cameras drops significantly in low-light or smoke-filled environments, resulting in insufficient accuracy in behavior recognition and compliance judgment; Third, existing monitoring methods lack a unified compliance rule conversion mechanism, failing to transform textual safety specifications into calculable and comparable judgment indicators, making it difficult to achieve automatic verification and accurate early warning of violations; Fourth, in existing technologies, dual-channel or multi-channel monitoring equipment lacks a synchronous triggering mechanism, resulting in time differences in the collected multimodal data, leading to low feature fusion accuracy and further affecting the reliability of compliance judgment.

[0004] Furthermore, existing technologies for compliance assessment of high-risk operations often only trigger simple risk warnings, failing to accurately pinpoint the causes of violations or provide targeted handling measures based on different violation scenarios. This results in low efficiency in post-warning response and difficulty in effectively preventing the escalation of accidents. Therefore, a new method for identifying high-risk operational behaviors is urgently needed to address the shortcomings of existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for multimodal behavior recognition and compliance judgment for high-risk operations, so as to solve the technical problems of single data and low positioning accuracy in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for multimodal behavior recognition and compliance assessment for high-risk operations, comprising: Acquire visible light images and safety specification data of high-risk work areas; preprocess the visible light images to obtain standard visible light images; and construct a multi-dimensional structured compliance text rule base based on the safety specification data. Visible light behavior features are extracted from standard visible light images, and compliant text semantic features are extracted from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The semantic embedding method is used to transform the semantic features of compliant texts into computable judgment indicators. The multimodal feature vectors are compared with the computable judgment indicators, and different levels of risk warnings are triggered based on the comparison results. Based on a multi-dimensional structured compliance text rule base, the violation verification is performed on the multi-modal feature vectors that trigger risk warnings to obtain the reasons for the violation.

[0007] Optionally, the visible light image is preprocessed to obtain a standard visible light image, including: An adaptive denoising method is used to denoise the visible light image to obtain a denoised visible light image; The denoised visible light image is standardized to obtain a standard visible light image.

[0008] Optionally, the visible light images are acquired from different high-risk work areas, and noise reduction is performed using filtering or restoration methods respectively: The visible light images collected from the fuel oil storage area and boiler maintenance area were filtered out using Gaussian filtering to remove flue gas interference and environmental noise. For visible light images acquired in high-voltage power distribution rooms, median filtering is used to remove electromagnetic radiation clutter. For visible light images acquired at hazardous chemical transfer channels, a bilateral filtering method was used to remove weak light noise. For visible light images acquired in high-altitude work areas, motion blur restoration method is used to eliminate jitter interference.

[0009] Optionally, visible light behavior features are extracted from standard visible light images, and compliant text semantic features are extracted from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector, specifically including: The global and local detail features of a standard visible light image are jointly extracted to output visible light behavior features. The text in the multi-dimensional structured compliance text rule base is segmented, stop words are removed, and the format is standardized to obtain a unified text. The unified text is then processed by word vector encoding, context semantic encoding, and pooling to obtain the semantic feature vector of each rule. The semantic vectors of all rules in the same high-risk operation area are then fused by mean and normalized by L2 to output the semantic features of the compliance text. The dynamic weighting weights of different high-risk work areas are calculated using the softmax function. The weighted visible light behavior features are then fused with the semantic features of compliant text to obtain a multimodal feature vector.

[0010] Optionally, a semantic embedding method is used to transform the semantic features of compliant text into computable judgment indicators. The multimodal feature vector is compared with the computable judgment indicators, and different levels of preset risk warnings are triggered based on the comparison results, specifically including: The structured compliance text rule base is decomposed into sentences and segments to extract the core semantic units of each rule; the core semantic units are mapped into semantic feature vectors using semantic embedding, and numerical and Boolean constraints in the rule base are extracted. Semantic feature vectors are mapped to Boolean constraints to behavior category determination indicators, and numerical constraints are mapped to numerical determination indicators. The behavior category determination indicators and numerical determination indicators are standardized in format to obtain computable determination indicators. Key sub-features are extracted from the multimodal feature vector, including: behavior recognition features and personnel presence features; these key sub-features are then compared with computable judgment indicators. When the behavior recognition feature is a smoking action, a violation smoking warning is triggered; When the behavior identification feature is that there is obvious open flame and no hot work permit, an illegal open flame warning is triggered; When the personnel are found to be unqualified to enter, an unauthorized entry warning is triggered. When the behavioral recognition feature is that the person is not wearing a safety belt or helmet at high altitude, a protection deficiency warning is triggered. When the behavior identification feature indicates that the device is not operated according to the standard procedure, a violation operation warning is triggered.

[0011] Optionally, the multi-dimensional structured compliance text rule base includes six dimensions, specifically: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness.

[0012] Optionally, based on a multi-dimensional structured compliance text rule base, violation verification is performed on the multi-modal feature vectors that trigger risk warnings to obtain the reasons for the violations, specifically including: Based on the triggered risk warnings, all rule clauses that are directly, indirectly, or constrained by the risk warnings are retrieved and screened from six dimensions: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness. After deduplication and sorting, a set of rules to be verified is formed. Based on the set of rules to be verified, logical operations are used to verify violations of the multimodal feature vectors that trigger risk warnings, in order to obtain the reasons for the violations. The multimodal feature vectors include behavioral recognition features, personnel presence features, scene information, and state information. The logical operations include: AND, OR, NOT, and implication operations. Specifically: The behavioral recognition features, personnel presence features, scene information, and status information in the multimodal feature vector that triggers the risk warning are used as logical judgment inputs. Logical operations are performed on the input and each rule in the set of rules to be verified in turn, and all rule items that do not meet the constraints, violate the prohibition clauses, fail to meet the requirements, or are not valid are marked. Merge, deduplicate, and prioritize conflicting rules, and output a unique or combined reason for violation.

[0013] In a second aspect, the present invention provides a multimodal behavior recognition and compliance assessment system for high-risk operations, comprising: The preprocessing unit is used to acquire visible light images and safety specification data of high-risk work areas, preprocess the visible light images to obtain standard visible light images, and construct a multi-dimensional structured compliance text rule base based on the safety specification data. An extraction and fusion unit is used to extract visible light behavior features from standard visible light images and extract compliant text semantic features from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The judgment unit is used to convert the semantic features of compliant text into computable judgment indicators using semantic embedding, compare the multimodal feature vector with the computable judgment indicators, and trigger different levels of risk warnings based on the comparison results. The verification unit is used to perform violation verification on the multimodal feature vectors that trigger risk warnings based on a multidimensional structured compliance text rule base, and to obtain the reasons for the violation.

[0014] In a third aspect, the present invention provides an electronic device, characterized in that it includes a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement a multimodal behavior recognition and compliance judgment method for high-risk operations as described in any one of the preceding claims.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the multimodal behavior recognition and compliance judgment method for high-risk operations as described in any one of the preceding claims.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method first acquires visible light images and simultaneously collects safety regulation data for the corresponding areas. Then, standard visible light images are obtained through scene-specific adaptive noise reduction and size / pixel normalization. Next, visible light behavior features are extracted using a deep learning model. A multi-dimensional structured compliance text rule base is constructed based on the safety regulation data, and semantic features of the compliance text are extracted using a semantic embedding algorithm. Then, the two types of features are fused into a multimodal feature vector according to different operation scenarios using dynamic weighting. Subsequently, the semantics of the compliance text rules are transformed into a unified format of computable judgment indicators. Threshold comparisons are used to achieve multi-level risk warnings for violations such as illegal smoking, illegal open flames, illegal trespassing, lack of protection, and illegal operations. Finally, logical operations are used based on the rule base to verify the violations of the warning behaviors, accurately determine the causes of violations, and implement differentiated handling measures, achieving automated recognition and full-process compliance judgment of high-risk operation behaviors.

[0017] Furthermore, preprocessing can adaptively select noise reduction strategies based on the interference characteristics of different operating scenarios and unify image specifications, providing high-quality, standardized input data for subsequent feature extraction.

[0018] Furthermore, the dynamic weighted fusion method can adaptively adjust the weights of visual features and textual semantic features according to the environmental characteristics and monitoring priorities of high-risk work areas, thereby achieving effective alignment and complementarity of cross-modal features and improving the accuracy of subsequent compliance judgments and risk warnings. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a system structure block diagram according to an embodiment of the present invention; Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0021] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0022] See Figure 1 This application discloses a method for multimodal behavior recognition and compliance judgment for high-risk operations, including: Acquire visible light images and safety specification data of high-risk work areas; preprocess the visible light images to obtain standard visible light images; and construct a multi-dimensional structured compliance text rule base based on the safety specification data. Visible light behavior features are extracted from standard visible light images, and compliant text semantic features are extracted from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The semantic embedding method is used to transform the semantic features of compliant texts into computable judgment indicators. The multimodal feature vectors are compared with the computable judgment indicators, and different levels of risk warnings are triggered based on the comparison results. Based on a multi-dimensional structured compliance text rule base, the violation verification is performed on the multi-modal feature vectors that trigger risk warnings to obtain the reasons for the violation.

[0023] By acquiring visible light images and safety specification data, and extracting visible light behavioral features and compliance text semantic features respectively, and then using a cross-modal attention feature alignment method for dynamic weighted fusion, effective alignment of visual perception and text rules can be achieved, improving the accuracy and robustness of behavior recognition. Through semantic embedding, compliance semantics are transformed into computable judgment indicators and compared with multimodal feature vectors, automatically triggering graded risk warnings, realizing end-to-end automated processing from image input to risk judgment. Based on this, further violation verification is performed on the multimodal feature vectors triggering warnings using a structured compliance text rule base, accurately outputting the reasons for violations, forming a complete closed loop of warning and cause determination. This effectively solves the problems of large monitoring blind spots, inaccurate judgments, and delayed response in existing technologies, significantly improving the intelligence level and efficiency of high-risk operation safety management. It can adapt to complex operating environments, achieving full-process automation and high-precision safety management, solving problems such as large monitoring blind spots, inaccurate judgments, and delayed response in traditional methods.

[0024] Example 1 This embodiment discloses a multimodal behavior recognition and compliance judgment method for high-risk operations, the method including: S1: Visible light cameras are deployed in multiple high-risk work areas using a combination of fixed-point and mobile blind spot filling methods. The visible light acquisition frame rate of the visible light cameras is preset, and visible light images of different high-risk work areas are collected using the visible light cameras, as well as safety standard data for different high-risk work areas. Specifically, visible light cameras are deployed in multiple high-risk work areas using a combination of fixed-point and mobile blind-spot coverage methods. The visible light acquisition frame rate of the cameras is preset, and visible light images of different high-risk work areas are acquired using these cameras, including: Visible light cameras are deployed in fixed locations in the fuel oil area, boiler maintenance area, and high-voltage power distribution room. Visible light cameras are deployed using a mobile blind spot filling method in hazardous chemical transfer channels and high-altitude operation areas; The default visible light acquisition frame rate of the visible light camera is 25fps.

[0025] S2: An adaptive denoising algorithm is used to denoise the visible light image to obtain a denoised visible light image, and the denoised visible light image is then standardized to obtain a standard visible light image; including: Gaussian filtering algorithm is used to filter out flue gas interference and environmental noise in visible light images collected from fuel oil area and boiler maintenance area to obtain denoised visible light images; The electromagnetic radiation clutter in the visible light image acquired from the high-voltage power distribution room was filtered out using the median filtering algorithm to obtain a denoised visible light image. A bilateral filtering algorithm is used to filter out weak light noise in visible light images acquired from hazardous chemical transfer channels to obtain denoised visible light images; Motion blur restoration algorithm is used to eliminate jitter interference in visible light images acquired from high-altitude work areas to obtain denoised visible light images; The size of the denoised visible light image was uniformly scaled to 1024×768 pixels, and the pixels of the denoised visible light image were normalized to the [0,1] interval to obtain a standard visible light image.

[0026] S3: A deep learning model is used to extract visible light behavior features from standard visible light images. A multi-dimensional structured compliance text rule base is constructed based on safety specification data. A semantic embedding algorithm is used to extract the compliance text semantic features from the structured compliance text rule base. A cross-modal attention feature alignment algorithm is used to dynamically weight and fuse the visible light behavior features and compliance text semantic features to obtain a multi-modal feature vector. Specifically, a deep learning model is used to extract visible light behavior features from standard visible light images, including: A deep learning model was used to extract global and local detail features from standard visible light images to obtain 256-dimensional visible light behavior features.

[0027] Specifically, a multi-dimensional structured compliance text rule base is constructed based on security specification data, including: Based on safety standard data, a structured compliance text rule base with six dimensions is constructed, including work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness.

[0028] Specifically, a cross-modal attention feature alignment algorithm is used to dynamically weight and fuse visible light behavioral features and compliant text semantic features to obtain a multimodal feature vector, including: The dynamic weighting weights of different high-risk work areas were calculated using the softmax function. The dynamic weighting weight of the visible light behavior features of the fuel oil area and the boiler maintenance area was 0.6, and the dynamic weighting weight of the compliance text semantic features of the fuel oil area and the boiler maintenance area was 0.4. The dynamic weighting weight of the visible light behavior features of the high-voltage power distribution room is 0.5, and the dynamic weighting weight of the compliance text semantic features of the high-voltage power distribution room is 0.5. The dynamic weighting weight of the visible light behavior features of high-altitude operation areas and hazardous chemical transfer channels is 0.7, and the dynamic weighting weight of the compliance text semantic features of high-altitude operation areas and hazardous chemical transfer channels is 0.3. The weighted visible light behavior features and compliant text semantic features are concatenated and fused to obtain a 512-dimensional multimodal feature vector.

[0029] S4: Use semantic embedding algorithms to transform compliance semantics in the structured compliance text rule base into computable judgment indicators, and use threshold comparison method to compare multimodal feature vectors with computable judgment indicators to trigger risk warnings at different levels. Specifically, semantic embedding algorithms are used to transform compliance semantics in the structured compliance text rule base into computable judgment indicators, including: The structured compliance text rule base is decomposed into sentences and segments to extract the core semantic units of each rule in the structured compliance text rule base; The semantic embedding algorithm is used to map the core semantic units into semantic feature vectors, and numerical and Boolean constraints are extracted from the structured compliant text rule base. Semantic feature vectors are mapped to Boolean constraints as behavior category determination indicators, and numerical constraints are mapped to numerical determination indicators. The behavioral category determination index and the numerical determination index are standardized in format to obtain a calculable determination index.

[0030] Specifically, a threshold comparison method is used to compare multimodal feature vectors with computable judgment indicators to trigger different levels of risk warnings, including: Extract key sub-features from the multimodal feature vector, including: behavior recognition features and personnel presence features; compare the key sub-features with computable judgment indicators; When the behavior is identified as a smoking action, a warning for illegal smoking is triggered. When the behavior is identified as having an obvious open flame without a hot work permit, an illegal open flame warning is triggered. When personnel are found to be unqualified to enter, an unauthorized entry warning is triggered. When the behavioral recognition characteristics indicate that the person is not wearing a safety belt or helmet at height, a warning of lack of protection is triggered. When the behavior identification characteristics indicate that the equipment is not operated in accordance with the standard procedures, a violation operation warning is triggered.

[0031] S5: Based on a structured compliance text rule base, logical verification algorithms are used to verify violations of operational behaviors that trigger risk warnings, in order to determine the reasons for the violations. Different handling measures are then taken based on different reasons for the violations, including: The set of rules to be verified has been obtained by retrieving all clauses related to risk warnings from the structured compliance text rule base; Based on the set of rules to be verified, AND / OR / NOT / Implication logical operations are used to verify the violation of the work behavior that triggers the risk warning, so as to obtain the reason for the violation.

[0032] Example 2 This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method first deploys visible light cameras in each high-risk operation area using a combination of fixed-point and mobile blind-spot cameras to acquire visible light images at 25fps and simultaneously obtain corresponding regional safety regulation data. Then, it processes the visible light images using appropriate noise reduction algorithms for different operation scenarios, and standardizes image size and pixels to obtain standard images. Subsequently, a deep learning model is used to extract 256-dimensional visible light behavior features. A six-dimensional compliance text rule base is constructed based on safety regulations, and semantic features are extracted. The two types of features are dynamically weighted and fused according to differences in operation scenarios to obtain a 512-dimensional multimodal feature vector. The semantic text rules are then transformed into calculable judgment indicators. Threshold comparisons are used to identify behavior and personnel characteristics, triggering multi-level risk warnings. Finally, based on the structured rule base, AND / OR / NOT / implied logical operations are used to complete violation verification, determine the cause of the violation, and take corresponding measures, achieving accurate identification and automated compliance judgment of high-risk operation behaviors.

[0033] This embodiment provides a multimodal behavior recognition and compliance assessment method for high-risk operations. The method includes: deploying visible light cameras in multiple high-risk operation areas using a parallel approach of fixed-point and mobile blind spot compensation; presetting the visible light acquisition frame rate of the visible light cameras; acquiring visible light images of different high-risk operation areas using the cameras; acquiring safety compliance data for different high-risk operation areas; using an adaptive noise reduction algorithm to denoise the visible light images to obtain denoised visible light images; standardizing the denoised visible light images to obtain standard visible light images; and using a deep learning model to extract visible light behavior features from the standard visible light images, constructing a multi-dimensional structured database based on the safety compliance data. The compliance text rule base utilizes semantic embedding algorithms to extract semantic features of compliance texts from the structured compliance text rule base. A cross-modal attention feature alignment algorithm is then used to dynamically weight and fuse visible light behavior features and compliance text semantic features to obtain a multimodal feature vector. Semantic embedding algorithms are used to transform the compliance semantics in the structured compliance text rule base into computable judgment indicators. A threshold comparison method is used to compare the multimodal feature vector with the computable judgment indicators to trigger different levels of risk warnings. Based on the structured compliance text rule base, a logical verification algorithm is used to verify violations of operational behaviors that trigger risk warnings, thereby determining the reasons for violations and implementing different handling measures based on different reasons for violations.

[0034] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method first deploys visible light cameras in each high-risk operation area using a parallel approach of fixed-point and mobile blind-spot cameras to acquire visible light images at a preset frame rate and simultaneously obtain safety specification data. Then, standard visible light images are obtained through adaptive noise reduction and standardization processing. Visible light behavior features are extracted using deep learning. Simultaneously, a multi-dimensional structured compliance text rule base is constructed based on safety specifications, and semantic features of the compliance text are extracted. These features are then dynamically weighted and fused using a cross-modal attention feature alignment algorithm to obtain a multimodal feature vector. Subsequently, the compliance semantics are transformed into computable judgment indicators. Tiered risk warnings are triggered through threshold comparison. Finally, the cause of the violation is determined based on the rule base and logical verification algorithms, and targeted measures are taken.

[0035] This method employs scenario-based deployment and adaptive noise reduction, effectively adapting to complex environmental interference in different high-risk work areas and ensuring the quality of image acquisition and preprocessing. It dynamically and weightedly fuses visible light behavioral features and compliant text semantic features across modalities, achieving deep alignment between visual perception and safety regulations, significantly improving the accuracy of behavior recognition and compliance judgment. Through semantic embedding, text rules are transformed into computable indicators, and combined with logical operations, accurate early warning and violation cause location are achieved, realizing full-process automation from monitoring to handling, effectively reducing false alarm rates and manual reliance, and improving the efficiency and precision of safety management in high-risk operations.

[0036] Specifically, visible light cameras are deployed in multiple high-risk work areas using a combination of fixed-point and mobile blind-spot coverage methods. The visible light acquisition frame rate of the visible light cameras is preset, and visible light images of different high-risk work areas are collected using the visible light cameras. These include: fixed-point deployment of visible light cameras in fuel oil areas, boiler maintenance areas, and high-voltage power distribution rooms; and mobile blind-spot coverage deployment of visible light cameras in hazardous chemical transfer channels and high-altitude work areas. The preset visible light acquisition frame rate of the visible light cameras is 25fps.

[0037] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method adopts a deployment method of fixed-point and mobile blind spot filling. Visible light cameras are deployed in fixed-point mode in fuel oil area, boiler maintenance area and high-voltage power distribution room, and visible light cameras are deployed in mobile blind spot filling mode in hazardous chemical transfer channel and high-altitude operation area. The acquisition frame rate of visible light cameras is uniformly preset to 25fps to complete the acquisition of visible light images in each high-risk operation area.

[0038] Based on the environmental characteristics and operational patterns of high-risk work areas, an installation strategy combining fixed-point installation and mobile patching is adopted: For areas with fixed work locations and constant risk points, such as fuel oil storage areas, boiler maintenance areas, and high-voltage power distribution rooms, fixed-point installation is adopted: visible light cameras are rigidly fixed to walls, equipment racks, or steel structure columns using brackets, clamps, or ceiling-mounted methods, at a height of 2.5m–4m. The lenses are pointed towards the core work area and personnel access routes to ensure comprehensive coverage without blind spots and long-term stability. For areas with dynamically changing work areas and potential monitoring blind spots, such as hazardous chemical transfer routes and high-altitude work areas, mobile blind spot installation is adopted: visible light cameras are mounted on mobile inspection vehicles, drones, lifting platforms, or movable supports, and their shooting positions are adjusted in real time as the work location moves to fill in blind spots in temporary work points, corners, and edge areas that cannot be covered by fixed cameras.

[0039] The frame rate of visible light cameras is uniformly preset to 25fps. The rationale and justification for this setting are as follows: It conforms to the standard of continuous perception in human vision. 25fps can completely record continuous behaviors such as personnel actions, operational procedures, and equipment wearing status, avoiding motion blur and missed detections due to excessively low frame rates. It balances recognition accuracy and system computing power. 25fps ensures the time resolution required for behavior recognition without causing excessive data volume, increased transmission pressure, or computational load due to excessively high frame rates, thus adapting to the real-time processing needs of on-site edge computing devices. Furthermore, 25fps is a common engineering parameter in the industry, facilitating unified configuration, synchronous calibration, and subsequent maintenance across multiple devices.

[0040] After the system is powered on and initialized, each visible light camera continuously acquires the video stream of the corresponding work area in real time at a fixed frame rate of 25fps, extracting visible light images frame by frame. During the acquisition process, automatic exposure and automatic white balance are kept on to adapt to changes in ambient light. The image data is transmitted to the back-end processing unit in real time via wired Ethernet or wireless LAN to complete caching and preprocessing, providing stable and continuous raw image input for subsequent noise reduction and feature extraction.

[0041] This deployment method balances long-term stable monitoring of fixed high-risk areas with flexible blind spot coverage of dynamic operation areas, effectively eliminating monitoring blind spots and improving monitoring integrity; a uniform 25fps acquisition frame rate can stably capture continuous operation behavior, simplifying system configuration and improving the convenience of project deployment and operation while ensuring the accuracy of behavior recognition.

[0042] Specifically, an adaptive denoising algorithm is used to denoise the visible light image to obtain a denoised visible light image. The denoised visible light image is then standardized to obtain a standard visible light image. This includes: using a Gaussian filtering algorithm to filter out flue gas interference and environmental noise from visible light images collected from the fuel oil area and boiler maintenance area to obtain a denoised visible light image; using a median filtering algorithm to filter out electromagnetic radiation clutter from visible light images collected from the high-voltage power distribution room to obtain a denoised visible light image; using a bilateral filtering algorithm to filter out weak light noise from visible light images collected from the hazardous chemical transfer channel to obtain a denoised visible light image; using a motion blur restoration algorithm to eliminate jitter interference from visible light images collected from the high-altitude operation area to obtain a denoised visible light image; and uniformly scaling the size of the denoised visible light image to 1024×768 pixels and normalizing the pixels of the denoised visible light image to the [0,1] interval to obtain a standard visible light image.

[0043] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method uses a scenario-based adaptive noise reduction algorithm to process visible light images based on the interference characteristics of different high-risk operation areas: Gaussian filtering is used to remove flue gas interference and environmental noise for images of fuel oil area and boiler maintenance area; median filtering is used to remove electromagnetic radiation clutter for images of high-voltage power distribution room; bilateral filtering is used to suppress weak light noise for images of hazardous chemical transfer channels; and motion blur restoration algorithm is used to eliminate jitter interference for images of high-altitude operation areas. After noise reduction, the images are uniformly scaled to 1024×768 pixels, and the pixel values ​​are normalized to the [0,1] interval to finally obtain a standard visible light image.

[0044] The system automatically matches and executes the corresponding noise reduction algorithm based on the job area type of the current image: The images of the fuel oil area and boiler maintenance area were denoised using a Gaussian filtering algorithm. The filter kernel size was set to 3×3, and the standard deviation σ was set to 0.8 to 1.2. The images were smoothed by Gaussian weighted average, which effectively filtered out high-frequency interference such as high-temperature flue gas, dust, and particulate noise, while preserving key behavioral features such as personnel contours and action edges, resulting in denoised visible light images.

[0045] The median filtering algorithm is used to process the images of high-voltage power distribution rooms. The filtering window adopts a 3×3 neighborhood. The gray values ​​in the neighborhood of each pixel are sorted by size and the median value is taken as the output pixel. This can effectively remove salt-and-pepper noise and pulse noise caused by electromagnetic radiation, avoid image spot and stripe distortion, and preserve the equipment outline and personnel boundary information to obtain a noise-reduced visible light image.

[0046] A bilateral filtering algorithm is used for images of hazardous chemical transfer channels under low light conditions. It takes into account both pixel spatial proximity and grayscale similarity, and preserves edge information while reducing noise. This effectively suppresses low light noise and low illumination noise, avoids edge blurring, and improves the clarity of personnel and equipment in low light, resulting in a noise-reduced visible light image.

[0047] For blurred images caused by wind vibration and scaffold swaying in high-altitude work areas, a motion blur restoration algorithm is adopted. First, the motion direction and displacement are estimated through spectrum analysis, and then inverse filtering or Wiener filtering is used to restore the image, eliminate motion blur and jitter blur, restore image details, and obtain a denoised visible light image.

[0048] After noise reduction, all images undergo uniform standardization to ensure consistent input specifications. All denoised visible light images were uniformly scaled to 1024×768 pixels using bilinear interpolation to ensure that the image content was not distorted and key features were not lost, thus meeting the fixed-size input requirements of deep learning models.

[0049] Image pixel grayscale values ​​are linearly mapped from 0 to 255 to the range [0, 1]. Normalization eliminates numerical biases caused by differences in lighting, distance, and device, thereby improving the stability and convergence speed of model training and feature extraction.

[0050] After the above processing, a standard visible light image with uniform size, standardized pixels, and no interference noise is finally obtained.

[0051] This method matches a dedicated noise reduction algorithm to the typical noise type of each work area, which is highly targeted and has a more thorough noise reduction effect. It can remove interference while preserving behavioral and equipment details to the greatest extent. The uniform size and pixel normalization processing can eliminate image differences caused by different scenes and different devices, ensure that the input specifications for subsequent feature extraction are consistent, and effectively improve the stability and accuracy of behavior recognition.

[0052] Specifically, a deep learning model is used to extract visible light behavior features from standard visible light images, including: using a deep learning model to extract global and local detail features from standard visible light images to obtain 256-dimensional visible light behavior features.

[0053] This application provides a method for multimodal behavior recognition and compliance judgment for high-risk operations. The method uses a deep learning model to jointly extract global features and local detail features from standard visible light images, and finally outputs visible light behavior features with a dimension of 256, which are used for subsequent multimodal feature fusion and compliance judgment.

[0054] A deep learning model is used to jointly extract global and local detail features from preprocessed standard visible light images, outputting a fixed-dimensional visible light behavior feature vector. This provides visual features of a unified specification for subsequent cross-modal fusion. The specific implementation method is as follows: A convolutional neural network (CNN) optimized for behavior recognition is used as the backbone model for feature extraction. Standard visible light images with a size of 1024×768 pixels and pixel values ​​normalized to the [0,1] interval are used as model input to ensure that the input format is completely matched with the model requirements.

[0055] The model uses the backbone convolutional and pooling layers to perform top-down global perception of the image, extracting global behavioral features such as the overall layout of the work area, personnel posture, work action type, and overall scene structure, thus representing the macroscopic semantics and overall behavioral patterns of the image.

[0056] By leveraging the model's high-resolution convolutional branches and attention-enhanced structures, key local details such as facial contours, limb joints, wearing status of protective equipment (helmets / safety belts), hand gestures, and equipment interaction details are extracted, thereby enhancing the ability to express features of minor violations.

[0057] The extracted global and local detail features are concatenated and normalized at the channel level and then compressed and mapped through the fully connected layers of the model. The final output is a visible light behavior feature vector with a fixed dimension of 256. This vector compactly represents all visual information related to the task behavior in the image in numerical form and can be directly used for subsequent cross-modal attention-weighted fusion.

[0058] This method captures both global behavioral information and local detailed features simultaneously through deep learning, enabling it to fully and accurately express key information such as personnel work behavior, protective equipment wearing, and operational actions. The 256-dimensional feature dimension is moderate, ensuring sufficient feature representation while also considering computational efficiency, which is beneficial for improving the speed and accuracy of subsequent fusion, matching, and recognition.

[0059] Specifically, a multi-dimensional structured compliance text rule base is constructed based on safety specification data, including: a six-dimensional structured compliance text rule base is constructed based on safety specification data, where the six dimensions include work qualifications, operating procedures, protective measures, environmental requirements, equipment status and emergency preparedness.

[0060] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. Based on the collected safety specification data, this method constructs a structured compliance text rule base for high-risk operations from six dimensions: operation qualification, operation specifications, protective measures, environmental requirements, equipment status, and emergency preparedness.

[0061] Based on the collected high-risk operation safety standard data, the data is structured, classified, and stored to construct a six-dimensional structured compliance text rule base covering all elements and processes of the operation. The specific implementation method is as follows: First, we collect and summarize safety specification data applicable to high-risk operations, including national and industry safety standards, enterprise on-site operation management systems, special operation permit requirements, risk control lists, etc. We then deduplicate, correct, segment, and sentence-by-sentence the textual specifications to form a basic rule text with a unified format.

[0062] The basic rules text is structured and labeled according to six dimensions: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness. Each dimension is an independent library with a clear hierarchy: Work Qualifications: Includes rules on personnel certification, work authority, training qualification, and approval processes; Operating Procedures: Includes standard operating procedures and prohibited behaviors for hot work, working at heights, confined spaces, and equipment start-up and shutdown; Protective Measures: Includes requirements for the wearing and use of safety helmets, safety belts, protective clothing, and protective equipment; Environmental Requirements: Includes environmental conditions such as temperature, lighting, ventilation, and unobstructed passageways in the work area; Equipment Status: Includes clauses on normal equipment operation, no abnormalities, reliable grounding, and intact protective devices; Emergency Preparedness: Includes clauses on emergency equipment allocation, emergency exits, on-site monitoring, and emergency response procedures.

[0063] The rules for each dimension are stored in a structured field format. Each rule includes attributes such as rule ID, applicable area, constraint type, judgment threshold, risk level, and related clauses. Keyword and behavior type indexes are also established to facilitate rapid subsequent retrieval, semantic extraction, and logical verification. The system supports dynamic expansion of the rule base, allowing for the addition, modification, and deletion of content across the six dimensions based on updates to on-site specifications, additions to operational scenarios, and upgrades to control requirements, ensuring the rule base remains valid in real time.

[0064] This method establishes a six-dimensional rule system from the perspective of the entire process and all elements, covering all key points of safety control such as personnel, behavior, environment, equipment, and emergency response. The rules are complete, clearly structured, and highly searchable, providing a comprehensive, standardized, and unified basis for subsequent compliance judgments, violation verification, and early warnings, avoiding rule omissions and judgment biases.

[0065] Specifically, a cross-modal attention feature alignment algorithm is used to dynamically weight and fuse visible light behavior features and compliance text semantic features to obtain a multimodal feature vector. This includes: calculating dynamic weights for different high-risk work areas using the softmax function, where the dynamic weighting weight for visible light behavior features in the fuel oil area and boiler maintenance area is 0.6, and the dynamic weighting weight for compliance text semantic features in the fuel oil area and boiler maintenance area is 0.4; the dynamic weighting weight for visible light behavior features in the high-voltage power distribution room is 0.5, and the dynamic weighting weight for compliance text semantic features in the high-voltage power distribution room is 0.5; the dynamic weighting weight for visible light behavior features in the high-altitude work area and hazardous chemical transfer channel is 0.7, and the dynamic weighting weight for compliance text semantic features in the high-altitude work area and hazardous chemical transfer channel is 0.3; and then concatenating and fusing the weighted visible light behavior features and compliance text semantic features to obtain a 512-dimensional multimodal feature vector.

[0066] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method uses a cross-modal attention feature alignment algorithm to dynamically weight and fuse visible light behavior features and compliance text semantic features. The softmax function is used to calculate and assign weights to different high-risk operation areas: visible light behavior features in fuel oil areas and boiler maintenance areas have a weight of 0.6, and compliance text semantic features have a weight of 0.4; the two types of features in high-voltage power distribution rooms each have a weight of 0.5; visible light behavior features in high-altitude operation areas and hazardous chemical transfer channels have a weight of 0.7, and compliance text semantic features have a weight of 0.3. The weighted two types of features are concatenated and fused to obtain a 512-dimensional multimodal feature vector.

[0067] From the six-dimensional structured compliance text rule base, all valid rule texts corresponding to the current work area are read, including clauses covering six dimensions: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness. The rule texts are processed by sentence segmentation, removal of stop words, removal of special symbols, and standardization of punctuation, dividing long rules into independent semantic units to ensure that each rule is semantically complete and unambiguous.

[0068] A semantic embedding algorithm is used to map text rules into numerical feature vectors: the preprocessed rule text is input into a pre-trained semantic encoding model, and the text constraints are transformed into fixed-dimensional semantic features through three steps: word vector encoding, context semantic encoding, and pooling output; the semantic vectors of all rules in the same work area are fused by mean fusion to obtain 256-dimensional compliant text semantic features, which are consistent with the dimensions of visible light behavior features.

[0069] L2 normalization is performed on the semantic features of compliant texts to make the feature vector magnitude 1, and the numerical range is constrained within a reasonable range to ensure that it is aligned with the visible light behavior features in terms of numerical magnitude, thus avoiding numerical imbalance during weighted fusion.

[0070] Based on the current image source, the system automatically determines the category of the work area: fuel oil area / boiler maintenance area, high-voltage power distribution room, high-altitude work area / hazardous chemical transfer channel.

[0071] Using the work area as a condition, initial weight scores are set, and the weight scores are mapped to dynamic weighting coefficients with a sum of 1 using the softmax function: fuel oil area and boiler maintenance area: visible light behavior feature weight 0.6, compliance text semantic feature weight 0.4; high-voltage power distribution room: visible light behavior feature weight 0.5, compliance text semantic feature weight 0.5; high-altitude work area and hazardous chemical transfer channel: visible light behavior feature weight 0.7, compliance text semantic feature weight 0.3.

[0072] Multiply each element of the 256-dimensional visible light behavior features by the corresponding region weight; multiply each element of the 256-dimensional compliant text semantic features by the corresponding region weight; thus obtaining two sets of weighted features: weighted visible light behavior features and weighted compliant text semantic features.

[0073] In terms of feature dimensions, the weighted 256-dimensional visible light behavior features and the 256-dimensional compliant text semantic features are concatenated end to end to form a 512-dimensional concatenated feature.

[0074] Batch normalization is performed on the concatenated 512-dimensional features to eliminate the distribution offset caused by feature concatenation and improve the stability of subsequent matching. Finally, a fixed-dimensional 512-dimensional multimodal feature vector is output. This vector carries both visible light behavior information and safety specification semantic information, which can be directly used for threshold comparison and compliance judgment.

[0075] This method assigns differentiated feature weights according to the risk characteristics and monitoring priorities of different work areas, achieving scene-adaptive feature alignment and making visual behavior more accurately matched with safety regulations. The 512-dimensional feature vector formed by the fusion of these features combines behavioral details with regulatory semantics, resulting in stronger expressive power and significantly improving the accuracy and robustness of subsequent compliance judgments and early warning identification.

[0076] Specifically, the semantic embedding algorithm is used to transform the compliance semantics in the structured compliance text rule base into computable judgment indicators. This includes: decomposing the structured compliance text rule base into sentences and segments to extract the core semantic units of each rule; mapping the core semantic units to semantic feature vectors using the semantic embedding algorithm, and extracting the numerical and Boolean constraints from the structured compliance text rule base; mapping the semantic feature vectors and Boolean constraints to behavior category judgment indicators, and mapping the numerical constraints to numerical judgment indicators; and unifying the format of the behavior category judgment indicators and the numerical judgment indicators to obtain computable judgment indicators.

[0077] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method uses a semantic embedding algorithm to transform a structured compliance text rule base into a computable judgment index. First, the rule base is decomposed into sentences and segments in a structured manner, and the core semantic units of each rule are extracted. Then, the core semantic units are mapped into semantic feature vectors through a semantic embedding algorithm. At the same time, numerical constraints and Boolean constraints in the rule base are extracted. Subsequently, the semantic feature vectors and Boolean constraints are mapped into behavior category judgment indexes, and the numerical constraints are mapped into numerical judgment indexes. Finally, the formats of the two types of judgment indexes are unified to obtain computable judgment indexes that can be directly used for calculation and comparison.

[0078] From the six-dimensional structured compliance text rule base, all rule clauses related to work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness are read one by one. Rules are segmented into sentences and paragraphs using periods, semicolons, and line breaks, breaking long rules down into independent, complete, and single-constraint short sentences, each expressing only one prohibition, requirement, or condition. Spaces, special symbols, redundant modifiers, and invalid punctuation are removed, retaining only the most core actions, objects, conditions, prohibited items, and permitted items.

[0079] Extract core semantic units with a fixed structure from each rule, including: Subject: Operator / Equipment / Environment; Behavior: Smoking, Hot work, Entry, Wearing, Operation, etc.; Constraints: Must / Prohibited / Required / Not Allowed; Conditions: Permit or not, Certificate or not, or in a designated area; Status: Normal / Abnormal / Intact / Missing. This results in a set of pure semantic, non-redundant, and machine-understandable core semantic units.

[0080] The core semantic units are encoded using a semantic embedding model: the text is input into the semantic embedding layer, and each core semantic unit is mapped into a fixed-dimensional semantic feature vector through word vector mapping, context encoding, and pooling output, thereby realizing the digital representation of text semantics.

[0081] Automatically identify and extract quantifiable values ​​from the rules, such as temperature range, height limit, time threshold, allowable / prohibited intervals, and equipment parameter ranges, and save them as numerical constraints.

[0082] Extract yes / no judgment constraints from the rules, such as: whether there is qualification, whether there is permission, whether equipment is worn, whether entry is allowed, whether it is compliant, and save them as Boolean constraint items (yes / no, true / false).

[0083] Semantic feature vectors are bound to Boolean constraints and mapped to machine-recognizable behavior category labels, such as: illegal smoking, illegal open flame, unauthorized entry, not wearing a safety helmet, not wearing a seat belt, and illegal operation. Each behavior category indicator includes: indicator ID, work area, risk type, triggering condition, and risk level.

[0084] Numerical constraints are directly converted into comparable thresholds / intervals, such as: temperature > 40℃, interval [0, 35], ≥ 600℃, 300–500℃, etc., forming numerical judgment indicators that can be directly used for numerical comparison.

[0085] The two types of indicators are unified into the same structure, which includes: indicator number; applicable work area; indicator type (behavior category / value); judgment rule; corresponding risk type; warning level.

[0086] Establish a one-to-one correspondence between multimodal feature vectors and computable judgment indicators, so that behavioral features, personnel features, and environmental features can be directly matched with indicators.

[0087] All behavioral category judgment indicators and numerical judgment indicators are integrated into the database to form a complete, unified, and directly applicable calculable judgment indicator system for threshold comparison.

[0088] This method transforms security specifications in natural language form into machine-recognizable and computable quantitative indicators, completely breaking down the semantic barriers between text rules and visual features, and achieving automation and standardization of compliance judgment. Rule decomposition and vector mapping can fully preserve the core constraint meaning of security specifications, taking into account both behavior category judgment and numerical threshold judgment, effectively improving the accuracy, consistency and interpretability of subsequent compliance comparison.

[0089] Specifically, a threshold comparison method is used to compare multimodal feature vectors with calculable judgment indicators to trigger different levels of risk warnings. This includes: extracting key sub-features from the multimodal feature vectors, including behavioral recognition features and personnel presence features; comparing the key sub-features with calculable judgment indicators; triggering a smoking violation warning when the behavioral recognition feature is smoking; triggering a fire violation warning when the behavioral recognition feature is an obvious open flame without a hot work permit; triggering a trespass warning when the personnel presence feature is unqualified entry; triggering a protection deficiency warning when the behavioral recognition feature is not wearing a safety belt or helmet at height; and triggering a violation operation warning when the behavioral recognition feature is not operating equipment according to standard procedures.

[0090] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. This method uses a threshold comparison method to trigger risk warnings. First, it extracts two key sub-features from the multimodal feature vector: behavior recognition features and personnel presence features. Then, it compares them item by item with calculable judgment indicators. When smoking is detected, a smoking violation warning is triggered. When an obvious open flame is detected without a hot work permit, a fire violation warning is triggered. When unqualified personnel enter the premises, an unprotected entry warning is triggered. When working at height without a safety belt or helmet, a safety deficiency warning is triggered. When equipment is not operated according to the standard procedure, a violation operation warning is triggered. This achieves graded warnings for different types of risks.

[0091] From the generated 512-dimensional multimodal feature vector, two types of key sub-features are extracted according to preset dimensions: behavioral recognition features: used to characterize visual behavioral information such as personnel actions, operating status, wearing of protective equipment, open flame / smoke, etc.; personnel presence features: used to characterize personnel status information such as whether personnel are present, whether they have the qualifications to work, and whether they are in the authorized area.

[0092] The two types of key sub-features are dimensionally aligned and normalized to ensure that their numerical distribution, feature length and computable judgment index are completely matched, thus ensuring consistency in subsequent comparisons.

[0093] Load the standardized, computable judgment indicators from the system, including behavioral category judgment indicators and numerical judgment indicators. Bind the warning rules to the computable judgment indicators one by one to form an executable comparison logic library. Each rule includes: triggering feature, judgment condition, warning type, and risk level.

[0094] The similarity between the behavioral recognition features and the "smoking action" behavior category judgment index is compared; when the feature matching degree exceeds the preset threshold, it is judged as illegal smoking.

[0095] Simultaneously compare two conditions: behavioral recognition features match the "obvious open flame" indicator; personnel presence features / permit features match the "no hot work permit" Boolean constraint; when both conditions are met, it is determined to be an illegal open flame.

[0096] The system compares the characteristics of personnel with the "qualification access" indicators; when personnel are detected but their qualification characteristics do not meet the constraints, it is determined to be an unauthorized entry.

[0097] The behavioral recognition features are compared with the "wearing of protective equipment" indicator; when no safety belt or safety helmet features are detected in high-altitude operation scenarios, it is determined that protection is lacking.

[0098] The behavioral recognition features are compared with the standard operating procedure indicators in terms of time sequence and action; when the action sequence, operation object, execution method are inconsistent with the standard indicators, it is judged as a violation of the operation.

[0099] When any set of comparisons meets the indicator constraints, the system immediately triggers an early warning: Behavioral recognition feature: smoking action → triggering a warning for illegal smoking; Behavioral characteristics include obvious open flame and no hot work permit → triggering an illegal open flame warning; The presence of unauthorized personnel triggers an unauthorized entry warning. Behavioral recognition feature: not wearing a safety belt or helmet at height → triggers a warning of lack of protection; The behavioral identification feature is that the equipment is not operated in accordance with the standard procedure → triggering a violation operation warning.

[0100] Once the warning is triggered, the system automatically records: the work area, timestamp, current frame image, key sub-feature values, and matching indicator ID; pushes the warning information to the backend platform, on-site audible and visual alarm devices, and mobile monitoring terminals; locks the current screen and saves evidence to provide a basis for subsequent violation verification.

[0101] This method is based on comparison and triggering of clear and quantifiable features and indicators. The early warning logic is clear and the judgment criteria are unified, which can effectively reduce false alarms and missed alarms. It covers typical hidden danger types of high-risk operations such as illegal smoking, illegal open flames, illegal entry, lack of protection, and illegal operation. The early warning scenarios are comprehensive, and it can realize real-time monitoring of personnel behavior and operation status in all dimensions, providing timely and reliable alarm support for on-site safety management.

[0102] Specifically, based on a structured compliance text rule base, a logical verification algorithm is used to verify the violations of operational behaviors that trigger risk warnings in order to obtain the reasons for the violations. This includes: retrieving all clauses related to risk warnings from the structured compliance text rule base to obtain the rule set to be verified; and using AND / OR / NOT / implication logical operations on the operational behaviors that trigger risk warnings based on the rule set to be verified in order to obtain the reasons for the violations.

[0103] This application provides a multimodal behavior recognition and compliance judgment method for high-risk operations. The method is based on a structured compliance text rule base to verify the reasons for violations. First, all clauses related to the current risk warning are called from the rule base to form a set of rules to be verified. Then, AND, OR, NOT, and implied logical operations are used to verify the compliance of the operation behavior that triggers the warning, and finally the specific reasons for the violation of the operation behavior are accurately determined.

[0104] The system reads the triggered risk warning categories, including: illegal smoking warning, illegal open flame warning, illegal trespassing warning, lack of protective measures warning, and illegal operation warning. Based on the warning type, it retrieves and filters all rule clauses directly, indirectly, and constrainedly related to the warning from six dimensions: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness, based on the structured compliance text rule base. All matched rule clauses are then extracted, deduplicated, and organized to form a set of rules to be verified. Each rule includes: rule number, constraint content, judgment conditions, prohibition / requirement attributes, and associated risk points.

[0105] The system performs compliance checks on each work action that triggers the warning, according to logical operation rules, as follows: The "AND" logical operator is used when a violation requires multiple constraints to be met simultaneously. For example: violation of open flame = presence of open flame and lack of hot work permit and non-hot work area. The violation is considered valid only if all conditions are true.

[0106] The "OR" logical operator is used when a violation is true if any one of several conditions is met. For example: lack of protection = not wearing a helmet or not wearing a seatbelt. If any one of these conditions is true, the violation is deemed valid.

[0107] The "NOT" logical operator is used to negate the validity of permissions, qualifications, and status. For example: unauthorized entry = unauthorized or unqualified personnel entering. If the negation of the condition is true, the violation is deemed valid.

[0108] The "implication" logical operation is used to determine whether an action conforms to a rule constraint, that is, "if a certain action occurs, then a certain condition must be met." For example: working at heights → safety belts must be worn; if the action is valid but the condition is not, then the implication is invalid, and a violation is determined.

[0109] The system uses the behavioral characteristics, personnel presence characteristics, scene information, and status information of the current work activity as input for logical judgment. It then compares each input with every rule in the set of rules to be verified. All rules that do not meet constraints, violate prohibitions, fail to meet requirements, or are invalid are marked. Conflicting rules are merged, deduplicated, and prioritized, ultimately outputting a unique or combined reason for violation.

[0110] After logical verification, the system outputs standardized reasons for violations, such as: Violation warning for smoking in the fuel area → Reason for violation: Violation of the no-smoking clause in the fuel area; Violation warning for open flame → Reason for violation: Use of open flame without a hot work permit, violating hot work approval regulations; Violation warning for unauthorized entry → Reason for violation: Unqualified personnel entering the high-voltage power distribution room, violating area access management regulations; Violation warning for lack of protection → Reason for violation: Failure to wear a safety belt / helmet while working at height, violating protection requirements; Violation warning for improper operation → Reason for violation: Failure to operate equipment according to standard procedures, violating operating specifications.

[0111] The reasons for violations are linked to early warning information, rule-based information, and image evidence and archived. This information is then pushed to the violation verification and processing module for tiered handling, voice alerts, on-site alarms, and record archiving. This forms a complete closed loop of early warning → verification → cause determination → handling.

[0112] This method automatically locates the cause of violations through association rule retrieval and logical operations. The warning results have complete evidence and interpretability, avoiding the problem of simply issuing warnings without being able to determine the root cause. The logical verification is rigorous and standardized, which can accurately distinguish between single violations and compound violations, greatly improving the credibility of compliance judgments and the targeted nature of handling.

[0113] Example 3 like Figure 2 As shown, based on the same inventive concept as the above embodiments, the present invention also provides a multimodal behavior recognition and compliance judgment system for high-risk operations, including: The preprocessing unit is used to acquire visible light images and safety specification data of high-risk work areas, preprocess the visible light images to obtain standard visible light images, and construct a multi-dimensional structured compliance text rule base based on the safety specification data. An extraction and fusion unit is used to extract visible light behavior features from standard visible light images and extract compliant text semantic features from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The judgment unit is used to convert the semantic features of compliant text into computable judgment indicators using semantic embedding, compare the multimodal feature vector with the computable judgment indicators, and trigger different levels of risk warnings based on the comparison results. The verification unit is used to perform violation verification on the multimodal feature vectors that trigger risk warnings based on a multidimensional structured compliance text rule base, and to obtain the reasons for the violation.

[0114] Example 4 This embodiment provides a multimodal behavior recognition and compliance judgment system for high-risk operations. The judgment system includes: The preprocessing unit is used to acquire visible light images and safety specification data of high-risk work areas, and to perform noise reduction processing on the visible light images using an adaptive noise reduction algorithm to obtain denoised visible light images. The denoised visible light images are then standardized to obtain standard visible light images. A multi-dimensional structured compliance text rule base is constructed based on the safety specification data. The extraction and fusion unit is connected to the preprocessing unit. The extraction and fusion unit is used to extract visible light behavior features from standard visible light images using a deep learning model, extract compliant text semantic features from a structured compliant text rule base using a semantic embedding algorithm, and dynamically weight and fuse the visible light behavior features and compliant text semantic features using a cross-modal attention feature alignment algorithm to obtain a multimodal feature vector. The judgment unit, connected to the extraction and fusion unit, is used to convert compliance semantics in the structured compliance text rule base into computable judgment indicators using semantic embedding algorithms, and to compare multimodal feature vectors with computable judgment indicators using threshold comparison methods to trigger risk warnings at different levels. The verification unit, connected to the judgment unit, is used to perform violation verification on operational behaviors that trigger risk warnings based on a structured compliance text rule base and using logical verification algorithms to obtain the reasons for the violation and take different handling measures according to different reasons for the violation.

[0115] This application provides a multimodal behavior recognition and compliance judgment system for high-risk operations. The system is mainly composed of a preprocessing unit, an extraction and fusion unit, a judgment unit, and a verification unit connected in sequence. The preprocessing unit performs adaptive noise reduction and standardization processing. The extraction and fusion unit extracts visible light behavior features and compliance text semantic features and performs cross-modal weighted fusion. The judgment unit converts rule semantics into computable indicators and triggers graded risk warnings through threshold comparison. The verification unit locates the cause of violations based on logical verification and executes differentiated handling measures.

[0116] The system adopts a modular serial architecture, with each module having a clear division of labor and smooth collaboration, enabling end-to-end automated operation from data collection to violation handling. It covers the entire process of monitoring and control of high-risk operations, and combines scenario-based deployment, adaptive noise reduction, cross-modal feature fusion and logical violation verification to significantly improve the robustness of behavior recognition, the accuracy of compliance judgment and the efficiency of early warning and handling, effectively reducing reliance on manual inspection and safety management risks.

[0117] Example 5 like Figure 3 As shown, the present invention also provides an electronic device 100 for implementing a multimodal behavior recognition and compliance judgment method for high-risk operations; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0118] The memory 101 can be used to store computer program 103. The processor 102 implements the steps of a multimodal behavior recognition and compliance judgment method for high-risk operations by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0119] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0120] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0121] The memory 101 in the electronic device 100 stores multiple instructions to implement a multimodal behavior recognition and compliance judgment method for high-risk operations, and the processor 102 can execute multiple instructions to achieve the following: Acquire visible light images and safety specification data of high-risk work areas; preprocess the visible light images to obtain standard visible light images; and construct a multi-dimensional structured compliance text rule base based on the safety specification data. Visible light behavior features are extracted from standard visible light images, and compliant text semantic features are extracted from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The semantic embedding method is used to transform the semantic features of compliant texts into computable judgment indicators. The multimodal feature vectors are compared with the computable judgment indicators, and different levels of risk warnings are triggered based on the comparison results. Based on a multi-dimensional structured compliance text rule base, the violation verification is performed on the multi-modal feature vectors that trigger risk warnings to obtain the reasons for the violation.

[0122] Example 6 If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for multimodal behavior recognition and compliance assessment for high-risk operations, characterized in that, include: Acquire visible light images and safety specification data of high-risk work areas, and preprocess the visible light images to obtain standard visible light images; A multi-dimensional structured compliance text rule base is built based on security specification data; Visible light behavior features are extracted from standard visible light images, and compliant text semantic features are extracted from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The semantic embedding method is used to transform the semantic features of compliant texts into computable judgment indicators. The multimodal feature vectors are compared with the computable judgment indicators, and different levels of risk warnings are triggered based on the comparison results. Based on a multi-dimensional structured compliance text rule base, the violation verification is performed on the multi-modal feature vectors that trigger risk warnings to obtain the reasons for the violation.

2. The method for multimodal behavior recognition and compliance judgment for high-risk operations according to claim 1, characterized in that, Preprocessing the visible light image yields a standard visible light image, including: An adaptive denoising method is used to denoise the visible light image to obtain a denoised visible light image; The denoised visible light image is standardized to obtain a standard visible light image.

3. The method for multimodal behavior recognition and compliance judgment for high-risk operations according to claim 2, characterized in that, The visible light images were acquired from different high-risk work areas, and noise reduction was performed using either filtering or restoration methods. The visible light images collected from the fuel oil storage area and boiler maintenance area were filtered out using Gaussian filtering to remove flue gas interference and environmental noise. For visible light images acquired in high-voltage power distribution rooms, median filtering is used to remove electromagnetic radiation clutter. For visible light images acquired at hazardous chemical transfer channels, a bilateral filtering method was used to remove weak light noise. For visible light images acquired in high-altitude work areas, motion blur restoration method is used to eliminate jitter interference.

4. The method for multimodal behavior recognition and compliance judgment for high-risk operations according to claim 1, characterized in that, Extract visible light behavior features from standard visible light images and extract semantic features of compliant text from a multi-dimensional structured compliant text rule base; A cross-modal attention feature alignment method is used to dynamically weight and fuse visible light behavior features and compliant text semantic features to obtain a multimodal feature vector, specifically including: The global and local detail features of a standard visible light image are jointly extracted to output visible light behavior features. The text in the multi-dimensional structured compliance text rule base is segmented, stop words are removed, and the format is standardized to obtain a unified text. The unified text is then processed by word vector encoding, context semantic encoding, and pooling to obtain the semantic feature vector of each rule. The semantic vectors of all rules in the same high-risk operation area are then fused by mean and normalized by L2 to output the semantic features of the compliance text. The dynamic weighting weights of different high-risk work areas are calculated using the softmax function. The weighted visible light behavior features are then fused with the semantic features of compliant text to obtain a multimodal feature vector.

5. The method for multimodal behavior recognition and compliance judgment for high-risk operations according to claim 1, characterized in that, Semantic embedding is used to transform the semantic features of compliant texts into computable judgment indicators. Multimodal feature vectors are compared with these computable judgment indicators, and different levels of pre-set risk warnings are triggered based on the comparison results. Specifically, these include: The structured compliance text rule base is decomposed into sentences and segments to extract the core semantic units of each rule; the core semantic units are mapped into semantic feature vectors using semantic embedding, and numerical and Boolean constraints in the rule base are extracted. Semantic feature vectors are mapped to Boolean constraints to behavior category determination indicators, and numerical constraints are mapped to numerical determination indicators. The behavior category determination indicators and numerical determination indicators are standardized in format to obtain computable determination indicators. Key sub-features are extracted from the multimodal feature vector, including: behavior recognition features and personnel presence features; these key sub-features are then compared with computable judgment indicators. When the behavior recognition feature is a smoking action, a violation smoking warning is triggered; When the behavior identification feature is that there is obvious open flame and no hot work permit, an illegal open flame warning is triggered; When the personnel are found to be unqualified to enter, an unauthorized entry warning is triggered. When the behavioral recognition feature is that the person is not wearing a safety belt or helmet at high altitude, a protection deficiency warning is triggered. When the behavior identification feature indicates that the device is not operated according to the standard procedure, a violation operation warning is triggered.

6. The method for multimodal behavior recognition and compliance judgment for high-risk operations according to claim 1, characterized in that, The multi-dimensional structured compliance text rule base includes six dimensions: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness.

7. The method for multimodal behavior recognition and compliance judgment for high-risk operations according to claim 6, characterized in that, Based on a multi-dimensional structured compliance text rule base, violation verification is performed on the multi-modal feature vectors that trigger risk warnings to obtain the reasons for the violations, specifically including: Based on the triggered risk warnings, all rule clauses that are directly, indirectly, or constrained by the risk warnings are retrieved and screened from six dimensions: work qualifications, operating procedures, protective measures, environmental requirements, equipment status, and emergency preparedness. After deduplication and sorting, a set of rules to be verified is formed. Based on the set of rules to be verified, logical operations are used to verify violations of the multimodal feature vectors that trigger risk warnings, in order to obtain the reasons for the violations. The multimodal feature vectors include behavioral recognition features, personnel presence features, scene information, and state information. The logical operations include: AND, OR, NOT, and implication operations. Specifically: The behavioral recognition features, personnel presence features, scene information, and status information in the multimodal feature vector that triggers the risk warning are used as logical judgment inputs. Logical operations are performed on the input and each rule in the set of rules to be verified in turn, and all rule items that do not meet the constraints, violate the prohibition clauses, fail to meet the requirements, or are not valid are marked. Merge, deduplicate, and prioritize conflicting rules, and output a unique or combined reason for violation.

8. A multimodal behavior recognition and compliance judgment system for high-risk operations, characterized in that, include: The preprocessing unit is used to acquire visible light images and safety specification data of high-risk work areas, and to preprocess the visible light images to obtain standard visible light images. A multi-dimensional structured compliance text rule base is built based on security specification data; An extraction and fusion unit is used to extract visible light behavior features from standard visible light images and extract compliant text semantic features from a multi-dimensional structured compliant text rule base. A cross-modal attention feature alignment method is used to dynamically weight and fuse the visible light behavior features and compliant text semantic features to obtain a multi-modal feature vector. The judgment unit is used to convert the semantic features of compliant text into computable judgment indicators using semantic embedding, compare the multimodal feature vector with the computable judgment indicators, and trigger different levels of risk warnings based on the comparison results. The verification unit is used to perform violation verification on the multimodal feature vectors that trigger risk warnings based on a multidimensional structured compliance text rule base, and to obtain the reasons for the violation.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement a multimodal behavior recognition and compliance judgment method for high-risk operations as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements a multimodal behavior recognition and compliance judgment method for high-risk operations as described in any one of claims 1 to 7.

Images