A method, system, device and medium for monitoring compliance of team safety operation

Abstract

The application provides a kind of team safety operation compliance monitoring method, system, equipment and medium, method includes the following steps: constructing the operation process atlas composed of action node and the directed edge encapsulated with logic constraint rule;Response operation trigger instruction, utilize intelligent vision terminal to establish the space corresponding relationship of operation site and atlas;Obtain first perspective vision and motion sensing data and carry out space-time reference alignment, generate multi-modal fusion feature vector;Actual action feature is obtained by using pre-training model inference;According to atlas logic, determine target action node, calculate action deviation and path logic compliance degree;Based on risk rating, trigger the hierarchical closed-loop control strategy including information guidance, sensory warning or equipment intervention.The technical solution provided by the application is implemented, the real-time compliance verification of operation process is realized, the risk identification coverage and feedback response timeliness are improved, and the safety management and control process of team operation is optimized.

Description

Technical Field

[0001] This application relates to the field of industrial safety management technology, and in particular to a method, system, equipment and medium for monitoring the compliance of safe operations in work teams. Background Technology

[0002] With the rapid development of modern industrial manufacturing towards intelligence and precision, the operational processes in complex scenarios such as automobile assembly, chemical production, and construction are becoming increasingly intricate, posing extremely high challenges to the real-time and accurate nature of on-site safety management. Transforming static safety regulations into dynamic operational guidelines, and conducting comprehensive real-time monitoring and closed-loop intervention of workers' physical movements, tool usage sequences, and environmental hazards, has become a core requirement for reducing production accident rates and ensuring the compliance of work team operations.

[0003] In existing technologies, safety monitoring at work sites typically relies on manual inspections, static document reviews, or alarm systems based on a single rule engine. For example, fixed cameras are used to identify obvious violations such as not wearing a safety helmet, or alarms are triggered when pressure or temperature values ​​collected by sensors exceed preset thresholds. However, existing technologies lack the ability to fuse and analyze multimodal data in practical applications, and independently deployed monitoring systems are often disconnected from standard operating procedures. This results in a weak ability to identify hidden risks such as incorrect operation sequences or substandard physical movements. When faced with nonlinear work processes or complex dynamic environments, there are risks of delayed response, false alarms and missed alarms, and difficulty in preventing dangerous operations in real time. Summary of the Invention

[0004] In view of this, this application provides a method, system, equipment and medium for monitoring the compliance of safe operations in work groups to solve the above problems.

[0005] Firstly, a method for monitoring the compliance of safe operations in work teams is provided, the method comprising:

[0006] Construct a job process graph, which consists of action nodes encapsulating standard action features and standard job space information, and directed edges encapsulating logical constraint rules between steps.

[0007] In response to the operation trigger command, the system uses a pre-set intelligent vision terminal to scan the operation site and construct a spatial feature model. The spatial feature model is then registered and mapped with the standard operation spatial information in the operation process map to establish a spatial correspondence between the operation site and the operation process map.

[0008] Acquire first-person visual data and motion sensing data of the operator during the operation, align the first-person visual data and motion sensing data with a spatiotemporal reference, and use a preset multimodal fusion algorithm to generate a fusion feature vector representing the current operation status.

[0009] The fused feature vector is input into a pre-trained action recognition model for inference operations to obtain the actual action features that represent the current actual work behavior.

[0010] Based on the logical constraint rules between steps in the work process diagram, the target action node to be evaluated is determined, the actual action characteristics are matched with the standard action characteristics in the target action node, and the action deviation and path logic conformity are calculated.

[0011] A comprehensive risk rating is generated based on the deviation of actions and the compliance of path logic. According to the preset level of the comprehensive risk rating, the corresponding closed-loop control strategy is triggered to adjust the operator's operation behavior or equipment status. The closed-loop control strategy includes information guidance strategy, sensory warning strategy or equipment intervention strategy.

[0012] The above technical solution constructs standard operating procedures into a work process graph containing action nodes and logical constraints, and registers and maps the on-site 3D spatial model with the standard spatial information in the graph to achieve spatial correspondence between the work environment and standard operations. Then, by collecting first-person perspective videos and motion sensor data from workers and performing multimodal fusion and action recognition, the actual action characteristics are obtained and the action deviation and path logic compliance are calculated, thereby achieving real-time risk quantification of work behavior. By automatically triggering closed-loop control strategies at different levels, the real-time monitoring and intervention capabilities for work safety are effectively improved.

[0013] Optionally, construct a job process graph, specifically including:

[0014] The layout of the pre-set standard work instructions is analyzed using a document parsing algorithm to separate the task description text block and the task illustration image block;

[0015] Semantic entity extraction is performed on the task description text block to identify verb entities representing task actions, noun entities representing operation objects, and time adverbial entities representing task order.

[0016] The initial action nodes of the job process graph are generated based on verb entities, and the temporal logic of action execution is parsed based on time adverbial entities. The temporal logic is then used to construct directed edges connecting each initial action node.

[0017] The pre-defined convolutional neural network is used to encode the features of the image blocks in the task diagram, generate standard visual feature vectors, and associate and encapsulate the standard visual feature vectors with the corresponding verb entities and noun entities to form action nodes;

[0018] A topological structure is constructed based on action nodes and directed edges to obtain the job process graph.

[0019] The above technical solution utilizes document parsing algorithms to separate text and images, extract semantic entities, and recognize temporal logic in standard operating procedures. It also combines convolutional neural networks to extract visual features from schematic images, automatically transforming the operating specifications into a structured graph containing action features, visual features, and logical relationships. This enables the automated generation of operating specifications from unstructured text into a machine-recognizable model, facilitating subsequent action recognition and compliance comparison.

[0020] Optionally, a pre-set intelligent vision terminal is used to scan the work site and construct a spatial feature model. The spatial feature model is then registered and mapped with the standard work spatial information in the work process map. Specifically, this includes:

[0021] The intelligent vision terminal is driven to emit infrared structured light and receive reflected signals. Based on the reflected signals, the depth point cloud data of the work site is calculated to obtain a spatial feature model.

[0022] Identify the preset reference markers in the spatial feature model, and establish a local three-dimensional coordinate system with the preset reference markers as the origin;

[0023] The point cloud features of the spatial feature model in the local 3D coordinate system are geometrically matched with the standard operation spatial information. The rotation matrix and translation vector of the local 3D coordinate system relative to the global coordinate system pre-stored in the standard operation spatial information are calculated using the iterative nearest point algorithm to complete the registration mapping.

[0024] The above technical solution uses an intelligent vision terminal to emit infrared structured light to acquire point cloud data on site, identifies benchmark markers to establish a local three-dimensional coordinate system, and uses an iterative nearest point algorithm to spatially register the local coordinate system with the global coordinate system, thereby achieving accurate matching between the on-site spatial model and the standard operating space, providing a reliable geometric reference basis for action recognition and spatial positioning.

[0025] Optionally, the first-view visual data and motion sensing data are spatiotemporally aligned, and a fusion feature vector representing the current operation state is generated using a preset multimodal fusion algorithm, specifically including:

[0026] The timestamps of the first-view visual data and the motion sensing data are analyzed separately. The time series interpolation algorithm is used to calculate the interpolation value of the motion sensing data at the timestamp of the first-view visual data to obtain the time-aligned motion sensing data.

[0027] Using a preset relative pose transformation matrix, the time-aligned motion sensing data is transformed from the inertial coordinate system corresponding to the motion sensing data to the camera coordinate system where the first-view visual data is located.

[0028] The transformed motion sensing data is substituted into the state prediction equation of the preset Kalman filter algorithm to calculate the prior state estimate at the current moment. The first-view visual data is then substituted into the observation update equation of the Kalman filter algorithm to correct the prior state estimate, resulting in a fused feature vector containing position, attitude, and velocity information.

[0029] The above technical solution analyzes the timestamps of first-person visual data and motion sensing data and performs interpolation alignment. Then, it unifies the sensing data into the visual coordinate system through relative pose transformation and combines the Kalman filter algorithm to predict and correct the state, thereby obtaining a unified feature vector of fused position, attitude and velocity. This achieves spatiotemporal consistency fusion of multi-source sensing information and improves the accuracy and stability of the operation state representation.

[0030] Optionally, the fused feature vector is input into a pre-trained action recognition model for inference operations to obtain actual action features representing the current actual work behavior, specifically including:

[0031] First-person visual data aligned with a spatiotemporal reference is input into the 3D convolutional neural network layer of the action recognition model to extract spatiotemporal feature maps;

[0032] The fused feature vectors and motion sensing data aligned with the spatiotemporal reference are input into the long short-term memory network layer of the action recognition model to extract temporal dependent features.

[0033] The spatiotemporal feature map and the temporal dependent feature are concatenated by channels and jointly decoded through the fully connected layer of the action recognition model to generate actual action features. The actual action features include a probability distribution vector representing the current action category and key point pose parameters.

[0034] The above technical solution extracts spatiotemporal features by inputting aligned visual data into a 3D convolutional neural network, extracts temporal-dependent features by inputting fused feature vectors and motion sensing data into a long short-term memory network, and then concatenates the two types of features and decodes them together through a fully connected layer to output actual action features containing action category probabilities and key point poses. This enables accurate reasoning for multimodal action recognition and provides a high-dimensional action description for compliance judgment.

[0035] Optionally, based on the inter-step logical constraint rules of the work process graph, the target action node to be evaluated is determined, the actual action characteristics are matched with the standard action characteristics in the target action node, and the action deviation and path logic conformity are calculated, specifically including:

[0036] Query the action nodes that have been confirmed to be completed in the previous moment in the task process graph, and traverse according to the direction of the directed edges to obtain all possible successor nodes as candidate target nodes.

[0037] Calculate the cosine similarity between the actual action features and the standard action features encapsulated in each candidate target node, and use the difference between the preset similarity benchmark and the cosine similarity as the action deviation.

[0038] Verify whether the actual action features meet the inter-step logical constraint rules of the directed edge encapsulation, and calculate the compliance ratio of meeting the inter-step logical constraint rules as the path logic compliance degree.

[0039] The above technical solution determines candidate target actions by traversing successor nodes based on completed action nodes in the graph, calculates the cosine similarity between the actual action features and the standard action features to obtain the action deviation, and verifies whether it conforms to logical constraint rules to obtain the path logic compliance. Thus, it accurately reflects the deviation of the operation action and the compliance of the execution process with quantitative indicators, providing a calculable basis for risk assessment.

[0040] Optionally, based on the preset level of the comprehensive risk rating, a corresponding closed-loop control strategy can be triggered, specifically including:

[0041] If the overall risk rating is less than the preset first risk threshold, the overall risk rating is determined to be at the first level. A virtual guide image containing the standard action trajectory is generated and rendered onto the display layer of the smart vision terminal for augmented reality overlay.

[0042] If the overall risk rating is greater than or equal to the first risk threshold and less than the preset second risk threshold, the overall risk rating is determined to be at the second level. A pulse control signal is generated and sent to the wearable device that collects motion sensing data to drive the vibration motor to generate tactile feedback at a preset frequency. The first risk threshold is less than the second risk threshold.

[0043] If the overall risk rating is greater than or equal to the second risk threshold, the overall risk rating is determined to be at the third level. A blocking control command is generated and sent to the controller of the work tool through the wireless communication module to cut off the power supply of the work tool or lock the mechanical operating structure of the work tool.

[0044] The above technical solution triggers different closed-loop control strategies based on the comprehensive risk rating. In low-risk situations, visual guidance is provided by overlaying standard action trajectories with augmented reality. In medium-risk situations, tactile prompts are provided through vibration feedback from wearable devices. In high-risk situations, blocking commands are sent via wireless communication to cut off tool power or lock the mechanism. This achieves graded safety control from prompting to intervention at different risk levels, ensuring dynamic safety protection during the operation.

[0045] Secondly, a compliance monitoring system for safe work operations in work teams is provided, the system including:

[0046] The graph construction module is configured to construct the job process graph. The job process graph consists of action nodes encapsulated with standard action features and standard job space information, and directed edges encapsulated with logical constraint rules between steps.

[0047] The spatial mapping module is configured to respond to the operation trigger command by using a preset intelligent vision terminal to scan the operation site and construct a spatial feature model. The spatial feature model is then registered and mapped with the standard operation spatial information in the operation process map to establish a spatial correspondence between the operation site and the operation process map.

[0048] The spatiotemporal fusion module is configured to acquire first-person visual data and motion sensing data of the operator during the operation, align the first-person visual data and motion sensing data with a spatiotemporal reference, and generate a fusion feature vector representing the current operation state using a preset multimodal fusion algorithm.

[0049] The action recognition module is configured to input the fused feature vector into a pre-trained action recognition model for inference operations to obtain the actual action features that represent the current actual work behavior.

[0050] The compliance verification module is configured to determine the target action node to be evaluated based on the logical constraint rules between steps in the work process diagram, match the actual action characteristics with the standard action characteristics in the target action node, and calculate the action deviation and path logic compliance.

[0051] The graded response module is configured to generate a comprehensive risk rating based on the degree of deviation of actions and the degree of compliance of path logic. According to the preset level of the comprehensive risk rating, it triggers the corresponding closed-loop control strategy to adjust the operator's operation behavior or equipment status. The closed-loop control strategy includes information guidance strategy, sensory warning strategy or equipment intervention strategy.

[0052] Thirdly, an electronic device is provided, including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the above.

[0053] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing instructions that, when executed, perform the method as described in any of the preceding claims.

[0054] In summary, implementing one or more technical solutions provided in this application has at least the following technical effects or advantages:

[0055] By establishing a data-driven closed-loop management system that standardizes, identifies, evaluates, and provides feedback on the entire process of safe work operations, the invention achieves visualized modeling, dynamic identification, and automatic intervention of the work process. Through the introduction of multimodal perception and graph-based knowledge representation during work execution, the system can automatically determine the compliance of work behaviors in complex on-site environments and correct deviations in real-time, effectively reducing safety risks caused by human negligence and violations. Simultaneously, by fusing text specifications, image information, and sensor data, the invention achieves end-to-end digital mapping from standard operating procedures to on-site execution, improving the informatization and intelligence level of safety management. Furthermore, through a multi-level risk response mechanism, the system possesses adaptive risk control capabilities, dynamically adjusting the intensity of prompts or interventions based on the work status, thereby maintaining work efficiency and consistency while ensuring personnel safety. Attached Figure Description

[0056] Figure 1 This is an exemplary system architecture diagram of a compliance monitoring method or system for safe work operations in a work group, which applies the present application.

[0057] Figure 2 This is a flowchart illustrating a method for monitoring the compliance of safe work operations in a work group, as disclosed in this application.

[0058] Figure 3 This is a schematic diagram of a module of a work team safety operation compliance monitoring system disclosed in this application;

[0059] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in this application.

[0060] Figure reference numerals: 100, System architecture; 101, First terminal device; 102, Second terminal device; 103, Third terminal device; 104, Network; 105, Server; 301, Map construction module; 302, Spatial mapping module; 303, Spatiotemporal fusion module; 304, Action recognition module; 305, Compliance verification module; 306, Hierarchical response module; 401, Processor; 402, Communication bus; 403, User interface; 404, Network interface; 405, Memory. Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0062] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0063] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0064] Figure 1 An exemplary system architecture diagram is shown, illustrating an embodiment of a compliance monitoring method or system for safe work operations in a work group, to which this application can be applied.

[0065] like Figure 1 As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0066] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0067] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0068] When terminals 101, 102, and 103 are hardware devices, video capture devices can also be installed on them. These video capture devices can be various devices capable of capturing video, such as cameras, sensors, etc. Users can use the video capture devices on terminals 101, 102, and 103 to capture video.

[0069] Server 105 can be a server that provides various services, such as a backend server for processing data displayed on terminal devices 101, 102, and 103. The backend server can analyze and process the received data and can feed back the processing results (such as recognition results) to the terminal devices.

[0070] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0071] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. In particular, if the target data does not need to be obtained remotely, the above system architecture may exclude the network and include only terminal devices or servers.

[0072] Figure 2This is a flowchart illustrating a method for monitoring the compliance of work group safety operations according to an embodiment of this application. This method can be implemented using a computer program, a microcontroller, or run on a work group safety operation compliance monitoring system. The computer program can be integrated into an application or run as a standalone tool application. The specific steps of the work group safety operation compliance monitoring method are described in detail below.

[0073] S201: Construct a job process graph, which consists of action nodes encapsulating standard action features and standard job space information, and directed edges encapsulating logical constraint rules between steps.

[0074] For example, this application transforms unstructured work procedures into a digital knowledge system with semantic descriptions and visual benchmarks by performing deep analysis and correlation modeling on preset standard work text and image information. This process aims to establish a topological structure containing standard action nodes and temporal logical constraints through automated extraction and logical reconstruction of work elements, thereby providing standardized comparison benchmarks and logical guidance for subsequent real-time monitoring and compliance judgment of complex work processes.

[0075] In one possible implementation, the construction of the work process graph can be achieved through manual input, database import, or automated parsing. In a preferred automated embodiment, the process specifically includes: using a document parsing algorithm to analyze the layout of a preset standard work instruction, separating the work description text block and the work diagram image block; extracting semantic entities from the work description text block, identifying verb entities representing work actions, noun entities representing operation objects, and time adverbial entities representing work order; generating initial action nodes for the work process graph based on verb entities, and parsing the temporal logic of action execution based on time adverbial entities, constructing directed edges connecting each initial action node using the temporal logic; using a preset convolutional neural network to perform feature encoding on the work diagram image block, generating standard visual feature vectors, and associating and encapsulating the standard visual feature vectors with the corresponding verb entities and noun entities to form action nodes; and constructing a topological structure based on the action nodes and directed edges to obtain the work process graph.

[0076] In this application embodiment, the work process graph refers to a structured topological model used to digitally describe standard operating procedures and their internal logical relationships. It represents a knowledge set consisting of standard action nodes and edges defining the sequence and logical constraints between actions. For example, in a power distribution room inspection scenario, the work process graph defines key steps such as "opening the cabinet door," "observing the instruments," and "recording values" as interconnected nodes, and uses directional line segments to specify the execution order that each action must strictly adhere to, thereby providing a digital comparison benchmark for subsequent identification of whether the actual operation is compliant.

[0077] Specifically, the system utilizes document parsing algorithms based on optical character recognition and layout analysis to perform layout analysis on the input pre-defined standard work process using document object model trees or visual segmentation algorithms. By recognizing the document's layout structure, it separates the task description text blocks containing textual explanations from the task illustration image blocks containing example operation images, achieving independent extraction of textual and graphical information. A pre-trained natural language processing model is used to extract semantic entities from the parsed task description text blocks. Through feature matching and contextual understanding, based on part-of-speech tagging and dependency parsing, verb entities representing task actions (such as "rotate," "press"), noun entities representing operation objects (such as "handle," "button"), and temporal adverbial entities representing task order (such as "first step," "then," "finally") are identified, transforming the natural language description into structured semantic tags. Based on the identified verb entities, initial action nodes are generated in the task process graph. Temporal adverbial entities are used to parse the temporal logic of action execution; for example, a causal chain is established between actions "first" and "then," thereby constructing directed edges connecting each initial action node using temporal logic, forming a preliminary task flow graph.

[0078] Furthermore, to endow the graph with visual perception capabilities, a pre-defined convolutional neural network (such as ResNet or VGG architecture) is used to encode features of the separated task diagram image blocks. Spatial texture and object pose features in the image are extracted through multi-layer convolutional operations to generate high-dimensional standard visual feature vectors. These standard visual feature vectors are then associated and encapsulated with corresponding verb and noun entities, mapping the visual features as "standard samples" to the corresponding action definitions, forming action nodes that contain action semantics and visual baseline information. Based on the encapsulated action nodes and directed edges defining logical flow, a topology is constructed. By establishing parent-child relationships and logical indices between nodes, the final task process graph is obtained.

[0079] Furthermore, in this embodiment, the internal data structure of the action node and the directed edge is defined as follows: the action node adopts a "semantic-visual-spatial" triple encapsulation structure: the semantic feature domain stores the standardized semantic label of the action (e.g., Action_ID: 1024, Type: "tightening", Object: "hexagonal nut"); the visual feature domain stores the standard image feature tensor of H×W×C dimensions extracted by the convolutional neural network, serving as the benchmark template for visual matching; and the spatial feature domain defines the legal three-dimensional coordinate range in which the action occurs. This is used to define the spatial boundaries of the operation. Furthermore, the inter-step logical constraint rules encapsulated by directed edges are subdivided into the following three specific types: strong order constraint, which defines that action A must be completed before action B, otherwise the path logic compliance is directly determined to be zero; time delay constraint, which defines that after action A is completed, action B must be triggered within a preset time ΔT, used to monitor whether the operation is slow or timed out; and tool mutual exclusion constraint, which defines that when executing a specific action node, the tool feature identified by motion sensor data must be consistent with the noun entity associated with the node (e.g., the node requires the use of an "insulated handle," and if the "barehanded" feature is detected, it is considered a violation).

[0080] S202: In response to the operation trigger command, use a preset intelligent vision terminal to scan the operation site and construct a spatial feature model. Register and map the spatial feature model with the standard operation spatial information in the operation process map to establish a spatial correspondence between the operation site and the operation process map.

[0081] For example, this application acquires real-time geometric information of the work site through proactive 3D environmental perception and establishes a benchmark association between the physical work space and the digital logical space using coordinate system transformation technology. This process aims to achieve geometric registration of the point cloud data of the actual work scenario and the spatial information of standard operating procedures in the same coordinate system through digital reconstruction and pose alignment of the site environment. This provides a unified spatial coordinate reference benchmark for subsequent spatial compliance assessment and risk identification of work activities.

[0082] In one possible implementation, a pre-set intelligent vision terminal is used to scan the work site and construct a spatial feature model. The spatial feature model is then registered and mapped with the standard work space information in the work process map. Specifically, this includes: driving the intelligent vision terminal to emit infrared structured light and receive reflected signals; calculating the depth point cloud data of the work site based on the reflected signals to obtain the spatial feature model; identifying pre-set reference markers in the spatial feature model and establishing a local three-dimensional coordinate system with the pre-set reference markers as the origin; geometrically matching the point cloud features of the spatial feature model in the local three-dimensional coordinate system with the standard work space information; and using an iterative nearest-point algorithm to calculate the rotation matrix and translation vector of the local three-dimensional coordinate system relative to the pre-stored global coordinate system in the standard work space information to complete the registration and mapping.

[0083] In this embodiment, the spatial feature model refers to a set of geometric features generated after digitally reconstructing the work site using 3D sensing technology. It represents a digital base containing the 3D contours and spatial positions of various equipment, obstacles, and markers within the work environment. For example, in a power distribution room maintenance scenario, the spatial feature model represents a digital map composed of massive amounts of high-precision spatial coordinate points, realistically reflecting the relative positions of the transformer casing, switchgear panels, and ground baseline. This provides a unified geometric reference for subsequent virtual-real registration of the actual scene and standard operating procedures. Registration mapping refers to the process of transforming spatial datasets from different coordinate systems to a unified coordinate system through mathematical transformation to achieve geometric pose alignment. It represents the spatial coordinate transformation relationship between the real-time spatial feature model of the work site and the pre-stored standard operating spatial information in the work process map. For example, in substation maintenance, registration mapping refers to calculating the rotation and translation of the 3D point cloud data of the switchgear obtained by the intelligent vision terminal in real-time scanning, ensuring it completely coincides with the standard spatial reference model defined in the work instruction under the same coordinate system. This allows the monitoring system to accurately determine whether the physical location of the operator is within the safe area required by the standard operating procedure.

[0084] Specifically, the system drives a pre-set intelligent vision terminal to project infrared structured light of a specific pattern, such as gratings, stripes, or speckle patterns, onto the work site to be monitored. When the infrared beam reaches the surface of an object, an infrared camera captures the reflected light signal. Based on the distortion of the structured light pattern on the object's surface, and combined with triangulation principles, the system calculates the depth values ​​of each detection point in real time, generating a depth point cloud data composed of a massive number of spatially scattered points with X, Y, and Z coordinate information. This results in a spatial feature model characterizing the physical environment of the work site. Using feature point detection and pattern recognition algorithms, the system searches for and locates pre-deployed or inherent pre-set reference markers within the generated spatial feature model. These markers may include QR codes with specific coding rules, reflective targets, or equipment edges with significant geometric features. Once successfully identified, the geometric center or specific vertex of the pre-set reference marker is used as the origin, and a local three-dimensional coordinate system with mutually perpendicular axes is established, combining the marker's normal direction or pre-set edge direction. The point cloud features of the spatial feature model in the local three-dimensional coordinate system are then correlated with pre-stored standard work space information in the work process map. In this process, the Iterative Closest Point (ICP) algorithm is used to perform multiple rounds of recursive calculations. By continuously rotating and translating the point cloud data, the minimum average distance between the local point cloud and the corresponding global point cloud in the standard operation space information is found. This allows for the accurate calculation of the rotation matrix and translation vector of the local 3D coordinate system relative to the pre-stored global coordinate system in the standard operation space information, achieving precise alignment between the physical space coordinate system and the map logical space coordinate system, and completing the registration mapping.

[0085] S203: Acquire first-person visual data and motion sensing data of the operator during the operation, align the first-person visual data and motion sensing data with a spatiotemporal reference, and use a preset multimodal fusion algorithm to generate a fusion feature vector representing the current operation status.

[0086] For example, this application eliminates the objective differences in sampling frequency and spatial representation between different sensors by performing spatiotemporal benchmark alignment on heterogeneous sensing data, thereby achieving spatiotemporal benchmark unification and data complementarity between first-person visual perception and physical motion characteristics. This process aims to utilize the feature complementarity of multi-source data (i.e., the spatial resolution of vision and the temporal resolution of inertial sensors) to construct a digital representation of the operational state with high-frequency updates. This ensures spatiotemporal consistency of the data while providing the system with a fused feature vector encompassing multi-dimensional information on position and attitude, supporting subsequent accurate analysis of complex operational behaviors.

[0087] In one possible implementation, the first-view visual data and motion sensing data are spatiotemporally aligned, and a fusion feature vector representing the current operational state is generated using a preset multimodal fusion algorithm. Specifically, this includes: parsing the timestamps of the first-view visual data and the motion sensing data respectively; calculating the interpolated values ​​of the motion sensing data at the timestamp of the first-view visual data using a time series interpolation algorithm to obtain time-aligned motion sensing data; transforming the time-aligned motion sensing data from the inertial coordinate system corresponding to the motion sensing data to the camera coordinate system where the first-view visual data is located using a preset relative pose transformation matrix; substituting the transformed motion sensing data into the state prediction equation of a preset Kalman filter algorithm to calculate the prior state estimate at the current moment; and substituting the first-view visual data into the observation update equation of the Kalman filter algorithm to correct the prior state estimate, thereby obtaining a fusion feature vector containing position, attitude, and velocity information.

[0088] In this embodiment, the fused feature vector refers to a comprehensive feature representation generated by reconstructing heterogeneous sensor data in spatiotemporal dimensions using multimodal fusion technology. This representation is used to indicate the real-time physical motion state and spatial position attributes of the worker within the three-dimensional work space. For example, when working in a narrow cable tunnel, the fused feature vector not only includes the spatial coordinates identified by the camera but also integrates the instantaneous travel speed and human posture angle calculated by the accelerometer and gyroscope. This forms a digital status package that can describe the coupling relationship between the current work environment and personnel behavior in a multi-dimensional and high-precision manner, ensuring that the monitoring system can maintain continuous tracking of the work status even under visual obstruction or violent movement.

[0089] Specifically, the system parses the timestamps of the first-view visual data collected by the intelligent vision terminal and the motion sensing data collected by the wearable sensing device. Since the sampling frequencies of the visual and motion sensors typically differ by orders of magnitude, a time-series interpolation algorithm (such as linear or cubic spline interpolation) is used, with the timestamp of the visual data as the alignment reference, to calculate the interpolated value of the motion sensing data at the timestamp of the first-view visual data. This aligns the high-frequency motion sensing data, resulting in time-aligned motion sensing data. Using a pre-calibrated relative pose transformation matrix (i.e., the extrinsic parameter matrix between the camera and the inertial measurement unit), a spatial mapping transformation between coordinate systems is performed. This transforms the time-aligned motion sensing data from the inertial coordinate system (such as the local physical coordinate system defined by the sensor itself) corresponding to the motion sensing data to the camera coordinate system where the first-view visual data resides, eliminating inconsistencies in spatial description caused by differences in the physical installation position and orientation of the sensors, and achieving a unified spatial reference.

[0090] Furthermore, to achieve high-precision state estimation, this embodiment constructs the following Kalman filter model: The system first constructs a multi-dimensional state vector. Where p, v, and a represent the three-dimensional position, velocity, and acceleration of the operator's hand in the world coordinate system, respectively; q represents the attitude quaternion; and ω represents the angular velocity. During the state prediction phase, the system utilizes the high-frequency acceleration from the Inertial Measurement Unit (IMU). and angular velocity As the control input, the prior state estimate for the next time step is calculated using the kinematic integral equation. During the observation update phase, the system uses first-person view data to calculate the hand pose in the camera coordinate system. As observed values, the system calculates the Kalman gain. The prior estimate is corrected posteriorly using the visual observation residuals, as shown in the following formula: Here, H is the observation matrix used to map the state space to the observation space (its dimension depends on the matching of the dimensions of the state vector and the observation vector). This process uses the absolute position accuracy of the visual data to correct the integral drift of the inertial sensor, and at the same time uses the high-frequency characteristics of the inertial data to fill the gaps between visual frames, thereby outputting a fused feature vector containing position, attitude and velocity information.

[0091] S204: Input the fused feature vector into the pre-trained action recognition model for inference operations to obtain the actual action features that represent the current actual work behavior.

[0092] For example, this application employs a multi-stream deep learning model to extract and encode multi-dimensional features from heterogeneous sensing data. Through multi-channel parallel processing of spatial texture evolution and long-range motion trends, it achieves vector space mapping between visual semantics and physical posture. This process aims to transform low-level multimodal sensing signals into high-dimensional action representations with high-level semantics. It can not only identify the category attributes of actions but also simultaneously lock the spatial geometric pose of the operator during execution, thereby providing objective data support with multi-dimensional descriptive capabilities for subsequent compliance logic judgments.

[0093] In one possible implementation, the fused feature vector is input into a pre-trained action recognition model for inference operations to obtain actual action features representing the current actual work behavior. Specifically, this includes: inputting first-view visual data aligned with a spatiotemporal reference into the 3D convolutional neural network layer of the action recognition model to extract a spatiotemporal feature map; inputting the fused feature vector and motion sensing data aligned with a spatiotemporal reference into the long short-term memory network layer of the action recognition model to extract temporal dependent features; concatenating the spatiotemporal feature map and temporal dependent features through channels, and jointly decoding them through the fully connected layer of the action recognition model to generate actual action features, wherein the actual action features include a probability distribution vector representing the current action category and key point pose parameters.

[0094] In this embodiment, actual action features refer to a set of digital information generated by deep analysis of multidimensional perception data through an algorithmic model. This set of information objectively describes the specific behavioral attributes of the worker at the moment of the operation and represents the semantic classification of the operation and the spatial geometric relationship of the human body structure. For example, in the task of "disconnecting a live wire," the actual action features represent the confidence value of the action category "cutting" determined by the model. They also include the precise coordinates and posture angles of key joints such as the worker's hands and arms in three-dimensional space, thus providing fine-grained judgment criteria for compliance verification.

[0095] Specifically, the system acquires first-person perspective visual data aligned with a spatiotemporal reference. This first-person perspective visual data typically consists of continuous video image frames captured by a smart vision terminal. The video image frame sequence is input into the 3D convolutional neural network layer of the action recognition model. The 3D convolutional kernels simultaneously extract features in the spatial dimension of the image and the temporal dimension of the video to capture the displacement vector of the action between consecutive frames and the texture evolution within the image, thereby extracting a spatiotemporal feature map that integrates spatial geometric contours and short-term motion trends. The fused feature vector (containing position, posture, and velocity information) generated in the previous steps, along with the motion sensing data aligned with the spatiotemporal reference (such as raw numerical sequences of acceleration and angular velocity), is used as the input sequence and input into the Long Short-Term Memory (LSTM) network layer of the action recognition model. The gating mechanism within the LSTM network layer is used to model the long-distance evolution of the motion data, learn the contextual correlation of the action on the time axis, overcome the instantaneous fluctuation interference caused by sensor noise, and then extract temporal dependent features containing long-term action evolution patterns. The extracted spatiotemporal feature maps and temporal dependent features are concatenated along the channel dimension to form a multimodal high-dimensional feature tensor that integrates visual spatial semantics and motion temporal logic. The multimodal high-dimensional feature tensor is then input into the fully connected layer of the action recognition model for joint decoding.

[0096] Furthermore, to improve the accuracy of multimodal feature fusion, this application introduces a multimodal attention mechanism during joint decoding. Specifically, before the fully connected layer decodes the feature tensor after channel concatenation, an attention network layer is used to calculate the visual weight coefficients (such as Weight_Visual) for the spatiotemporal feature map and the sensing weight coefficients (such as Weight_Sensor) for the temporally dependent features. These two weighting coefficients are used to quantify the contribution of different modal features to the action recognition result at the current time step, and are dynamically allocated according to the saliency of the features: when the model detects that it is currently in an action-triggered or contact state (e.g., the moment when the hand makes contact with the tool), the spatial geometric features have high discriminative value, and the attention mechanism will automatically adjust the weight allocation so that Weight_Visual > Weight_Sensor, thereby strengthening the model's feature extraction of visual spatial texture; when the model detects that it is currently in a continuous motion or rapid operation state (e.g., the moment of continuous twisting or large-amplitude swinging), the visual data may be affected by motion blur, leading to feature degradation, and the attention mechanism will adjust the weight allocation so that Weight_Sensor > Weight_Visual, thereby making more use of the high-frequency temporal features of the inertial sensor for compensation. The feature vector after the above weighting coefficients is input into the fully connected layer. The fully connected layer performs nonlinear transformations and weight redistribution on features of different dimensions to achieve a weighted combination and feature space mapping of visual features and motion features, and finally decodes and generates the actual action features. Specifically, the actual action features include: a probability distribution vector representing the likelihood that the current action belongs to various preset standard action categories, and key point pose parameters that quantitatively describe the state of key parts of the worker's body in three-dimensional space. This dynamically weighted joint decoding method effectively reduces the impact of single-modal data quality fluctuations on the recognition results and improves the stability of feature representation under complex working conditions.

[0097] S205: Determine the target action node to be evaluated based on the logical constraint rules between steps in the work process diagram, match the actual action characteristics with the standard action characteristics in the target action node, and calculate the action deviation and path logic compliance.

[0098] For example, this application constructs a multi-dimensional compliance verification mechanism by combining a preset operational logic topology with real-time multimodal behavior perception. This mechanism can automatically lock the benchmark for subsequent standardized actions based on the currently completed tasks and perform quantitative analysis from two dimensions: the accuracy of the action form and the logicality of the process path. In this way, the system, based on the dual dimensions of spatial feature matching and business logic verification, correlates the execution quality of individual actions with the constraints of the overall business process, thereby providing a quantitative evaluation basis for safe operations that combines spatial location accuracy with temporal logical rigor.

[0099] In one possible implementation, the target action node to be evaluated is determined according to the inter-step logical constraint rules of the work process graph. The actual action features are matched with the standard action features in the target action node, and the action deviation and path logic compliance are calculated. Specifically, this includes: querying the action nodes that have been confirmed to be completed in the previous moment in the work process graph, and obtaining all possible successor nodes as candidate target nodes by traversing according to the direction of the directed edges; calculating the cosine similarity between the actual action features and the standard action features encapsulated in each candidate target node, and using the difference between the preset similarity benchmark value and the cosine similarity as the action deviation; verifying whether the actual action features meet the inter-step logical constraint rules encapsulated by the directed edges, and calculating the compliance ratio of meeting the inter-step logical constraint rules as the path logic compliance.

[0100] In this embodiment, path logic compliance refers to an evaluation index that quantitatively characterizes the degree to which the currently executed work sequence matches the preset standardized work process in terms of logical constraints. It is used to indicate the extent to which the actual work path follows the timing, conditions, and causal relationships specified in the work process diagram. For example, in the "power outage and voltage testing" work procedure, if the diagram stipulates that the "voltage testing" action must be performed before "grounding the wire" and that an "inductive voltage tester" must be used, then the path logic compliance will be calculated based on the proportion of these logical rules (such as sequence, tool matching, etc.) satisfied in the actual operation, thereby intuitively reflecting whether there are any skipped items, reversed items, or violations in the work process.

[0101] Specifically, the system retrieves execution status records stored in the job process graph in real time, querying action nodes that were triggered and confirmed to be completed in the previous moment. Following the directed edges radiating outward from the action nodes, it traverses the graph to obtain all successor nodes with direct logical connections to the currently completed nodes, and uses these as candidate target nodes to be matched at the current moment, thereby dynamically predicting the next action that the operator may perform. For each candidate target node, it retrieves the standard action features encapsulated within it. These standard action features are typically high-dimensional feature vectors trained from standard job samples. Using a cosine similarity algorithm in vector space, it calculates the cosine of the angle between the currently generated actual action features and each standard action feature, representing the degree of similarity between the two in terms of action category, pose, and spatial trajectory.

[0102] Furthermore, a preset similarity benchmark is introduced, representing the minimum tolerable threshold for compliant actions. The difference between this benchmark and the cosine similarity is calculated to obtain the action deviation, which quantitatively describes the deviation of the current actual action from the standard specification. Simultaneously, verification of logical consistency is conducted, verifying whether the actual action features satisfy the inter-step logical constraints encapsulated by the corresponding directed edge during execution. These inter-step logical constraints include, but are not limited to, the mandatory order of action execution, the specific tool states required for the operation, and the minimum time interval for the action's duration. Based on this, the compliance ratio is obtained by statistically analyzing the number of instances where inter-step logical constraints are satisfied in actual operations and calculating the ratio of this number to the total number of rules involved in the directed edge. This ratio serves as the path logical compliance. Finally, the combination of action deviation and path logical compliance enables a multi-dimensional and comprehensive accurate assessment of operational compliance.

[0103] S206: Generate a comprehensive risk rating based on the degree of deviation of actions and the degree of compliance of path logic, and trigger the corresponding closed-loop control strategy according to the preset level of the comprehensive risk rating to adjust the operation behavior of operators or the status of equipment. The closed-loop control strategy includes information guidance strategy, sensory warning strategy or equipment intervention strategy.

[0104] For example, this application transforms complex operational perception data into quantifiable risk characterization indicators by weighted coupling calculation of spatial trajectory deviation indicators of operational actions and logical compliance indicators of operational processes. Through a pre-set multi-level response mechanism, the system can automatically match differentiated control measures, ranging from non-intrusive information guidance to mandatory physical intervention, according to the severity of violations. This allows safety monitoring to go beyond simply "detecting risks" and enable real-time correction of dangerous behaviors by operators or changes in equipment operating status through dynamic feedback mechanisms, thus constructing an automated closed-loop feedback mechanism from perception and monitoring to intervention and control.

[0105] In one possible implementation, a corresponding closed-loop control strategy is triggered based on the preset level of the comprehensive risk rating. Specifically, this includes: if the comprehensive risk rating is less than a preset first risk threshold, the comprehensive risk rating is determined to be at the first level; a virtual guidance image containing a standard motion trajectory is generated and rendered onto the display layer of the intelligent vision terminal for augmented reality overlay; if the comprehensive risk rating is greater than or equal to the first risk threshold and less than a preset second risk threshold, the comprehensive risk rating is determined to be at the second level; a pulse control signal is generated and sent to a wearable device that collects motion sensing data to drive a vibration motor to generate tactile feedback at a preset frequency, wherein the first risk threshold is less than the second risk threshold; if the comprehensive risk rating is greater than or equal to the second risk threshold, the comprehensive risk rating is determined to be at the third level; a blocking control command is generated and sent to the controller of the work tool via a wireless communication module to cut off the power supply to the work tool or lock the mechanical operating structure of the work tool.

[0106] In this application embodiment, the closed-loop control strategy refers to an automatic response mechanism that dynamically feeds back and intervenes at the execution end based on real-time monitoring results. It represents a technical solution that, based on the results of work compliance assessments, instantly corrects the behavior of operators or restricts the status of equipment through digital means. For example, in substation maintenance work, if the system detects that the path of the operator holding the grounding wire deviates from the standard safety area, the closed-loop control strategy will immediately project the correct operating trajectory (correction guidance) onto the augmented reality glasses worn by the operator. If the operator continues to violate the rules, a warning will be issued via a vibrating wristwatch, or even the operating tool may be directly locked, thereby constructing a complete automated management loop from "perception-assessment" to "intervention-correction".

[0107] Specifically, the system performs a weighted calculation based on the deviation of actions and the compliance of path logic calculated in the preceding steps to generate a comprehensive risk rating reflecting the safety level of the current operation. This comprehensive risk rating is then matched against preset first and second risk thresholds. If the comprehensive risk rating is less than the preset first risk threshold, it is determined to be at level one, indicating only minor operational irregularities. In this case, standard 3D motion coordinates stored in the operation process map are extracted to generate a virtual guidance image containing the standard action trajectory, such as a semi-transparent "ghost hand" model or dynamic path arrows. This virtual guidance image is rendered onto the transparent display layer of a smart vision terminal (such as AR glasses) using spatial anchoring technology for augmented reality overlay, allowing operators to visually observe deviations from standard procedures and spontaneously correct them. If the comprehensive risk rating is in the middle range, i.e., greater than or equal to the first risk threshold and less than the preset second risk threshold, it is determined to be at level two, indicating that the operation has triggered a safety warning. At this point, a pulse control signal with a specific duty cycle is generated and sent to a wearable device worn by the operator to collect motion sensor data via communication protocols such as Bluetooth Low Energy or Wi-Fi. The pulse control signal drives a vibration motor inside the wearable device to generate tactile feedback at a preset frequency, prompting the operator to return to a standard working posture. If the overall risk rating continues to rise and is greater than or equal to the second risk threshold, the overall risk rating is determined to be at level three, representing an extremely high risk of personal injury or equipment accidents. At this point, a blocking control command with the highest priority is generated and sent in real time to the controller of the controlled work tool via a wireless communication module. Upon receiving the command, the controller immediately triggers the emergency braking logic, executing operations to cut off the power supply to the work tool (e.g., cutting off the power circuit of an electric wrench) or lock the mechanical operating structure of the work tool (e.g., triggering a mechanical lock stop), forcibly stopping the illegal operation process through physical blocking measures to ensure the safety of the work team.

[0108] It is worth noting that, to ensure the stability and safety of the industrial site control strategy, this embodiment introduces hysteresis judgment and safety lockout logic. Strategy debouncing logic: To prevent frequent jumps in the comprehensive risk rating at threshold boundaries (i.e., signal jitter), the system sets an anti-jitter window. For example, when the rating rises from level one to level two, the sensory warning strategy will only be triggered if the rating is greater than or equal to the first risk threshold for N consecutive frames (e.g., 5 frames); and when the risk decreases, the rating must be lower than the first risk threshold - δ (δ is a preset hysteresis value) to revert to the information guidance strategy. Safety lockout for level three risks: For level three equipment intervention strategies (blocking control), the system executes a self-locking intervention. That is, once a blocking control command is issued, even if the risk rating decreases instantaneously within a few seconds, the controller will not immediately resume equipment operation. The system requires a "manual confirmation signal" from the management end or detection of a specific "reset action" performed by the operator (e.g., hands completely leaving the danger zone and remaining still for more than 3 seconds) before the wireless communication module sends an unlock command. This logic achieves mandatory blocking and closed-loop reset management of high-risk operations through hardware-level locking and dual unlocking verification mechanisms.

[0109] Figure 3 This is a schematic diagram of a compliance monitoring system for safe work operations in a work group, as described in this application. This system can be implemented through software, hardware, or a combination of both, forming all or part of the overall system. For example... Figure 3 As shown, the system includes:

[0110] The graph construction module 301 is configured to construct a job process graph. The job process graph consists of action nodes encapsulated with standard action features and standard job space information, and directed edges encapsulated with logical constraint rules between steps.

[0111] The spatial mapping module 302 is configured to respond to the operation triggering command, scan the operation site using a preset intelligent vision terminal and construct a spatial feature model, register and map the spatial feature model with the standard operation spatial information in the operation process map, and establish a spatial correspondence between the operation site and the operation process map.

[0112] The spatiotemporal fusion module 303 is configured to acquire first-view visual data and motion sensing data of the operator during the operation, align the first-view visual data and motion sensing data with a spatiotemporal reference, and generate a fusion feature vector representing the current operation state using a preset multimodal fusion algorithm.

[0113] The action recognition module 304 is configured to input the fused feature vector into a pre-trained action recognition model for inference operations to obtain the actual action features that represent the current actual work behavior.

[0114] The compliance verification module 305 is configured to determine the target action node to be evaluated based on the inter-step logical constraint rules of the work process diagram, match the actual action characteristics with the standard action characteristics in the target action node, and calculate the action deviation and path logic compliance.

[0115] The graded response module 306 is configured to generate a comprehensive risk rating based on the degree of deviation of actions and the degree of compliance of path logic, and trigger the corresponding closed-loop control strategy according to the preset level of the comprehensive risk rating to adjust the operation behavior of operators or the status of equipment. The closed-loop control strategy includes information guidance strategy, sensory warning strategy or equipment intervention strategy.

[0116] Based on the above embodiments, as an optional embodiment, the graph construction module 301 is specifically used for: analyzing the layout of a preset standard work instruction using a document parsing algorithm to separate the work description text block and the work illustration image block; extracting semantic entities from the work description text block to identify verb entities representing work actions, noun entities representing operation objects, and time adverbial entities representing work order; generating initial action nodes for the work process graph based on verb entities, and parsing the temporal logic of action execution according to time adverbial entities, and constructing directed edges connecting each initial action node using temporal logic; using a preset convolutional neural network to perform feature encoding on the work illustration image block to generate standard visual feature vectors, and associating and encapsulating the standard visual feature vectors with the corresponding verb entities and noun entities to form action nodes; and constructing a topological structure based on action nodes and directed edges to obtain the work process graph.

[0117] Based on the above embodiments, as an optional embodiment, the spatial mapping module 302 is specifically used to: drive the intelligent vision terminal to emit infrared structured light and receive reflected signals, calculate the depth point cloud data of the work site based on the reflected signals, and obtain a spatial feature model; identify the preset reference markers in the spatial feature model, and establish a local three-dimensional coordinate system with the preset reference markers as the origin; perform geometric matching between the point cloud features of the spatial feature model in the local three-dimensional coordinate system and the standard work space information, and use the iterative nearest point algorithm to calculate the rotation matrix and translation vector of the local three-dimensional coordinate system relative to the pre-stored global coordinate system in the standard work space information, and complete the registration mapping.

[0118] Based on the above embodiments, as an optional embodiment, the spatiotemporal fusion module 303 is specifically used to: parse the timestamps of the first-view visual data and the motion sensing data respectively; calculate the interpolation value of the motion sensing data at the timestamp of the first-view visual data using a time series interpolation algorithm to obtain time-aligned motion sensing data; transform the time-aligned motion sensing data from the inertial coordinate system corresponding to the motion sensing data to the camera coordinate system where the first-view visual data is located using a preset relative pose transformation matrix; substitute the transformed motion sensing data into the state prediction equation of the preset Kalman filter algorithm to calculate the prior state estimate at the current moment; and substitute the first-view visual data into the observation update equation of the Kalman filter algorithm to correct the prior state estimate, thereby obtaining a fusion feature vector containing position, attitude, and velocity information.

[0119] Based on the above embodiments, as an optional embodiment, the action recognition module 304 is specifically used to: input the first-view visual data aligned with the spatiotemporal reference to the three-dimensional convolutional neural network layer of the action recognition model to extract the spatiotemporal feature map; input the fused feature vector and the motion sensing data aligned with the spatiotemporal reference to the long short-term memory network layer of the action recognition model to extract the temporal dependent features; concatenate the spatiotemporal feature map and the temporal dependent features through channels, and perform joint decoding through the fully connected layer of the action recognition model to generate actual action features, wherein the actual action features include a probability distribution vector representing the current action category and key point pose parameters.

[0120] Based on the above embodiments, as an optional embodiment, the compliance verification module 305 is specifically used for: querying the action nodes that have been confirmed to be completed in the previous moment in the work process graph, and traversing according to the direction of the directed edges to obtain all possible successor nodes as candidate target nodes; calculating the cosine similarity between the actual action features and the standard action features encapsulated in each candidate target node, and using the difference between the preset similarity benchmark value and the cosine similarity as the action deviation; verifying whether the actual action features meet the inter-step logical constraint rules encapsulated by the directed edges, and calculating the compliance ratio that meets the inter-step logical constraint rules as the path logical compliance degree.

[0121] Based on the above embodiments, as an optional embodiment, the graded response module 306 is specifically used for: if the comprehensive risk rating is less than a preset first risk threshold, then the comprehensive risk rating is determined to be at the first level, a virtual guidance image containing a standard motion trajectory is generated, and the virtual guidance image is rendered onto the display layer of the intelligent vision terminal for augmented reality overlay; if the comprehensive risk rating is greater than or equal to the first risk threshold and less than a preset second risk threshold, then the comprehensive risk rating is determined to be at the second level, a pulse control signal is generated and sent to the wearable device that collects motion sensing data, driving the vibration motor to generate tactile feedback at a preset frequency, wherein the first risk threshold is less than the second risk threshold; if the comprehensive risk rating is greater than or equal to the second risk threshold, then the comprehensive risk rating is determined to be at the third level, a blocking control command is generated and sent to the controller of the work tool through the wireless communication module, cutting off the power supply of the work tool or locking the mechanical operating structure of the work tool.

[0122] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0123] This embodiment also discloses an electronic device, as shown in the reference. Figure 4 The electronic device may include: at least one processor 401, at least one communication bus 402, user interface 403, network interface 404, and at least one memory 405.

[0124] The communication bus 402 is used to enable communication between these components.

[0125] The user interface 403 may include a display screen and a camera. Optionally, the user interface 403 may also include a standard wired interface and a wireless interface.

[0126] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0127] The processor 401 may include one or more processing cores. The processor 401 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 405, and by calling data stored in memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 401.

[0128] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for monitoring the compliance of work group safety operations.

[0129] exist Figure 4In the electronic device shown, the user interface 403 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 401 can be used to call the application program stored in the memory 405 for a compliance monitoring method of team safety operation. When executed by one or more processors 401, the electronic device executes one or more methods as described in the above embodiments.

[0130] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0135] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 405 and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory 405 includes various media capable of storing program code, such as a USB flash drive, external hard drive, magnetic disk, or optical disk.

[0136] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the disclosure in this specification. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope of this application is defined by the claims.

Claims

1. A method for monitoring the compliance of safe operations in work groups, characterized in that, The method includes: Construct a job process graph, which consists of action nodes encapsulating standard action features and standard job space information, and directed edges encapsulating logical constraint rules between steps. In response to a job trigger command, a preset intelligent vision terminal is used to scan the job site and construct a spatial feature model. The spatial feature model is then registered and mapped with the standard job spatial information in the job process map to establish a spatial correspondence between the job site and the job process map. First-person visual data and motion sensing data of the operator during the operation are acquired. The first-person visual data and the motion sensing data are aligned with a spatiotemporal reference. A fusion feature vector representing the current operation state is generated using a preset multimodal fusion algorithm. The fused feature vector is input into a pre-trained action recognition model for inference operations to obtain the actual action features that characterize the current actual work behavior. The target action node to be evaluated is determined according to the inter-step logical constraint rules of the operation process graph. The actual action features are matched with the standard action features in the target action node, and the action deviation and path logic conformity are calculated. A comprehensive risk rating is generated based on the deviation of the action and the conformity of the path logic. According to the preset level of the comprehensive risk rating, a corresponding closed-loop control strategy is triggered to adjust the operator's behavior or equipment status. The closed-loop control strategy includes information guidance strategy, sensory warning strategy or equipment intervention strategy.

2. The method according to claim 1, characterized in that, The construction process graph specifically includes: The layout of the pre-set standard operating instructions is analyzed using a document parsing algorithm to separate the task description text block and the task illustration image block; Semantic entity extraction is performed on the task description text block to identify verb entities representing task actions, noun entities representing operation objects, and time adverbial entities representing task order. The initial action nodes of the operation process graph are generated based on the verb entities, and the temporal logic of the action execution is parsed according to the time adverbial entities. The directed edges connecting each initial action node are constructed using the temporal logic. The task diagram image blocks are feature-encoded using a pre-defined convolutional neural network to generate standard visual feature vectors. The standard visual feature vectors are then associated and encapsulated with the corresponding verb entities and noun entities to form the action nodes. The topology is constructed based on the action nodes and the directed edges to obtain the operation process graph.

3. The method according to claim 1, characterized in that, The step of scanning the work site using a pre-set intelligent vision terminal and constructing a spatial feature model, and then registering and mapping the spatial feature model with the standard work spatial information in the work process map, specifically includes: The intelligent vision terminal is driven to emit infrared structured light and receive reflected signals. The depth point cloud data of the work site is calculated based on the reflected signals to obtain the spatial feature model. Identify the preset reference markers in the spatial feature model, and establish a local three-dimensional coordinate system with the preset reference markers as the origin; The point cloud features of the spatial feature model in the local three-dimensional coordinate system are geometrically matched with the standard operation spatial information. The rotation matrix and translation vector of the local three-dimensional coordinate system relative to the global coordinate system pre-stored in the standard operation spatial information are calculated using the iterative nearest point algorithm to complete the registration mapping.

4. The method according to claim 1, characterized in that, The step of aligning the first-view visual data and the motion sensing data with a spatiotemporal reference, and generating a fusion feature vector representing the current operation state using a preset multimodal fusion algorithm, specifically includes: The timestamps of the first-view visual data and the motion sensing data are analyzed separately. The time series interpolation algorithm is used to calculate the interpolation value of the motion sensing data at the timestamp of the first-view visual data to obtain the time-aligned motion sensing data. Using a preset relative pose transformation matrix, the time-aligned motion sensing data is transformed from the inertial coordinate system corresponding to the motion sensing data to the camera coordinate system where the first viewpoint visual data is located. The transformed motion sensing data is substituted into the state prediction equation of the preset Kalman filter algorithm to calculate the prior state estimate at the current moment. The first viewpoint visual data is then substituted into the observation update equation of the Kalman filter algorithm to correct the prior state estimate, thereby obtaining the fused feature vector containing position, attitude, and velocity information.

5. The method according to claim 4, characterized in that, The step of inputting the fused feature vector into a pre-trained action recognition model for inference operations to obtain actual action features representing the current actual work behavior specifically includes: The first-view visual data, aligned with the spatiotemporal reference, is input into the three-dimensional convolutional neural network layer of the action recognition model to extract spatiotemporal feature maps; The fused feature vector and the motion sensing data aligned with the spatiotemporal reference are input into the long short-term memory network layer of the action recognition model to extract temporal dependent features. The spatiotemporal feature map and the temporal dependency feature are concatenated by channels and jointly decoded through the fully connected layer of the action recognition model to generate the actual action features, wherein the actual action features include a probability distribution vector representing the current action category and key point pose parameters.

6. The method according to claim 5, characterized in that, The step of determining the target action node to be evaluated based on the inter-step logical constraint rules of the work process graph, matching the actual action features with the standard action features in the target action node, and calculating the action deviation and path logic conformity specifically includes: Query the action nodes that have been confirmed to be completed in the previous moment in the operation process graph, and traverse according to the direction of the directed edge to obtain all possible successor nodes as candidate target nodes. Calculate the cosine similarity between the actual action features and the standard action features encapsulated in each candidate target node, and use the difference between the preset similarity benchmark value and the cosine similarity as the action deviation. Verify whether the actual action features satisfy the inter-step logical constraint rules encapsulated by the directed edge, and calculate the compliance ratio of satisfying the inter-step logical constraint rules as the path logical compliance degree.

7. The method according to claim 6, characterized in that, The step of triggering a corresponding closed-loop control strategy based on the preset level of the comprehensive risk rating specifically includes: If the overall risk rating is less than a preset first risk threshold, the overall risk rating is determined to be at the first level, a virtual guide image containing a standard action trajectory is generated, and the virtual guide image is rendered onto the display layer of the smart vision terminal for augmented reality overlay. If the comprehensive risk rating is greater than or equal to the first risk threshold and less than the preset second risk threshold, then the comprehensive risk rating is determined to be at the second level, a pulse control signal is generated and sent to the wearable device that collects the motion sensing data, driving the vibration motor to generate tactile feedback at a preset frequency, wherein the first risk threshold is less than the second risk threshold. If the comprehensive risk rating is greater than or equal to the second risk threshold, the comprehensive risk rating is determined to be at the third level. A blocking control command is generated and sent to the controller of the work tool through the wireless communication module to cut off the energy supply of the work tool or lock the mechanical operating structure of the work tool.

8. A compliance monitoring system for safe operation in work groups, characterized in that, The system includes: The graph construction module is configured to construct a job process graph, which consists of action nodes encapsulated with standard action features and standard job space information, and directed edges encapsulated with logical constraint rules between steps. The spatial mapping module is configured to respond to a job triggering command by scanning the job site using a preset intelligent vision terminal and constructing a spatial feature model. The spatial feature model is then registered and mapped with the standard job spatial information in the job process map to establish a spatial correspondence between the job site and the job process map. The spatiotemporal fusion module is configured to acquire first-view visual data and motion sensing data of the operator during the operation, align the first-view visual data and the motion sensing data with a spatiotemporal reference, and generate a fusion feature vector representing the current operation state using a preset multimodal fusion algorithm. The action recognition module is configured to input the fused feature vector into a pre-trained action recognition model for inference operations to obtain actual action features that characterize the current actual work behavior. The compliance verification module is configured to determine the target action node to be evaluated based on the inter-step logical constraint rules of the work process graph, match the actual action features with the standard action features in the target action node, and calculate the action deviation and path logic compliance. The graded response module is configured to generate a comprehensive risk rating based on the deviation of the action and the conformity of the path logic, and trigger a corresponding closed-loop control strategy according to the preset level of the comprehensive risk rating to adjust the operator's operation behavior or equipment status. The closed-loop control strategy includes information guidance strategy, sensory warning strategy or equipment intervention strategy.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-7.

Images