A Method and System for Identifying Violations by Workers in Multiple Scenarios Based on Video Analytics

Abstract

This invention discloses a method and system for identifying violations by workers in multiple scenarios based on video analysis, belonging to the field of intelligent operation safety monitoring and artificial intelligence recognition technology. The method includes: acquiring real-time video streams from multiple work scenarios; using a deep motion recognition model to identify the sequence of continuous actions in the real-time video stream; constructing the sequence of continuous actions into a sequence of action behaviors; constructing the sequence of action behaviors into a directed behavior graph with time, space, and action labels, and comparing it with the directed behavior graph corresponding to a standard action behavior sequence to identify violations; and providing multimodal early warning for the identified violations. This method overcomes the bottlenecks of traditional image recognition technology, such as weak semantic understanding of action sequences and poor environmental adaptability, achieving accurate identification and real-time early warning of violations in multiple work scenarios.

Description

Technical Field

[0001] This invention belongs to the field of intelligent operation safety monitoring and artificial intelligence recognition, specifically involving a method and system for recognizing violations by workers in multiple scenarios based on video analysis. Background Technology

[0002] In various operational scenarios, including industrial production, construction, coal mining, and power maintenance, the level of operational compliance among workers directly impacts production safety, work efficiency, and the safety of personnel. In recent years, with the expansion of industry scale and the increasing complexity of working environments, safety accidents caused by worker violations have become frequent, resulting not only in significant economic losses but also posing a serious threat to the personal safety of workers. Therefore, timely and accurate identification and early warning of worker violations have become a crucial link in ensuring safety in various operational scenarios.

[0003] Traditional methods for identifying violations mainly rely on manual inspections or playback of fixed-point surveillance videos. These methods have significant limitations: on the one hand, manual inspections are limited by labor costs, working hours, and subjective judgment differences, making it difficult to achieve real-time monitoring that is 24 / 7 and fully covered, and are prone to missed or false detections; on the other hand, post-event playback analysis can only be used for accident tracing and cannot intervene in a timely manner when violations occur, making it difficult to fundamentally prevent accidents from happening.

[0004] With the development of video surveillance and artificial intelligence technologies, automated recognition methods based on video analytics have gradually become a research hotspot. Existing technologies include some solutions that identify the individual status of workers through static images (such as whether they are wearing safety helmets or reflective clothing), but these methods can only identify isolated static violation features and cannot handle violations in continuous actions (such as not wearing safety belts as required when working at heights, or violating operating procedures when operating machinery). Other solutions attempt to analyze actions in videos using simple motion recognition models, but due to the large variations in lighting, complex backgrounds, frequent occlusions, and diverse work processes in various work environments, traditional models struggle to effectively extract the temporal features and spatial relationships of actions. This results in low accuracy in recognizing complex continuous actions and poor adaptability, making it difficult to apply across different work scenarios.

[0005] Furthermore, existing methods for identifying violations often only output a single identification result, lacking the integration of temporal and spatial contextual information about the violation, making it difficult to form a complete behavioral chain and resulting in insufficient targeting and effectiveness of warning information. At the same time, in multi-scenario operations, the violation standards differ across scenarios, and existing methods struggle to flexibly adapt to the rules of different scenarios, exhibiting poor versatility and scalability.

[0006] Therefore, developing a method that can adapt to multiple operating environments, accurately identify violations in continuous action sequences, and provide effective multimodal early warnings has become a pressing technical problem in the field of operational safety monitoring. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, the present invention provides a method and system for identifying violations by workers in multiple scenarios based on video analysis. This solution addresses the problems of weak semantic understanding of actions and poor environmental adaptability in traditional image recognition technologies for identifying violations in multiple scenarios, and enables accurate identification and real-time early warning of violations in multiple scenarios.

[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a method for identifying violations by workers in multiple scenarios based on video analysis.

[0009] Includes the following steps:

[0010] Collect real-time video streams from multiple work sites;

[0011] A deep action recognition model is used to identify the temporal sequence of continuous actions in a real-time video stream; the deep action recognition model includes a pose estimation layer, a region semantic attention layer, a temporal modeling layer, and an action classification layer.

[0012] Construct a sequence of continuous actions into a sequence of actions;

[0013] The action sequence is constructed into a directed behavior graph with time, space and action labels, and compared with the directed behavior graph corresponding to the standard action sequence to identify violations.

[0014] Multimodal early warning is provided for identified violations.

[0015] Furthermore, the action recognition deep model is used to identify the timing of continuous actions in a real-time video stream, including:

[0016] In the pose estimation layer, based on the improved OpenPose architecture, the real-time video stream is output as multi-frame skeleton key points. , , This represents the coordinates of the skeleton keypoints in the i-th frame image. This represents the confidence level of the skeleton keypoint coordinates of the i-th frame image. To address the visual confidence correction factor under environmental occlusion, This indicates the local image blur around the key points of the skeleton. Indicates the environmental modulation coefficient. Represents an exponential function;

[0017] In the region semantic attention layer, frame-level region attention features are calculated based on the region labels defined in the task scenario. , This represents the j-th identified action region bounding box. This indicates the standard area marked in the work scenario. Z represents the category confidence level, and Z represents the normalization coefficient. Represents the confidence function;

[0018] In the temporal modeling layer, contextual action features are extracted using a BiLSTM network based on skeleton keypoints and region attention features. Furthermore, a locality-sensitive weighted window mechanism is introduced to handle blurred action boundaries and obtain enhanced action features. In the formula, Indicates the 1st to Multi-frame skeleton key points at different times Indicates the 1st to Frame-level attention features at each moment;

[0019] In the action classification layer, the Softmax classifier outputs the action label probability at each time step corresponding to the enhanced action features. This leads to a continuous sequence of actions; among which, , Represents the weight matrix. This represents the bias vector. express Actions at any given moment.

[0020] Furthermore, the continuous action sequence is constructed into an action behavior sequence, including:

[0021] Arrange the consecutive action sequences in ascending order of time to obtain the ascending action sequence;

[0022] Redundancy filtering and duration aggregation are performed on the ascending sequence of actions to obtain several action segments. ; Each represents an action The corresponding start and end times;

[0023] Constructing action behavior sequences according to action segment time sequence .

[0024] Furthermore, the sequence of actions is constructed into a directed behavior graph with time, space, and action labels, including:

[0025] Each action segment in the action sequence is constructed as a graph node. , Each represents an action The corresponding start and end times, Indicates a region label;

[0026] Form a node set based on the nodes in the graph. ;

[0027] For graph node set Each pair of adjacent actions Construct directed edges And assign weights to multidimensional attributes. ;

[0028] Based on the graph nodes, directed edges and their weights, construct a directed behavioral graph G=(V,E,W) with time, space and action labels;

[0029] Each graph node in the directed behavioral graph G=(V,E,W) Each directed edge represents an action label and its occurrence time and location. Connect two adjacent actions, edge weight It reflects the temporal, semantic, and spatial consistency between actions.

[0030] Furthermore, directed edges Multidimensional attribute weights for:

[0031]

[0032] In the formula, Indicates action and semantic similarity, This indicates that the time interval between actions is normalized. Indicates action End time, Indicates action The start time, Represents the normalization factor. Indicates action tag and Spatial continuity discriminant function, , , Represents the weighting coefficients, and The spatial continuity discriminant function is expressed as:

[0033] .

[0034] Furthermore, the directed behavior graph corresponding to the standard action sequence is compared to identify violations, including:

[0035] Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence;

[0036] The minimum image editing distance is compared with a set threshold to determine and identify violations.

[0037] Furthermore, the minimum graph edit distance for:

[0038]

[0039] In the formula, Indicates the cost of node replacement. Indicates the cost of edge errors or missing edges. and These represent the graph nodes and directed edges in the directed behavior graph corresponding to the action sequence, respectively. and These represent the graph nodes and directed edges in the directed behavior graph corresponding to the standard action sequence, respectively. Represents a set of nodes. Represents the set of edges;

[0040] The function for determining violations is:

[0041]

[0042] In the formula, This indicates that a threshold value is set.

[0043] Furthermore, in multi-scenario deployments, the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence is calculated by introducing graph embedding representation, thereby enabling violation behavior identification.

[0044] A multi-scenario worker violation identification system based on video analytics includes:

[0045] Image acquisition module: Used to acquire real-time video streams from key locations at the work site;

[0046] Action recognition module: Used to identify the sequence of continuous actions in real-time video streams using a deep action recognition model, and to optimize the recognition of action behavior in occluded environments through region awareness and robustness optimization;

[0047] Behavior sequence construction module: used to construct the identified continuous action sequence into a unified action behavior sequence;

[0048] Standard Operation Model Library: Used to store preset standard action behavior sequences for different operation scenarios;

[0049] Behavior semantic graph construction module: used to construct directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences;

[0050] Behavior comparison module: Used to compare the similarity between the directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences based on minimum graph edit distance and / or cosine similarity, thereby identifying violations;

[0051] Early warning module: Used to issue multimodal warnings based on the severity level after identifying violations;

[0052] Storage and Management Module: Used to store and manage behavior recognition results, comparison results, and early warning results.

[0053] Furthermore, the deep action recognition model includes a pose estimation layer, a region semantic attention layer, a temporal modeling layer, and an action classification layer;

[0054] The pose estimation layer is used to identify key points of the skeleton in multiple frames of real-time video streams, and by introducing an image blur perception mechanism, confidence penalty weights are assigned to occluded and blurred areas.

[0055] The region semantic attention layer is used to calculate frame-level region attention features based on the region labels defined in the task scenario;

[0056] The temporal modeling layer is used to fuse skeleton key points and region attention features, extract contextual action features through a BiLSTM network, and introduce a local sensitive weighted window mechanism to handle action boundary ambiguity and obtain enhanced action features.

[0057] The action classification layer is used to output the probability of the action label at each moment corresponding to the enhanced action features through the Softmax classifier, thereby forming a continuous action sequence.

[0058] The beneficial effects of this invention are as follows:

[0059] This invention has significant technical advantages and practical application effects over existing technologies in identifying worker behavior and determining process sequence under complex working conditions in multiple scenarios, specifically reflected in the following aspects:

[0060] (1) Environmental recognition robustness optimization: The MinePose-Seq model proposed in this invention introduces an image fuzziness perception mechanism for common complex environments such as dust, occlusion, and low light, and performs weighted modeling of the confidence of skeleton key points (fuzziness robustness mechanism), which significantly improves the accuracy of action recognition under non-ideal visual conditions and has stronger tolerance for low-quality images.

[0061] (2) Possesses spatial awareness and action understanding ability: Through the regional semantic fusion mechanism, the key action area is matched with the spatial position of the operator by IoU. In addition to the skeleton action, the important information dimension of "the place where the behavior occurs" is introduced to realize the semantic understanding of "what is being done where". This solves the problem that existing methods cannot judge position deviation by relying only on posture and improves the ability to understand complex behaviors.

[0062] (3) Enhanced temporal context and boundary modeling: The MinePose-Seq model enhances the modeling ability of action transition stages on the basis of temporal models (such as BiLSTM) by introducing a local sensitive weighted window mechanism, making the action segment division more accurate and avoiding action segment breakage or mixing due to short-term jitter and misidentification. Compared with existing methods that are mainly based on still frame analysis, it significantly improves the stability of continuous behavior sequence modeling.

[0063] (4) Graph structure representation of behavioral sequences and violation reasoning ability: This invention uses a behavioral semantic graph (with temporal and spatial attributes) to represent the identified action sequences, integrates standard operating procedure modeling, and uses graph comparison algorithms to achieve intelligent judgment of complex violations such as disordered sequence, skipped steps, and execution beyond the specified level, thus having a stronger process compliance detection capability. Compared with traditional edit distance or single-label comparison, it has richer expressive capabilities and higher semantic accuracy.

[0064] (5) Multi-scenario universality and unified structure: By modularizing the structure of the MinePose-Seq model, it supports rapid adaptation to different underground operation scenarios such as tunneling, fully mechanized mining, transportation, and maintenance. It can be deployed by simply adjusting the standard template and regional label definition, and has good engineering promotion value. Existing methods are often limited to specific rules in a single scenario and have poor universality.

[0065] (6) Edge deployment friendly and real-time early warning capability: This system adopts a lightweight structure design, supports deployment and operation on edge computing nodes in the work scenario, and, in conjunction with real-time behavior modeling and multi-modal alarm mechanisms (such as sound and light, platform prompts, and APP push), enables rapid perception and timely intervention of violations.

[0066] (7) Traceable trajectory data and management interface: The system stores and archives all behavior nodes, sequence edges, identification tags and violation results, supports full-process retrospective and analysis of single-person and multi-person operation processes, and provides data basis for post-accident tracing, responsibility division and management assessment. Attached Figure Description

[0067] Figure 1 The flowchart of the multi-scenario worker violation identification method based on video analysis provided by the present invention is shown. Detailed Implementation

[0068] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0069] This invention provides a method for identifying violations by workers in multiple scenarios based on video analytics, such as... Figure 1 As shown, it includes the following steps:

[0070] Collect real-time video streams from multiple work sites;

[0071] A deep action recognition model is used to identify the temporal sequence of continuous actions in real-time video streams. The deep action recognition model includes a pose estimation layer, a region semantic attention layer, a temporal modeling layer, and an action classification layer.

[0072] Construct a sequence of continuous actions into a sequence of actions;

[0073] The action sequence is constructed into a directed behavior graph with time, space and action labels, and compared with the directed behavior graph corresponding to the standard action sequence to identify violations.

[0074] Multimodal early warning is provided for identified violations.

[0075] In this embodiment of the invention, taking an underground coal mine as an example, high-definition cameras deployed at multiple key locations in the underground coal mine work site are used to collect real-time video streams.

[0076] In this embodiment of the invention, the MinePose-Seq motion depth recognition model is used to identify continuous motion in a real-time video stream; specifically, it includes:

[0077] In the pose estimation layer, based on the improved OpenPose architecture, the real-time video stream is output as multi-frame skeleton key points. , , This represents the coordinates of the skeleton keypoints in the i-th frame image. This represents the confidence level of the skeleton keypoint coordinates of the i-th frame image. This represents the visual confidence correction factor under environmental occlusion. The local image blur around the skeleton keypoints. For environmental modulation coefficients, The exponential function is represented by the visual confidence correction factor, which aims to quantify the visual confidence of occluded and blurred areas by introducing an image blur perception mechanism, thereby adjusting the reliability of features under low visibility conditions.

[0078] In the region semantic attention layer, frame-level region attention features are calculated based on the region labels defined in the task scenario. , This represents the j-th identified action region bounding box. This indicates the standard area marked in the work scenario. Z represents the category confidence level, and Z represents the normalization coefficient. Represents the confidence function;

[0079] In the temporal modeling layer, contextual action features are extracted using a BiLSTM network based on skeleton keypoints and region attention features. In the formula, Indicates the 1st to Multi-frame skeleton key points at different times Indicates the 1st to Frame-level attention features at each moment;

[0080] Furthermore, to improve the recognition accuracy of micro-movements and short-term anomalies, MinePose-Seq introduces a locally sensitive weighted window mechanism to handle blurred action boundaries. This mechanism can enhance the feature representation ability of action transition boundaries and obtain enhanced action features. :

[0081]

[0082] In the formula, This represents the temporal Gaussian weights, used to measure the time step. Features of the current time step The degree of contribution, Indicates time step The original action characteristics, Indicates relative to the current time step Integer offset, This indicates the control of the decay rate. The above formula assigns higher weights to nearby time steps and lower weights to farther time steps through an exponential decay function, thereby enhancing the smoothness of action boundaries while preserving short-term anomaly characteristics. Control the decay rate; the smaller the rate, the more sensitive it is to anomalies.

[0083] In the action classification layer, the Softmax classifier outputs the action label probability at each time step corresponding to the enhanced action features. This leads to a continuous sequence of actions; among which, , Represents the weight matrix. This represents the bias vector. express The action at time step. The weight matrix is ​​used to perform a linear transformation on the enhanced action features, mapping the features to the score spaces of different action categories. The bias vector is used to adjust the result after the linear transformation, providing a base score offset for each action category.

[0084] In this embodiment of the invention, the execution process of the above-mentioned action recognition depth model can realize the temporal recognition of continuous actions, and effectively recognize human behavior in occluded environments through area perception and robustness optimization.

[0085] In this embodiment of the invention, the action tags identified in each frame or time segment are based on the output from MinePose-Seq. With the corresponding timestamp t, the continuous action sequence is constructed into an action behavior sequence, including:

[0086] Arrange the consecutive action sequences in ascending order of time to obtain the ascending action sequence;

[0087] Redundancy filtering and duration aggregation are performed on the ascending sequence of actions to obtain several action segments. ; Each represents an action The corresponding start and end times;

[0088] Specifically, taking the work scenario of underground coal mine as an example, if multiple consecutive frames are identified as the same action (such as "support"), they are merged into one action segment. At the same time, a minimum time threshold or a continuous ratio is set to remove noise actions, such as some frames being briefly misidentified as non-critical actions like "walking".

[0089] Constructing action behavior sequences according to action segment time sequence .

[0090] In this invention, in a specific example taking an underground coal mine as the work scenario, if the original continuous action sequence results are as shown in Table 1, then the constructed action behavior sequence is as follows:

[0091] S = [(“Drilling”, t1, t2),(“Support”, t3, t4),(“Evacuation”, t5, t5)]

[0092] Table 1: Results of Continuous Action Timing Recognition

[0093]

[0094] In this embodiment of the invention, the behavioral semantic graph G=(V,E,W) is a graph structure representation of the action sequence, used to model the sequential dependency, semantic similarity and spatial logical relationship between actions, so as to support intelligent comparison of behavioral sequence deviations.

[0095] In this embodiment, the action sequence is constructed as a directed behavior graph with time, space, and action labels, including:

[0096] Each action segment in the action sequence is constructed as a graph node. , Each represents an action The corresponding start and end times, Indicates area labels; for example, actions include "drilling" and "blasting", and area labels include "tunneling face" and "support zone".

[0097] Form a node set based on the nodes in the graph. ;

[0098] For graph node set Each pair of adjacent actions Construct directed edges And assign weights to multidimensional attributes. ;

[0099] Based on the graph nodes, directed edges and their weights, construct a directed behavioral graph G=(V,E,W) with time, space and action labels;

[0100] In the directed behavioral graph G=(V,E,W), each graph node Each directed edge represents an action label and its occurrence time and location. Connect two adjacent actions, edge weight It reflects the temporal, semantic, and spatial consistency between actions.

[0101] In this embodiment, directed edges Multidimensional attribute weights for:

[0102]

[0103] In the formula, Indicates action and semantic similarity, This indicates that the time interval between actions is normalized. Indicates action End time, Indicates action The start time, Represents the normalization factor. Indicates action tag and Spatial continuity discriminant function, , , Represents the weighting coefficients, and The spatial continuity discriminant function is expressed as:

[0104] .

[0105] In this embodiment, the action and semantic similarity Indicates action and The semantic similarity between them is used to handle situations where the action labels are not completely identical but semantically similar (such as "evacuate" and "walk out"). The similarity can be set to a floating-point number between 0 and 1. The higher the similarity, the higher the tolerance for interchangeability between the two actions in the standard process.

[0106] The value of this value reflects the rationality of the time connection between actions; the shorter the interval (or within a reasonable range), the larger the value.

[0107] Spatial continuity discriminant function This is used to determine whether two actions are spatially reasonable and continuous; for example, if "drilling" should occur at the "excavation face" and "evacuation" should occur at the "tunnel exit", and the areas where the two actions are located are reasonably connected, then this item scores 1; otherwise, it scores 0.

[0108] Weighting coefficients , , Adjusting the contribution of each of the three factors to the edge weights respectively, to satisfy... It can be set through experiments or expert experience.

[0109] In one specific embodiment of the present invention, a corresponding standard action sequence S* (such as "drilling → blasting → evacuation → support") is set for different underground coal mine operation scenarios. When constructing the corresponding directed behavior graph, a standard directed behavior graph G*=(V*,E*,W*) is constructed using the same rules to represent the standard job sequence.

[0110] In this embodiment of the invention, a similarity comparison is performed between a standard directed behavior graph and a directed behavior graph to identify violations, including:

[0111] Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence;

[0112] The minimum image editing distance is compared with a set threshold to determine and identify violations.

[0113] In this embodiment, similarity is determined by calculating the minimum graph edit distance (GED). The evaluation is performed, and the distance is defined as the sum of the minimum total operation costs required to transform graph G into G*, yielding the minimum graph edit distance. for:

[0114]

[0115] In the formula, Indicates the cost of node replacement. Indicates the cost of edge errors or missing edges. and These represent the graph nodes and directed edges in the directed behavior graph corresponding to the action sequence, respectively. and These represent the graph nodes and directed edges in the directed behavior graph corresponding to the standard action sequence, respectively. Represents a set of nodes. This represents a set of edges; where node replacement cost refers to the cost of replacement if the actual action label is inconsistent with the standard action (e.g., support is misidentified as walking), and edge error or missing cost refers to the cost of edge editing if the actual action sequence is inconsistent with the standard or if the action steps are missing.

[0116] In this embodiment, the determination function for judging violations is:

[0117]

[0118] In the formula, This indicates that a threshold value is set.

[0119] The method provided in this embodiment can not only detect explicit violations (such as missing steps or reversed order), but also tolerate minor operational fluctuations (such as semantically similar action substitutions), through an adjustable threshold. Achieve flexible fault-tolerant control and differentiated early warning.

[0120] In this embodiment of the invention, to further improve system operating efficiency, under multi-scenario deployment, a graph embedding representation is introduced to calculate the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence, thereby enabling violation behavior identification; wherein, the cosine similarity between the behavior graph and the standard graph embedding vector is calculated. Quickly determine the degree of behavioral deviation:

[0121]

[0122] The mechanism provided in this embodiment is particularly suitable for real-time violation screening in edge computing environments, balancing accuracy and computing speed, and further improving the system's practicality and responsiveness.

[0123] In this embodiment of the invention, once a violation is identified by the above method, an early warning message is immediately issued according to the severity level through means such as sound and light alarms, control platforms, and mobile apps; furthermore, the identification results, comparison results, and alarm information of the actions are stored in a database for management review and accident tracing.

[0124] In this embodiment of the invention, based on the above-mentioned method for identifying violations by workers in multiple scenarios based on video analysis, a system for identifying violations by workers in multiple scenarios is also provided, comprising:

[0125] Image acquisition module: Used to acquire real-time video streams from key locations at the work site;

[0126] Action recognition module: Used to identify the sequence of continuous actions in real-time video streams using a deep action recognition model, and to optimize the recognition of action behavior in occluded environments through region awareness and robustness optimization;

[0127] Behavior sequence construction module: used to construct the identified continuous action sequence into a unified action behavior sequence;

[0128] Standard Operation Model Library: Used to store preset standard action behavior sequences for different operation scenarios;

[0129] Behavior semantic graph construction module: used to construct directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences;

[0130] Behavior comparison module: Used to compare the similarity between the directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences based on minimum graph edit distance and / or cosine similarity, thereby identifying violations;

[0131] Early warning module: Used to issue multimodal warnings based on the severity level after identifying violations;

[0132] Storage and Management Module: Used to store and manage behavior recognition results, comparison results, and early warning results.

[0133] In this embodiment of the invention, the action recognition deep model includes a pose estimation layer, a region semantic attention layer, a temporal modeling layer, and an action classification layer;

[0134] The pose estimation layer is used to identify multi-frame skeleton key points in a real-time video stream. Furthermore, by introducing an image blur perception mechanism, confidence penalty weights are assigned to occluded and blurred areas; among them, the multi-frame skeleton key point information includes key point coordinates and their confidence scores.

[0135] Furthermore, by introducing an image blur perception mechanism, confidence penalty weights are assigned to occluded and blurred regions, resulting in a visual confidence correction factor expressed as:

[0136]

[0137] The semantic attention layer is used to calculate frame-level region attention features based on the region labels defined in the task scenario. ;

[0138] The temporal modeling layer is used to fuse skeleton keypoints and region attention features, and extracts contextual action features through a BiLSTM network. Furthermore, a locality-sensitive weighted window mechanism is introduced to handle blurred action boundaries and obtain enhanced action features. ;

[0139] The action classification layer is used to output the action label probability at each time step corresponding to the enhanced action features through the Softmax classifier. This leads to a continuous sequence of actions.

[0140] In this embodiment of the invention, the method for the behavior semantic graph construction module to construct a directed behavior graph corresponding to an action behavior sequence is as follows:

[0141] Each action segment in the action sequence is constructed as a graph node. , Each represents an action The corresponding start and end times, Indicates a region label;

[0142] Form a node set based on the nodes in the graph. ;

[0143] For graph node set Each pair of adjacent actions Construct directed edges And assign weights to multidimensional attributes. ;

[0144] Based on the graph nodes, directed edges and their weights, construct a directed behavioral graph G=(V,E,W) with time, space and action labels;

[0145] In the directed behavioral graph G=(V,E,W), each graph node Each directed edge represents an action label and its occurrence time and location. Connect two adjacent actions, edge weight It reflects the temporal, semantic, and spatial consistency between actions.

[0146] In this embodiment of the invention, the method for identifying violations through a behavior comparison module is as follows:

[0147] Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence;

[0148] The minimum image editing distance is compared with a set threshold to determine and identify violations.

[0149] Furthermore, in multi-scenario deployments, the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence is calculated by introducing graph embedding representation, thereby enabling violation behavior identification.

[0150] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0151] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for identifying violations by workers in multiple scenarios based on video analytics, characterized in that, Includes the following steps: Collect real-time video streams from multiple work sites; A deep action recognition model is used to identify the temporal sequence of continuous actions in a real-time video stream; the deep action recognition model includes a pose estimation layer, a region semantic attention layer, a temporal modeling layer, and an action classification layer. Construct a sequence of continuous actions into a sequence of actions; The action sequence is constructed into a directed behavior graph with time, space and action labels, and compared with the directed behavior graph corresponding to the standard action sequence to identify violations. Provide multimodal early warnings for identified violations; Using deep motion recognition models to identify the timing of continuous actions in real-time video streams, including: In the pose estimation layer, based on the improved OpenPose architecture, the real-time video stream is output as multi-frame skeleton key points. , , This represents the coordinates of the skeleton keypoints in the i-th frame image. This represents the confidence level of the skeleton keypoint coordinates of the i-th frame image. To address the visual confidence correction factor under environmental occlusion, This indicates the local image blur around the key points of the skeleton. Indicates the environmental modulation coefficient. Represents an exponential function; In the region semantic attention layer, frame-level region attention features are calculated based on the region labels defined in the task scenario. , This represents the j-th identified action region bounding box. This indicates the standard area marked in the work scenario. Z represents the category confidence level, and Z represents the normalization coefficient. Represents the confidence function; In the temporal modeling layer, contextual action features are extracted using a BiLSTM network based on skeleton keypoints and region attention features. Furthermore, a locality-sensitive weighted window mechanism is introduced to handle blurred action boundaries and obtain enhanced action features. In the formula, Indicates the 1st to Multi-frame skeleton key points at different times Indicates the 1st to Frame-level attention features at each moment; In the action classification layer, the Softmax classifier outputs the action label probability at each time step corresponding to the enhanced action features. This leads to a continuous sequence of actions; among which, , Represents the weight matrix. This represents the bias vector. express Actions at any given moment.

2. The method for identifying violations by workers in multiple scenarios based on video analysis according to claim 1, characterized in that, Constructing continuous action sequences into action behavior sequences, including: Arrange the consecutive action sequences in ascending order of time to obtain the ascending action sequence; Redundancy filtering and duration aggregation are performed on the ascending sequence of actions to obtain several action segments. ; Each represents an action The corresponding start and end times; Constructing action behavior sequences according to action segment time sequence .

3. The method for identifying violations by workers in multiple scenarios based on video analysis according to claim 1, characterized in that, Constructing action sequences into directed behavior graphs with temporal, spatial, and action labels, including: Each action segment in the action sequence is constructed as a graph node. , Each represents an action The corresponding start and end times, Indicates a region label; Form a node set based on the nodes in the graph. ; For graph node set Each pair of adjacent actions Construct directed edges And assign weights to multidimensional attributes. ; Based on the graph nodes, directed edges and their weights, construct a directed behavioral graph G=(V,E,W) with time, space and action labels; Each graph node in the directed behavioral graph G=(V,E,W) Each directed edge represents an action label and its occurrence time and location. Connect two adjacent actions, edge weight It reflects the temporal, semantic, and spatial consistency between actions.

4. The method for identifying violations by workers in multiple scenarios based on video analysis according to claim 3, characterized in that, Directed edge Multidimensional attribute weights for: In the formula, Indicates action and semantic similarity, This indicates that the time interval between actions is normalized. Indicates action End time, Indicates action The start time, Represents the normalization factor. Indicates action tag and Spatial continuity discriminant function, , , Represents the weighting coefficients, and The spatial continuity discriminant function is expressed as: 。 5. The method for identifying violations by workers in multiple scenarios based on video analysis according to claim 1, characterized in that, The directed behavior graph corresponding to the standard action sequence is compared to identify violations, including: Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence; The minimum image editing distance is compared with a set threshold to determine and identify violations.

6. The method for identifying violations by workers in multiple scenarios based on video analysis according to claim 5, characterized in that, The minimum image editing distance for: In the formula, Indicates the cost of node replacement. Indicates the cost of edge errors or missing edges. and These represent the graph nodes and directed edges in the directed behavior graph corresponding to the action sequence, respectively. and These represent the graph nodes and directed edges in the directed behavior graph corresponding to the standard action sequence, respectively. Represents a set of nodes. Represents the set of edges; The function for determining violations is: In the formula, This indicates that a threshold value is set.

7. The method for identifying violations by workers in multiple scenarios based on video analysis according to claim 1, characterized in that, In multi-scenario deployments, the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence is calculated by introducing graph embedding representation, thereby enabling violation behavior identification.

8. A multi-scenario worker violation identification system based on video analysis, implemented based on the multi-scenario worker violation identification method according to any one of claims 1 to 7, characterized in that, include: Image acquisition module: Used to acquire real-time video streams from key locations at the work site; Action recognition module: Used to identify the sequence of continuous actions in real-time video streams using a deep action recognition model, and to optimize the recognition of action behavior in occluded environments through region awareness and robustness optimization; Behavior sequence construction module: used to construct the identified continuous action sequence into a unified action behavior sequence; Standard Operation Model Library: Used to store preset standard action behavior sequences for different operation scenarios; Behavior semantic graph construction module: used to construct directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences; Behavior comparison module: Used to compare the similarity between the directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences based on minimum graph edit distance and / or cosine similarity, thereby identifying violations; Early warning module: Used to issue multimodal warnings based on the severity level after identifying violations; Storage and Management Module: Used to store and manage behavior recognition results, comparison results, and early warning results.

9. The multi-scenario worker violation identification system based on video analysis according to claim 8, characterized in that, The deep action recognition model includes a pose estimation layer, a region semantic attention layer, a temporal modeling layer, and an action classification layer. The pose estimation layer is used to identify key points of the skeleton in multiple frames of real-time video streams, and by introducing an image blur perception mechanism, confidence penalty weights are assigned to occluded and blurred areas. The region semantic attention layer is used to calculate frame-level region attention features based on the region labels defined in the task scenario; The temporal modeling layer is used to fuse skeleton key points and region attention features, extract contextual action features through a BiLSTM network, and introduce a local sensitive weighted window mechanism to handle action boundary ambiguity and obtain enhanced action features. The action classification layer is used to output the probability of the action label at each moment corresponding to the enhanced action features through the Softmax classifier, thereby forming a continuous action sequence.

Images