Video analysis-based multi-scene operator violation behavior identification method and system

Abstract

The invention discloses a video analysis-based multi-scene operator violation behavior identification method and system, and belongs to the technical field of intelligent operation safety monitoring and artificial intelligence identification, and the method comprises the steps: collecting a real-time video stream of a multi-scene operation site; recognizing a continuous action time sequence in the real-time video stream by using an action recognition depth model; constructing the continuous action time sequence into an action behavior sequence; the action behavior sequence is constructed into a directed behavior graph with time, space and action labels, the directed behavior graph is compared with a directed behavior graph corresponding to the standard action behavior sequence, and illegal behaviors are recognized; and carrying out multi-mode early warning on the identified illegal behaviors. According to the method, the bottleneck that the traditional image recognition technology is weak in action sequence semantic understanding and poor in environmental adaptability is broken through, and accurate recognition and real-time early warning of illegal behaviors in multi-scene operation are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent operation safety monitoring and artificial intelligence recognition, and specifically relates to a method and system for identifying illegal behaviors of operators in multiple scenarios based on video analysis. Background Art

[0002] In diverse operational scenarios, such as industrial production, construction, coal mining, and power operation and maintenance, the degree of operator compliance is directly linked to production safety, operational efficiency, and the safety of personnel. In recent years, with the expansion of industries and the increasing complexity of operating environments, safety accidents caused by operator misconduct have become frequent, resulting in significant economic losses and posing a serious threat to the personal safety of employees. Therefore, timely and accurate identification and early warning of operator misconduct have become critical for ensuring operational safety in these diverse scenarios. Traditional violation identification relies mainly on manual inspections or fixed-point surveillance video playback inspections, which have obvious limitations. On the one hand, manual inspections are limited by labor costs, working hours and subjective judgment differences, making it difficult to achieve all-weather, full-coverage real-time monitoring, and are prone to missed detections and false detections. On the other hand, post-event playback analysis can only be used for accident tracing, and cannot intervene in time when violations occur, making it difficult to fundamentally prevent accidents. With the development of video surveillance and artificial intelligence technologies, automated recognition methods based on video analysis have gradually become a research hotspot. Some existing solutions use static images to identify individual worker status (such as whether they are wearing a hard hat or reflective clothing). However, these methods can only identify isolated, static violation features and are unable to address violations during continuous motion (such as not wearing a safety belt as required during high-altitude work or violating operating procedures when operating machinery). Other solutions attempt to use simple action recognition models to analyze actions in videos. However, due to the large lighting variations, complex backgrounds, frequent occlusions, and diverse work processes in multi-scene work environments, traditional models struggle to effectively extract the temporal features and spatial correlation information of actions. This results in low recognition accuracy for complex continuous actions, poor adaptability, and difficulty in general application across different work scenarios. Furthermore, existing violation identification methods often only output a single identification result, lacking the integration of temporal and spatial contextual information about the violation. This makes it difficult to form a complete behavioral chain, resulting in insufficiently targeted and effective early warning information. Furthermore, in multi-scenario operations, violation standards vary across different scenarios, making existing methods difficult to flexibly adapt to the rules of different scenarios, resulting in poor versatility and scalability. Therefore, developing a method that can adapt to multi-scenario working environments, accurately identify violations in continuous action sequences, and perform effective multi-modal early warning has become a technical problem that needs to be urgently solved in the current field of work safety monitoring. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the method and system for identifying violations by workers in multiple scenarios based on video analysis provided by the present invention solve the problems of weak semantic understanding of actions and poor environmental adaptability of traditional image recognition technology in the existing process of identifying violations at multi-scenario work sites, and realize the accurate identification and real-time warning of violations in multi-scenario operations.

[0004] In order to achieve the above-mentioned purpose of the invention, the technical solution adopted by the present invention is: a multi-scene operator violation behavior identification method based on video analysis, The following steps are involved: Collect real-time video streams from multiple job sites; Using a deep action recognition model to identify continuous action sequences in real-time video streams; the deep action recognition model includes a posture estimation layer, a regional semantic attention layer, a time sequence modeling layer, and an action classification layer; Constructing the sequential action sequence into an action behavior sequence; The action sequence is constructed as a directed behavior graph with time, space and action labels, and compared with the directed behavior graph corresponding to the standard action sequence to identify illegal behaviors; Provide multimodal early warning for identified violations.

[0005] Furthermore, the action recognition deep model is used to identify the timing of continuous actions in the real-time video stream, including: In the pose estimation layer, based on the improved OpenPose structure, the real-time video stream is output as multi-frame skeleton key points , 、 Represents the coordinates of the skeleton key points of the i-th frame image, Represents the confidence of the coordinates of the skeleton key points of the i-th frame image, is the visual confidence correction factor for environmental occlusion, represents the local image blur around the skeleton key points, represents the environmental modulation coefficient, represents the exponential function; In the regional semantic attention layer, frame-level regional attention features are calculated based on the regional labels defined in the job scenario. , represents the jth recognized action region box, Indicates the standard area marked in the operation scene, represents the category confidence, Z represents the normalization coefficient, represents the confidence function; In the temporal modeling layer, contextual action features are extracted through the BiLSTM network based on the skeleton key points and regional attention features. , and introduces a local sensitive weighted window mechanism to deal with the blurred action boundary and obtain enhanced action features Where, Indicates the 1st~ Multi-frame skeleton key points at each moment, Indicates the 1st~ Frame-level attention features at each moment; In the action classification layer, the probability of the action label at each moment corresponding to the enhanced action feature is output by the Softmax classifier , thus forming a continuous action sequence; among them, , represents the weight matrix, represents the bias vector, express Moment of action.

[0006] Furthermore, the continuous action sequence is constructed into an action behavior sequence, including: Arrange the continuous action sequence in ascending order of time to obtain the ascending action sequence; Perform redundant filtering and duration aggregation on the ascending action sequence to obtain several action segments ; Respectively represent actions The corresponding start and end times; Construct action behavior sequence according to action segment timing .

[0007] Furthermore, the action sequence is constructed as a directed behavior graph with time, space, and action labels, including: Construct each action segment in the action sequence as a graph node , Respectively represent actions The corresponding start and end times, Indicates the region label; Forming a node set based on graph nodes ; Graph node collection Each pair of adjacent actions in , construct directed edges , and assign multidimensional attribute weights ; Based on the graph nodes, directed edges and their weights, a directed behavior graph G = (V, E, W) with time, space and action labels is constructed; In the directed behavior graph G=(V,E,W), each graph node Represents an action label and its occurrence time and area. Each directed edge Connecting two adjacent actions, edge weight Reflects the temporal, semantic and spatial consistency between actions.

[0008] Furthermore, directed edges Multidimensional attribute weights for: Where, Indicates action and The semantic similarity of represents the normalized time interval between actions, Indicates action The end time, Indicates action The start time, represents the normalization factor, Indicates action label and The spatial continuous discriminant function of 、 、 represents the weighting coefficient, and ; Among them, the spatial continuous discriminant function is expressed as: .

[0009] Furthermore, the directed behavior graph corresponding to the standard action behavior sequence is compared to identify illegal behaviors, including: Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence; The minimum graph edit distance is compared with the set threshold to determine the violation and identify the violation.

[0010] Furthermore, the minimum graph edit distance for: Where, represents the node replacement cost, represents the edge error or missing cost, and They represent the graph nodes and directed edges in the directed behavior graph corresponding to the action behavior sequence, and They represent the graph nodes and directed edges in the directed behavior graph corresponding to the standard action behavior sequence, Represents a collection of nodes, represents an edge set; The judgment function for determining violations is: Where, Indicates setting a threshold.

[0011] Furthermore, in multi-scenario deployment, by introducing graph embedding representation to calculate the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence, illegal behavior identification can be performed.

[0012] The multi-scenario operator violation behavior recognition system based on video analysis includes: Image acquisition module: used to collect real-time video streams of key locations at the work site; Action Recognition Module: This module uses a deep action recognition model to identify continuous action sequences in real-time video streams, and uses region perception and robustness optimization to identify action behaviors in occluded environments. Behavior sequence construction module: used to construct the recognized continuous action sequence into an action behavior sequence in a unified format; Standard operation model library: used to store preset standard action behavior sequences in different operation scenarios; Behavior semantic graph construction module: used to construct directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences; Behavior comparison module: used to compare the similarity between the directed behavior graphs of the action behavior sequence and its corresponding standard action behavior sequence based on the minimum graph edit distance and / or cosine similarity, thereby identifying violations; Early warning module: used to issue multi-modal warnings based on severity levels after identifying violations; Storage and management module: used to store and manage behavior recognition results, comparison results and warning results.

[0013] Furthermore, the action recognition deep model includes a posture estimation layer, a regional semantic attention layer, a temporal modeling layer, and an action classification layer; The pose estimation layer is used to identify multi-frame skeleton key points of real-time video streams and assign confidence penalty weights to occluded and blurred areas by introducing an image fuzziness perception mechanism; The regional semantic attention layer is used to calculate frame-level regional attention features based on the regional labels defined in the operation scenario; The temporal modeling layer is used to fuse skeleton key points and regional attention features, extract contextual action features through the BiLSTM network, and introduce a local sensitive weighted window mechanism to process action boundary blur to obtain enhanced action features; The action classification layer is used to output the enhanced action feature corresponding to the action label probability at each moment through the Softmax classifier, thereby forming a continuous action sequence.

[0014] The beneficial effects of the present invention are: The present invention has significant technical advantages and practical application effects over existing technologies in identifying operator behavior and determining process sequence in complex working conditions with multiple scenarios, which are specifically reflected in the following aspects: (1) Environment-oriented recognition robustness optimization: The MinePose-Seq model proposed in this paper introduces an image fuzziness perception mechanism for common complex environments such as dust, occlusion, and low illumination, and performs weighted modeling of the confidence of skeleton key points (fuzzy robustness mechanism), which significantly improves the accuracy of action recognition under non-ideal visual conditions and has stronger tolerance for low-quality images.

[0015] (2) Capability of motion understanding with spatial perception: Through the regional semantic fusion mechanism, the key action area is matched with the spatial position of the operator by IoU, and the important information dimension of "behavior location" is introduced in addition to the skeleton action to achieve the semantic understanding of "what to do where", which solves the problem that the existing method cannot judge the position deviation by relying solely on posture, and improves the ability to understand complex behaviors.

[0016] (3) Enhanced temporal context and boundary modeling: The MinePose-Seq model introduces a local sensitive weighted window mechanism to enhance the modeling capability of the action transition phase based on the temporal model (such as BiLSTM), making the action segment division more accurate and avoiding the breakage or mixing of action segments due to short-term jitter and misidentification. Compared with the existing methods that are mainly based on static frame analysis, the stability of continuous behavior sequence modeling is significantly improved.

[0017] (4) Graph structure representation of behavior sequences and violation reasoning capabilities: This invention uses behavioral semantic graphs (with temporal and spatial attributes) to represent the identified action sequences, integrates standard operating process modeling, and uses graph comparison algorithms to achieve intelligent judgment of complex violations such as sequence disorder, step skipping, and skipping execution, thus providing stronger process compliance detection capabilities. Compared with traditional edit distance or single-label comparison, it has richer expression capabilities and higher semantic accuracy.

[0018] (5) Multi-scenario universality and structural unification: By modularizing the MinePose-Seq model structure, it supports rapid adaptation to different underground operation scenarios such as tunneling, fully mechanized mining, transportation, and maintenance. It can be deployed by simply adjusting the standard template and regional label definition, which has good engineering promotion value. Existing methods are often limited to specific rules of a single scenario and have poor universality.

[0019] (6) Edge deployment-friendly and real-time warning capabilities: This system adopts a lightweight structural design and supports deployment and operation on edge computing nodes in operational scenarios. It cooperates with real-time behavior modeling and multimodal alarm mechanisms (such as sound and light, platform prompts, and APP push) to achieve rapid perception and timely intervention of violations.

[0020] (7) Traceable trajectory data and management interface: The system stores and archives all behavior nodes, sequential edges, identification labels and violation results, supports full-process backtracking and analysis of single-person and multi-person operation processes, and provides data basis for post-accident tracing, responsibility division, and management assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the method for identifying illegal behaviors of operators in multiple scenarios based on video analysis provided by the present invention. DETAILED DESCRIPTION

[0022] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0023] The embodiment of the present invention provides a method for identifying illegal behaviors of operators in multiple scenarios based on video analysis, such as Figure 1 As shown, the following steps are included: Collect real-time video streams from multiple job sites; Use the action recognition deep model to identify the continuous action sequence in real-time video stream; the action recognition deep model includes a posture estimation layer, a regional semantic attention layer, a time sequence modeling layer, and an action classification layer; Constructing the sequential action sequence into an action behavior sequence; The action sequence is constructed as a directed behavior graph with time, space and action labels, and compared with the directed behavior graph corresponding to the standard action sequence to identify illegal behaviors; Provide multimodal early warning for identified violations.

[0024] In the embodiment of the present invention, taking the operation site as an underground coal mine as an example, real-time video streams are collected by high-definition cameras deployed at multiple key locations of the operation site in the underground coal mine.

[0025] In an embodiment of the present invention, the action depth recognition model MinePose-Seq is used to recognize continuous actions in real-time video streams. Specifically, it includes: In the pose estimation layer, based on the improved OpenPose structure, the real-time video stream is output as multi-frame skeleton key points , 、 Represents the coordinates of the skeleton key points of the i-th frame image, Represents the confidence of the coordinates of the skeleton key points of the i-th frame image, Represents the visual confidence correction factor for environmental occlusion, is the local image blur around the skeleton key points, is the environmental modulation coefficient, represents an exponential function; the purpose of defining the visual confidence correction factor is to quantify the visual confidence of occluded and blurred areas by introducing an image blur perception mechanism, thereby adjusting feature reliability under low visibility conditions; In the regional semantic attention layer, frame-level regional attention features are calculated based on the regional labels defined in the job scenario. , represents the jth recognized action region box, Indicates the standard area marked in the operation scene, represents the category confidence, Z represents the normalization coefficient, represents the confidence function; In the temporal modeling layer, contextual action features are extracted through the BiLSTM network based on the skeleton key points and regional attention features. Where, Indicates the 1st~ Multi-frame skeleton key points at each moment, Indicates the 1st~ Frame-level attention features at each moment; Furthermore, in order to improve the recognition accuracy of micro-movements and short-term anomalies, MinePose-Seq introduces a local sensitive weighted window mechanism to deal with the fuzzy motion boundary. This mechanism can enhance the feature expression ability of the motion transition boundary and obtain enhanced motion features. : Where, Represents the time series Gaussian weight, which is used to measure the time step The features of the current time step The degree of contribution, Represents the time step The original action characteristics, Relative to the current time step The integer offset of The above formula uses the exponential decay function to give higher weights to adjacent time steps and lower weights to distant time steps, thereby enhancing the smoothness of the action boundary while retaining short-term abnormal characteristics. Controls the decay rate, the smaller it is, the more sensitive it is to anomalies; In the action classification layer, the probability of the action label at each moment corresponding to the enhanced action feature is output by the Softmax classifier , thus forming a continuous action sequence; among them, , represents the weight matrix, represents the bias vector, express The weight matrix is ​​used to perform a linear transformation on the enhanced action features, mapping the features to the score space of different action categories. The bias vector is used to adjust the result after the linear transformation and provide a basic score offset for each action category.

[0026] In an embodiment of the present invention, through the execution process of the above-mentioned action recognition deep model, the temporal recognition of continuous actions can be achieved, and the behavior of people in an occluded environment can be effectively identified through regional perception and robustness optimization.

[0027] In the embodiment of the present invention, based on the output results from MinePose-Seq, that is, the action labels identified in each frame or each time segment And the corresponding timestamp t, the continuous action sequence is constructed into an action behavior sequence, including: Arrange the continuous action sequence in ascending order of time to obtain the ascending action sequence; Perform redundant filtering and duration aggregation on the ascending action sequence to obtain several action segments ; Respectively represent actions The corresponding start and end times; Specifically, taking the coal mine operation scene as an example, if multiple consecutive frames are identified as the same action (such as "support"), they are merged into one action segment. At the same time, a minimum time threshold or duration ratio is set to remove noise actions, such as some frames that briefly misidentify non-critical actions such as "walking"; Construct action behavior sequence according to action segment timing .

[0028] In the present invention, in a specific example taking the working scene of an underground coal mine as an example, if the original continuous action sequence results are as shown in Table 1, the constructed action behavior sequence is: S = [ (“Drilling”, t1, t2), (“Support”, t3, t4), (“Evacuate”, t5, t5)] Table 1: Continuous action sequence recognition results In an embodiment of the present invention, the behavioral semantic graph G=(V, E, W) is a graph structure representation of an action sequence, which is used to model the sequential dependency, semantic similarity and spatial logical relationship between actions to support intelligent comparison of behavioral sequence deviations.

[0029] In this embodiment, the action sequence is constructed as a directed action graph with time, space and action labels, including: Construct each action segment in the action sequence as a graph node , Respectively represent actions The corresponding start and end times, Indicates area labels; for example, actions include "drilling" and "blasting", and area labels include "excavation face" and "support area". Forming a node set based on graph nodes ; Graph node collection Each pair of adjacent actions in , construct directed edges , and assign multidimensional attribute weights ; Based on the graph nodes, directed edges and their weights, a directed behavior graph G = (V, E, W) with time, space and action labels is constructed; In the directed behavior graph G=(V,E,W), each graph node Represents an action label and its occurrence time and area. Each directed edge Connecting two adjacent actions, edge weight Reflects the temporal, semantic and spatial consistency between actions.

[0030] In this embodiment, the directed edge Multidimensional attribute weights for: Where, Indicates action and The semantic similarity of represents the normalized time interval between actions, Indicates action The end time, Indicates action The start time, represents the normalization factor, Indicates action label and The spatial continuous discriminant function of 、 、 represents the weighting coefficient, and ; Among them, the spatial continuous discriminant function is expressed as: .

[0031] In this embodiment, the action and The semantic similarity of Indicates action and The semantic similarity between the two actions is used to handle situations where the action labels are not completely consistent but have similar semantics (such as "evacuate" and "exit"). The similarity can be set as a floating point number between 0 and 1. The higher the similarity, the higher the tolerance for interchangeability between the two actions in the standard process. The value of reflects the rationality of the time connection between actions. The shorter the interval (or within a reasonable range), the larger the value; Spatially continuous discriminant function Used to determine whether two actions are reasonably continuous in space; for example: if "drilling" should occur at the "excavation face" and "evacuation" should occur at the "tunnel exit", if the areas where the two actions are located are reasonably connected, the score for this item is 1, otherwise it is 0.

[0032] Weighting coefficient 、 、 Adjust the contribution of the three factors in the edge weight respectively to meet , which can be set through experiments or expert experience.

[0033] In a specific embodiment of the present invention, corresponding standard action behavior sequences S* (such as "drilling → blasting → evacuation → support") are set for different coal mine underground operation scenes. ,When constructing the corresponding directed behavior graph, the standard directed behavior graph G*=(V*,E*,W*) is constructed with the same rules to represent the standardized operation sequence.

[0034] In an embodiment of the present invention, a similarity comparison is performed between a standard directed behavior graph and a directed behavior graph to identify illegal behaviors, including: Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence; The minimum graph edit distance is compared with the set threshold to determine the violation and identify the violation.

[0035] In this embodiment, similarity is determined by calculating the minimum graph edit distance (GED) The distance is defined as the sum of the minimum operation costs required to transform the graph G into G*, and the minimum graph edit distance is obtained. for: Where, represents the node replacement cost, represents the edge error or missing cost, and They represent the graph nodes and directed edges in the directed behavior graph corresponding to the action behavior sequence, and They represent the graph nodes and directed edges in the directed behavior graph corresponding to the standard action behavior sequence, Represents a collection of nodes, represents the edge set; the node replacement cost refers to the replacement cost if the actual action label is inconsistent with the standard action (for example, support is misidentified as walking), and the edge error or missing cost refers to the edge editing cost if the actual action sequence is inconsistent with the standard or the action steps are missing; In this embodiment, the judgment function for judging illegal behavior is: Where, Indicates setting a threshold.

[0036] The above method provided in this embodiment can not only detect obvious violations (such as missing steps, reversed order), but also tolerate minor operational fluctuations (such as substitution of semantically similar actions). Achieve flexible fault-tolerant control and differentiated early warning.

[0037] In the embodiment of the present invention, in order to further improve the operating efficiency of the system, in multi-scenario deployment, the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence is introduced to represent the calculation, thereby identifying illegal behaviors; wherein, the cosine similarity between the behavior graph and the standard graph embedding vector is calculated. Quickly determine the degree of behavioral deviation: The above mechanism provided in this embodiment is particularly suitable for real-time violation screening in edge computing environments, taking into account both accuracy and computing speed, and further improving the practicality and responsiveness of the system.

[0038] In an embodiment of the present invention, when a violation is identified through the above method, an early warning message is immediately issued according to the severity level through sound and light alarms, control platforms, mobile phone APPs, etc.; further, the recognition results, comparison results and alarm information of the action behavior are stored in the database for management review and accident tracing.

[0039] In an embodiment of the present invention, based on the above-mentioned method for identifying illegal behaviors of operators in multiple scenarios based on video analysis, a system for identifying illegal behaviors of operators in multiple scenarios is further provided, comprising: Image acquisition module: used to collect real-time video streams of key locations at the work site; Action Recognition Module: This module uses a deep action recognition model to identify continuous action sequences in real-time video streams, and uses region perception and robustness optimization to identify action behaviors in occluded environments. Behavior sequence construction module: used to construct the recognized continuous action sequence into an action behavior sequence in a unified format; Standard operation model library: used to store preset standard action behavior sequences in different operation scenarios; Behavior semantic graph construction module: used to construct directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences; Behavior comparison module: used to compare the similarity between the directed behavior graphs of the action behavior sequence and its corresponding standard action behavior sequence based on the minimum graph edit distance and / or cosine similarity, thereby identifying violations; Early warning module: used to issue multi-modal warnings based on severity levels after identifying violations; Storage and management module: used to store and manage behavior recognition results, comparison results and warning results.

[0040] In an embodiment of the present invention, the action recognition deep model includes a posture estimation layer, a regional semantic attention layer, a temporal modeling layer, and an action classification layer; Among them, the posture estimation layer is used to identify the multi-frame skeleton key points of the real-time video stream By introducing the image fuzziness perception mechanism, confidence penalty weights are assigned to occluded and blurred areas; among them, the multi-frame skeleton key point information includes the key point coordinates and their confidence; Furthermore, by introducing the image blur perception mechanism, confidence penalty weights are assigned to occluded and blurred areas, and the visual confidence correction factor is expressed as: The semantic attention layer is used to calculate the frame-level regional attention features based on the regional labels defined in the job scenario. ; The temporal modeling layer is used to fuse skeleton key points and regional attention features, and extract contextual action features through the BiLSTM network. , and introduces a local sensitive weighted window mechanism to deal with the blurred action boundary and obtain enhanced action features ; The action classification layer is used to output the enhanced action feature corresponding to the action label probability at each moment through the Softmax classifier , thus forming a continuous action sequence.

[0041] In an embodiment of the present invention, the method for the behavior semantic graph construction module to construct a directed behavior graph corresponding to an action behavior sequence is as follows: Construct each action segment in the action sequence as a graph node , Respectively represent actions The corresponding start and end times, Indicates the region label; Forming a node set based on graph nodes ; Graph node collection Each pair of adjacent actions in , construct directed edges , and assign multidimensional attribute weights ; Based on the graph nodes, directed edges and their weights, a directed behavior graph G = (V, E, W) with time, space and action labels is constructed; In the directed behavior graph G=(V,E,W), each graph node Represents an action label and its occurrence time and area. Each directed edge Connecting two adjacent actions, edge weight Reflects the temporal, semantic and spatial consistency between actions.

[0042] In an embodiment of the present invention, the method for identifying illegal behaviors through the behavior comparison module is specifically as follows: Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence; The minimum graph edit distance is compared with the set threshold to determine the violation and identify the violation.

[0043] Furthermore, in multi-scenario deployment, by introducing graph embedding representation to calculate the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence, illegal behavior identification can be performed.

[0044] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0045] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A multi-scenario operator violation behavior identification method based on video analysis, characterized by: The following steps are involved: Collect real-time video streams from multiple job sites; Using a deep action recognition model to identify continuous action sequences in real-time video streams; the deep action recognition model includes a posture estimation layer, a regional semantic attention layer, a time sequence modeling layer, and an action classification layer; Constructing the sequential action sequence into an action behavior sequence; The action sequence is constructed as a directed behavior graph with time, space and action labels, and compared with the directed behavior graph corresponding to the standard action sequence to identify illegal behaviors; Provide multimodal early warning for identified violations.

2. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 1 is characterized in that: Use deep action recognition models to identify continuous action sequences in real-time video streams, including: In the pose estimation layer, based on the improved OpenPose structure, the real-time video stream is output as multi-frame skeleton key points , 、 Represents the coordinates of the skeleton key points of the i-th frame image, Represents the confidence of the coordinates of the skeleton key points of the i-th frame image, is the visual confidence correction factor for environmental occlusion, represents the local image blur around the skeleton key points, represents the environmental modulation coefficient, represents the exponential function; In the regional semantic attention layer, frame-level regional attention features are calculated based on the regional labels defined in the job scenario. , represents the jth recognized action region box, Indicates the standard area marked in the operation scene, represents the category confidence, Z represents the normalization coefficient, represents the confidence function; In the temporal modeling layer, contextual action features are extracted through the BiLSTM network based on the skeleton key points and regional attention features. , and introduces a local sensitive weighted window mechanism to deal with the blurred action boundary and obtain enhanced action features Where, Indicates the 1st~ Multi-frame skeleton key points at each moment, Indicates the 1st~ Frame-level attention features at each moment; In the action classification layer, the probability of the action label at each moment corresponding to the enhanced action feature is output by the Softmax classifier , thus forming a continuous action sequence; among them, , represents the weight matrix, represents the bias vector, express Moment of action.

3. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 2 is characterized in that: Construct the continuous action sequence into an action behavior sequence, including: Arrange the continuous action sequence in ascending order of time to obtain the ascending action sequence; Perform redundant filtering and duration aggregation on the ascending action sequence to obtain several action segments ; Respectively represent actions The corresponding start and end times; Construct action behavior sequence according to action segment timing .

4. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 1 is characterized in that: Construct the action sequence into a directed behavior graph with time, space and action labels, including: Construct each action segment in the action sequence as a graph node , Respectively represent actions The corresponding start and end times, Indicates the region label; Forming a node set based on graph nodes ; Graph node collection Each pair of adjacent actions in , construct directed edges , and assign multidimensional attribute weights ; Based on the graph nodes, directed edges and their weights, a directed behavior graph G = (V, E, W) with time, space and action labels is constructed; In the directed behavior graph G=(V,E,W), each graph node Represents an action label and its occurrence time and area. Each directed edge Connecting two adjacent actions, edge weight Reflects the temporal, semantic and spatial consistency between actions.

5. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 4 is characterized in that: Directed edges Multidimensional attribute weights for: Where, Indicates action and The semantic similarity of represents the normalized time interval between actions, Indicates action The end time, Indicates action The start time, represents the normalization factor, Indicates action label and The spatial continuous discriminant function of 、 、 represents the weighting coefficient, and ; Among them, the spatial continuous discriminant function is expressed as: 。 6. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 1 is characterized in that: The directed behavior graph corresponding to the standard action behavior sequence is compared to identify illegal behaviors, including: Calculate the minimum graph edit distance between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence; The minimum graph edit distance is compared with the set threshold to determine the violation and identify the violation.

7. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 6 is characterized in that: The minimum graph edit distance for: Where, represents the node replacement cost, represents the edge error or missing cost, and They represent the graph nodes and directed edges in the directed behavior graph corresponding to the action behavior sequence, and They represent the graph nodes and directed edges in the directed behavior graph corresponding to the standard action behavior sequence, Represents a collection of nodes, represents an edge set; The judgment function for determining violations is: Where, Indicates setting a threshold.

8. The method for identifying illegal behaviors of operators in multiple scenarios based on video analysis according to claim 1 is characterized in that: In multi-scenario deployment, by introducing graph embedding representation to calculate the cosine similarity between the directed behavior graph corresponding to the action behavior sequence and the directed behavior graph corresponding to the standard action behavior sequence, violations can be identified.

9. A multi-scenario operator violation behavior recognition system based on video analysis, implemented based on the multi-scenario operator violation behavior recognition method according to any one of claims 1 to 8, characterized in that: include: Image acquisition module: used to collect real-time video streams of key locations at the work site; Action Recognition Module: This module uses a deep action recognition model to identify continuous action sequences in real-time video streams, and uses region perception and robustness optimization to identify action behaviors in occluded environments. Behavior sequence construction module: used to construct the recognized continuous action sequence into an action behavior sequence in a unified format; Standard operation model library: used to store preset standard action behavior sequences in different operation scenarios; Behavior semantic graph construction module: used to construct directed behavior graphs of action behavior sequences and their corresponding standard action behavior sequences; Behavior comparison module: used to compare the similarity between the directed behavior graphs of the action behavior sequence and its corresponding standard action behavior sequence based on the minimum graph edit distance and / or cosine similarity, thereby identifying violations; Early warning module: used to issue multi-modal warnings based on severity levels after identifying violations; Storage and management module: used to store and manage behavior recognition results, comparison results and warning results.

10. The multi-scenario operator violation behavior identification system based on video analysis according to claim 9 is characterized in that: The action recognition deep model includes a posture estimation layer, a regional semantic attention layer, a temporal modeling layer, and an action classification layer; The pose estimation layer is used to identify multi-frame skeleton key points of real-time video streams and assign confidence penalty weights to occluded and blurred areas by introducing an image fuzziness perception mechanism; The regional semantic attention layer is used to calculate frame-level regional attention features based on the regional labels defined in the operation scenario; The temporal modeling layer is used to fuse skeleton key points and regional attention features, extract contextual action features through the BiLSTM network, and introduce a local sensitive weighted window mechanism to process action boundary blur to obtain enhanced action features; The action classification layer is used to output the enhanced action feature corresponding to the action label probability at each moment through the Softmax classifier, thereby forming a continuous action sequence.

Images