Method, system and equipment for intelligently identifying unsafe behaviors of coal mine operating personnel and medium

Abstract

The invention relates to a method, a system and equipment for intelligently identifying unsafe behaviors of coal mine operating personnel and a medium. The method comprises the following steps: extracting individual trajectory data from an underground video stream through a multi-target tracking algorithm, decomposing continuous actions into an atomic behavior sequence with a space-time mark by using attitude estimation and a space-time diagram convolutional network, and capturing space-time relevance of a behavior chain through an attention mechanism enhanced long and short term memory network to generate a feature vector; risk reasoning is carried out in combination with the coal mine safety knowledge graph to predict the risk event type and probability, and finally early warning information is generated based on a multi-level early warning strategy. According to the scheme, the crossing from isolated action recognition to behavior chain risk prediction is realized, and the early warning capability of potential safety risks in coal mine operation is remarkably improved through fusion of space-time correlation analysis and domain knowledge of the behavior sequence.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence application technology in industrial safety, and in particular relates to a method, system, equipment and medium for intelligent identification of unsafe behaviors of coal mine workers. Background Technology

[0002] Coal mines operate in complex and variable environments with high safety risks, and unsafe acts by personnel are a major contributing factor to coal mine accidents. Traditional coal mine safety monitoring systems primarily rely on sensors to monitor environmental parameters (such as gas concentration and temperature) and simple video surveillance, offering limited capabilities for monitoring and analyzing personnel behavior. In recent years, with the development of artificial intelligence technology, computer vision-based unsafe behavior recognition technology has been widely applied, such as identifying individual behaviors like not wearing a safety helmet, sleeping on duty, leaving one's post, and violating operating procedures.

[0003] However, existing unsafe behavior identification technologies primarily focus on isolated, instantaneous atomic actions (such as not wearing a helmet or falling), lacking the ability to deeply understand and analyze continuous, interconnected "behavioral chains." In reality, many safety accidents are caused by a series of seemingly normal but risky sequences of actions. For example, reversing the order of certain operational steps, omitting critical steps, or performing certain combinations of actions in specific environments can all lead to safety accidents. These complex behavioral sequence patterns cannot be detected and prevented through simple single-behavior identification.

[0004] While existing coal mine safety risk monitoring and early warning systems can monitor and provide early warnings about the coal mine production environment, they primarily focus on monitoring environmental parameters and equipment status, lacking sufficient ability to analyze personnel behavior sequences and predict risks. Furthermore, these systems often rely on historical data and simple statistical models for risk assessment, lacking a deep understanding and analysis of the spatiotemporal correlations of behavioral sequences. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, system, equipment, and medium for intelligent identification of unsafe behaviors of coal mine workers to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for intelligent identification of unsafe behaviors of coal mine workers, including:

[0007] S1. Based on the video data stream collected by cameras deployed underground in the coal mine, individual trajectory data including personnel ID, timestamp and spatial coordinates are obtained through a multi-target tracking algorithm;

[0008] S2. Based on individual trajectory data, the pose estimation algorithm is used to process the human image segments in the video sequence of the video data stream to obtain the human skeleton key point sequence; based on the human skeleton key point sequence, the spatiotemporal graph convolutional network is used to identify atomic behaviors and generate a spatiotemporally labeled atomic behavior sequence.

[0009] S3. By analyzing the atomic behavior sequence through a long short-term memory network based on the attention mechanism, the spatiotemporal correlation of the behavior chain is captured, and feature vectors representing the inherent patterns of the behavior chain are generated.

[0010] S4. Based on feature vectors and combined with a pre-constructed coal mine safety knowledge graph, risk reasoning is performed to obtain risk prediction results; the risk prediction results include the type of risk event and the probability of occurrence.

[0011] S5. Based on the risk prediction results, generate early warning information through a multi-level early warning strategy and send the early warning information to operators and safety supervisors.

[0012] Secondly, this application also provides a system for intelligent identification of unsafe behaviors of coal mine workers, used to implement the method described in the first aspect, the system comprising:

[0013] The real-time trajectory perception module is used to obtain individual trajectory data, including personnel ID, timestamp, and spatial coordinates, based on video data streams collected by cameras deployed underground in coal mines and through multi-target tracking algorithms.

[0014] The behavior feature modeling module is used to process human image segments in video sequences of video data streams using pose estimation algorithms based on individual trajectory data to obtain human skeleton keypoint sequences; based on the human skeleton keypoint sequences, atomic behaviors are identified through spatiotemporal graph convolutional networks to generate atomic behavior sequences with spatiotemporal labels;

[0015] The behavior chain correlation analysis module is used to analyze atomic behavior sequences through a long short-term memory network based on an attention mechanism, capture the spatiotemporal correlation of behavior chains, and generate feature vectors that characterize the inherent patterns of behavior chains.

[0016] The risk reasoning and decision-making module is used to perform risk reasoning based on feature vectors and a pre-built coal mine safety knowledge graph to obtain risk prediction results. The risk prediction results include the type of risk event and its probability of occurrence.

[0017] The intelligent early warning execution module is used to generate early warning information based on risk prediction results through multi-level early warning strategies, and send the early warning information to operators and safety supervisors.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a method for intelligent identification of unsafe behaviors of coal mine workers as described in the first aspect.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for intelligent identification of unsafe behaviors of coal mine workers as described in the first aspect.

[0020] The aforementioned method, system, equipment, and medium for intelligent identification of unsafe behaviors of coal mine workers extract individual trajectory data from underground video streams using a multi-target tracking algorithm. It then decomposes continuous actions into spatiotemporally labeled atomic behavior sequences using attitude estimation and spatiotemporal graph convolutional networks. Next, it captures the spatiotemporal correlation of the behavior chains using an attention-enhanced long short-term memory network to generate feature vectors. Finally, it combines this with a coal mine safety knowledge graph to perform risk reasoning and predict the types and probabilities of risk events. Based on a multi-level early warning strategy, it generates early warning information. This scheme achieves a leap from isolated action identification to behavior chain risk prediction. Through spatiotemporal correlation analysis of behavior sequences and domain knowledge fusion, it significantly improves the early warning capability for potential safety risks in coal mine operations. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a method for intelligent identification of unsafe behaviors of coal mine workers provided by the present invention;

[0023] Figure 2 This is a schematic diagram of the process for generating a sequence of atomic behaviors with spatiotemporal markers in an optional embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a system for intelligent identification of unsafe behaviors of coal mine workers provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] refer to Figure 1 The document presents a flowchart illustrating a method for intelligent identification of unsafe behaviors of coal mine workers, as provided in this application. The method includes the following steps:

[0027] S1. Based on the video data stream collected by cameras deployed underground in the coal mine, individual trajectory data including personnel ID, timestamp and spatial coordinates are obtained through a multi-target tracking algorithm.

[0028] Specifically, this step first completes the parameter calibration and data preprocessing of the underground camera, and then generates trajectory data through a three-level process of "target detection - feature association - trajectory optimization". First, an explosion-proof high-definition network camera conforming to national coal mine safety standards is selected. The intrinsic parameter matrix K and extrinsic parameter matrix are obtained using the Zhang Zhengyou calibration method. The extrinsic parameter matrix includes a rotation matrix R and a translation vector t, ensuring that the conversion accuracy from pixel coordinates to underground three-dimensional spatial coordinates meets the requirements of industrial monitoring. In the video data stream preprocessing stage, Gaussian filtering is used to remove image noise caused by underground dust, and histogram equalization is used to enhance the contrast between personnel and background, avoiding target loss in low-light environments.

[0029] The target detection module can utilize a lightweight deep learning model, which possesses real-time detection capabilities and can meet the concurrent monitoring needs of multiple personnel underground. The model training dataset covers images of typical underground work scenarios, annotating personnel bounding boxes and related attributes. Data augmentation techniques are employed to improve generalization ability, and the detection accuracy on the validation set meets the requirements of practical applications. During the detection process, the bounding box coordinates, confidence score, and category label for each personnel are output, and low-confidence targets are filtered to ensure the reliability of the detection results.

[0030] The multi-target tracking algorithm employs a method that integrates appearance features and motion models. Its core is to achieve continuous association of person IDs through dual feature matching. Appearance feature extraction is performed using a deep convolutional network. The detected person images are cropped, resized, and then input into the network, outputting a high-dimensional feature vector. Euclidean distance is used to calculate feature similarity, and a similarity threshold is set to distinguish between the same target. The motion model uses Kalman filtering, and its state vector can be defined as x = [x, y, w, h, v]. x ,v y ,v w ,v h ] T x and y represent the center coordinates of the bounding box, w and h represent the width and height of the bounding box, respectively, and v x and v y v represents the velocity in the x and y directions, respectively. w and v hThese represent the rates of change of width and height, respectively. In the Kalman filtering process, the process noise covariance matrix Q and the observation noise covariance matrix are set, and then the Hungarian algorithm is used to achieve the association and matching of predicted and detected boxes.

[0031] When generating trajectory data, the timestamp is synchronized with the camera clock based on the network time protocol, and the spatial coordinates are transformed using the perspective transformation formula, which is:

[0032]

[0033] In the formula, X, Y, and Z represent the three-dimensional spatial coordinates downhole, u and v represent the pixel coordinates of the image, and R... -1 K represents the inverse matrix of the rotation matrix R. -1 Let represent the inverse of the intrinsic parameter matrix K, and t represent the translation vector in the extrinsic parameter matrix. The height dimension in the spatial coordinates is periodically calibrated by the positioning system, and the final output includes trajectory data containing personnel ID, timestamp, and spatial coordinates. The continuous storage duration of a single trajectory must meet the temporal continuity requirements of behavioral chain analysis.

[0034] S2. Based on individual trajectory data, the pose estimation algorithm is used to process the human image segments in the video sequence of the video data stream to obtain the human skeleton key point sequence; based on the human skeleton key point sequence, the atomic behavior is identified through the spatiotemporal graph convolutional network to generate the atomic behavior sequence with spatiotemporal labels.

[0035] Specifically, based on the individual trajectory data output by S1, image segments of people are first extracted. According to the timestamp and spatial coordinates of the trajectory, the corresponding frame of the person image is cropped from the video sequence. During cropping, the bounding box is appropriately expanded to avoid loss of edge features, and the image segments are assembled into a sequence in chronological order.

[0036] The pose estimation algorithm employs a deep learning model with multi-resolution feature processing capabilities. This model maintains high-resolution features through parallel feature branches, effectively addressing the issue of clothing occlusion by personnel underground. The model input is a resized RGB image fragment, and the output consists of key points of the human skeleton. These key points conform to general dataset standards, covering key locations of the head, torso, and limbs. The output for each key point includes coordinates and a confidence score. For key points with low confidence, linear interpolation is used for completion; the completion formula is as follows:

[0037]

[0038] In the formula, x t and y t Let conf represent the padded coordinates of the missing keypoint in frame t. t-1 and conf t+1Let x represent the confidence scores of the keypoints corresponding to the (t-1)th and (t+1)th frames, respectively. t-1 and y t-1 Let x_{t+1} and y_{t+1} represent the coordinates of the keypoints in frame t-1, respectively, and let x_{t+1} and y_{t+1} represent the coordinates of the keypoints in frame t+1, respectively. This method ensures the integrity of the skeletal keypoint sequence, and the final generated skeletal keypoint sequence is a tensor containing the time dimension, the keypoint count dimension, and the coordinate dimension.

[0039] Atomic behavior recognition employs a Spatiotemporal Graph Convolutional Network (ST-GCN), first constructing a graph structure. The nodes of the graph represent key points of the human skeleton, and edges are defined based on human anatomical structure, i.e., the connections between nodes are determined according to the connectivity relationships between different parts of the body. The adjacency matrix assigns weights based on the connection strength between nodes. The ST-GCN network structure contains multiple spatiotemporal convolutional blocks, each consisting of a spatial graph convolutional layer, a temporal convolutional layer, a batch normalization layer, and an activation function. The spatial graph convolutional layers can use Chebyshev multinomial expansion to aggregate graph features, while the temporal convolutional layers can use one-dimensional convolution to capture action changes over time. Behavioral features are extracted progressively through multiple convolutional blocks.

[0040] The model training dataset covers common atomic behaviors in coal mines, with sufficient samples collected for each behavior category. Each sample is a skeletal keypoint sequence. Training is performed using the cross-entropy loss function and an adaptive optimizer until the recognition accuracy on the validation set meets the requirements of practical applications. During atomic behavior recognition, the skeletal keypoint sequence is input into the trained ST-GCN, which outputs the probability distribution of behavior categories. The category with the highest probability is taken as the recognition result. Simultaneously, spatiotemporal markers are added to the trajectory data of S1, ultimately generating a spatiotemporally marked atomic behavior sequence. The sequence contains the personnel ID, behavior category, and corresponding spatiotemporal information.

[0041] S3. By analyzing the atomic behavior sequence through a long short-term memory network based on the attention mechanism, the spatiotemporal correlation of the behavior chain is captured, and feature vectors representing the inherent patterns of the behavior chain are generated.

[0042] Specifically, this step first preprocesses the atomic behavior sequence, and then uses an Attention-LSTM network to capture the spatiotemporal correlation of the behavior chain. First, the input format of the atomic behavior sequence is standardized. The sequence is truncated to a fixed length, and each atomic behavior is encoded into a high-dimensional vector through an embedding layer. The behavior category is converted into a vector form using one-hot encoding. Numerical features in the spatiotemporal markers, such as time difference (the time interval between the current behavior and the previous behavior) and spatial displacement (the spatial coordinate difference between the current behavior and the previous behavior), are standardized to a specified interval. Then, the category encoding vector is concatenated with the standardized numerical feature vector, and finally mapped to a fixed-dimensional embedding vector through a fully connected layer, ultimately forming the input matrix.

[0043] The Attention-LSTM network structure consists of LSTM layers and self-attention layers. The hidden layers of the LSTM layers have appropriate dimensions, and the forget gate, input gate, and output gate all use the sigmoid activation function. Cell state updates use the tanh activation function, and dropout is used to prevent overfitting. The computation process of the LSTM layer includes the following four key steps:

[0044] 1) Forget gate calculation: f t =σ(W f ·[h t-1 ,s t ]+b f In the formula, f t This represents the output of the forget gate, σ represents the sigmoid activation function, and W... f The weight matrix h represents the weight of the forget gate. t-1 s represents the hidden state at the previous moment. t Let b represent the input embedding vector at the current time step. f This represents the bias vector of the forget gate.

[0045] 2) Input gate calculation: i t =σ(W i ·[h t-1 ,s t ]+b i ), In both formulas, i t Indicates the input gate output, W i Let b represent the weight matrix of the input gate. i This represents the bias vector of the input gate; Represents the candidate cell state, tanh represents the hyperbolic tangent activation function, and W... C The weight matrix representing the candidate cell state, b C The bias vector representing the candidate cell state.

[0046] 3) Cell state update: In the formula, C t Represents the current cell state, ⊙ represents element-wise multiplication, and C t-1 This indicates the cell state at the previous moment.

[0047] 4) Output gate calculation: o t =σ(W o ·[h t-1 ,s t ]+b o ), h t =o t ⊙tanh(C t In both formulas, o tIndicates the output gate output, W o Let b represent the weight matrix of the output gate. o h represents the bias vector of the output gate. t This indicates the hidden state at the current moment.

[0048] Through the above calculations, a hidden state matrix containing the temporal features of each atomic behavior is obtained, with each row of the matrix corresponding to the hidden state of an atomic behavior.

[0049] The self-attention layer is used to highlight the weights of key behaviors in the behavioral chain. The calculation process includes four steps:

[0050] 1) Generate query (Q), key (K), and value (V): Q = H·W q K = H·W k V = H·W v In the formula, Q represents the query matrix, K represents the key matrix, V represents the value matrix, H represents the hidden state matrix output by the LSTM layer, and W represents the hidden state matrix output by the LSTM layer. q W k W v These represent the trainable weight matrices for query, key, and value, respectively.

[0051] 2) Calculate attention score: In the formula, Score represents the attention score matrix, and K T d represents the transpose of the bond matrix K. k This represents the dimension of the query matrix Q and the key matrix K, which is used to scale the score to avoid gradient vanishing.

[0052] 3) Weight normalization: α t =softmax(Score[t,:]); In the formula, α t Let represent the attention weights for the t-th hidden state, softmax represent the normalization function, and Score[t,:] represent the score value of the t-th row of the attention score matrix.

[0053] 4) Weighted summation: In the formula, h att This represents the attention feature vector, where N represents the length of the atomic behavior sequence, and h... t This represents the hidden state of the t-th atomic behavior.

[0054] Finally, the attention feature vector is input into the fully connected layer, and a fixed-dimensional feature vector is output through multi-layer mapping. This vector contains information such as the temporal order, spatial correlation, and key behavior weights of the behavior chain. The model is trained using a dataset with behavior chain labels, where the labels reflect the correspondence between the behavior chain and the risk event. The mean squared error loss function and an optimizer with weight decay are used for training until the similarity between the feature vector and the label vector on the validation set meets the requirements.

[0055] S4. Based on feature vectors, risk reasoning is performed in conjunction with a pre-constructed coal mine safety knowledge graph to obtain risk prediction results; the risk prediction results include the type of risk event and the probability of occurrence.

[0056] Specifically, the first step is to pre-construct a coal mine safety knowledge graph. The construction process includes three steps: "ontology modeling - data import - relation annotation". The ontology layer defines core entities and relation types. Core entities can cover five categories: atomic behavior, risk event, work scenario, equipment type, and personnel role. Relation types can cover four categories: causing, occurring in, involving, and needing to be satisfied. For example, "atomic behavior - causing - risk event" represents the causal relationship between atomic behavior and risk event, and "risk event - occurring in - work scenario" represents the spatial association between risk event and work scenario. The data layer is constructed by fusing structured and unstructured data. Structured data comes from official documents such as coal mine safety regulations and historical accident reports, while unstructured data comes from interview records with safety experts. Entities and relations are extracted using natural language processing tools. The knowledge graph is stored in a graph database, and node attributes include entity ID, name, and confidence level.

[0057] Risk reasoning is divided into three stages: preliminary candidate generation, graph verification, and probability calculation. In the first stage, the feature vector output by S3 is input into a multilayer perceptron (MLP). The MLP contains multiple hidden layers, which use the ReLU activation function, and the output layer uses the softmax activation function to obtain the preliminary probabilities of various risk events. The events with the highest probabilities are selected as candidate risk events.

[0058] The second stage involves verifying the rationality of candidate events based on knowledge graphs. Taking a candidate risk event as an example, the graph database query language is used to retrieve the set of atomic behaviors in the graph that have a "cause" relationship with the event. The atomic behaviors in the S3 behavior chain are compared to see if they contain sufficient elements of this set. If they do, it is preliminarily determined that the behavior chain is related to the candidate event. At the same time, the set of work scenarios in the graph that have an "occurrence" relationship with the event is retrieved, and the work scenarios in the S1 trajectory data are compared to see if they match. If they match, the rationality of the candidate event is further confirmed.

[0059] The third stage involves calculating the final probability of the risk event using a weighted product model. The calculation formula is as follows:

[0060]

[0061] In the formula, P(E) represents the final probability of risk event E occurring, k represents the number of atomic behaviors related to event E in the behavioral chain, and P(a i →E) represents atomic behavior a iThe confidence level leading to event E (this confidence level comes from the knowledge graph node attributes), P(S→E) represents the confidence level of event E occurring in job scenario S (this confidence level also comes from the knowledge graph node attributes), P MLP (E) represents the initial probability of event E output by the multilayer perceptron, a i Let S represent the i-th atomic behavior related to event E in the behavior chain, and let S represent the job scenario corresponding to the trajectory data.

[0062] The final output is the risk prediction result, which may include personnel ID, risk event type, probability of occurrence, and reasoning basis. The reasoning basis can explain the supporting relationship between the key atomic behaviors in the behavior chain and the work scenario for the risk event.

[0063] S5. Based on the risk prediction results, generate early warning information through a multi-level early warning strategy and send the early warning information to operators and safety supervisors.

[0064] Specifically, a four-level early warning strategy is first formulated. The classification of early warning levels is based on a two-dimensional matrix of "severity-probability of occurrence" of risk events: Level 1 warning (red) corresponds to a risk event with a severity of "major accident", Level 2 warning (orange) corresponds to a risk event with a severity of "relatively serious accident", Level 3 warning (yellow) corresponds to a risk event with a severity of "general accident", and Level 4 warning (blue) corresponds to a situation with a severity of "potential risk". An early warning is not triggered when the probability of occurrence is too low. The severity classification of the early warning levels can be based on the relevant national regulations on reporting and handling coal mine production safety accidents.

[0065] The response measures and information transmission methods differ for each warning level. A Level 1 warning triggers the underground audible and visual alarm equipment and simultaneously sends instructions to the workers' smart wearable devices via the underground communication network. The wearable devices convey emergency evacuation information to the workers through vibration and voice prompts. At the same time, the warning information is sent to the monitoring platform of the surface safety supervision center. The platform displays key information to the supervisors through pop-ups and audible and visual prompts. The key information includes personnel ID, risk location, event type, and evacuation route. Supervisors must confirm receipt of the information within a specified time. If no confirmation is received, the notification mechanism of the superior emergency command center is automatically triggered.

[0066] The Level 2 early warning system adjusts the alarm mode of the underground audible and visual alarm equipment. Smart wearable devices convey instructions to workers to stop operation and check protective equipment through text and voice prompts. The monitoring platform also sends information to supervisors through pop-up prompts. Supervisors confirm within the specified time. If no confirmation is made, a notification is sent to the team leader through the SMS gateway.

[0067] Level 3 early warning only sends text alerts to workers via smart wearable devices. At the same time, the alert information is also sent to the team leader's smart terminal. The smart terminal supports offline storage of the alert information. The team leader must provide feedback on the rectification status within the specified time. Failure to provide feedback will trigger an upgrade of the alert level.

[0068] Level 4 warnings only record information in designated modules of the regulatory platform. The records are stored in categories according to personnel IDs and are pushed to the mobile application of the corresponding regional safety officers. The mobile application supports viewing video clips of the behavior chain. The safety officers complete on-site verification and record the rectification status within the specified time.

[0069] The technical transmission of early warning information can employ a dual-mode communication method: industrial Ethernet covering the main underground working areas, and short-range wireless communication technology covering Ethernet blind spots; communication between the surface and underground is achieved via fiber optics. Early warning information can include a unique identifier, generation time, personnel ID, risk details, response measures, and feedback timeframe. The unique identifier ensures traceability, and the information is stored in a database for subsequent retrieval. A closed-loop early warning feedback mechanism is also established: after operators and supervisors provide feedback, the system automatically records the feedback time and content. After rectification is completed, the safety officer uploads rectification documentation to the platform. Once the system confirms the documentation is satisfactory, it archives the early warning event, forming a complete management closed loop of "early warning - feedback - rectification - archiving."

[0070] The aforementioned method for intelligent identification of unsafe behaviors of coal mine workers extracts individual trajectory data from underground video streams using a multi-target tracking algorithm. It then decomposes continuous actions into spatiotemporally labeled atomic behavior sequences using pose estimation and a spatiotemporally graph convolutional network. Next, an attention-enhanced long short-term memory network captures the spatiotemporal correlation of the behavior chains to generate feature vectors. Finally, it combines a coal mine safety knowledge graph to perform risk reasoning and predict the types and probabilities of risk events. Based on a multi-level early warning strategy, it generates early warning information. This scheme achieves a leap from isolated action identification to behavior chain risk prediction. Through spatiotemporal correlation analysis of behavior sequences and domain knowledge fusion, it significantly improves the early warning capability for potential safety risks in coal mine operations.

[0071] refer to Figure 2 In one optional embodiment, based on the sequence of key points in a human skeleton, atomic behaviors are identified through a spatiotemporal graph convolutional network to generate a sequence of atomic behaviors with spatiotemporal labels, including the following steps:

[0072] S11. Construct a spatiotemporal graph model based on the sequence of key points of the human skeleton to obtain spatiotemporal graph structure data; wherein, the spatiotemporal graph model includes nodes, spatial edges and temporal edges, nodes represent human joints, spatial edges connect adjacent anatomical joints in the same frame of the video data stream, and temporal edges connect the same joints in adjacent frames of the video data stream.

[0073] Specifically, this step is based on the sequence of key points of the human skeleton, and constructs a spatiotemporal graph model according to the three-level logic of "node definition - edge structure design - data format encapsulation", so that the model can simultaneously represent the spatial correlation and inter-frame temporal correlation of human joints.

[0074] The nodes of the spatiotemporal graph directly map to key points of the human skeleton. It adopts a common dataset standard consistent with the pose estimation described above (covering 17 key joints in the head, torso, and limbs, such as nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left eye, right eye, left ear, and right ear). The feature vector of each node consists of the coordinate information of the corresponding joint, specifically the two-dimensional coordinates (x, y) of the joint in the image coordinate system. If there is interpolation to complete the coordinates as described above, the completion marker is marked in the node feature to distinguish between the original coordinates and the completed coordinates during subsequent feature extraction, thus avoiding error accumulation.

[0075] When constructing spatial edges, the anatomical adjacency relationships are strictly followed, meaning only joints with physiological structural connections within the same frame are connected. For example, left shoulder node and left elbow node, left elbow node and left wrist node (upper limb bone connection), left hip node and left knee node, left knee node and left ankle node (lower limb bone connection), nose node and left eye node, nose node and right eye node (local head connection), etc., defining a total of 19 fixed spatial edge relationships. For nodes without direct anatomical connections (such as left shoulder and right knee), no spatial edges are constructed. The weights of spatial edges are set using a "distance decay rule": let d be the Euclidean distance between node i and node j within the same frame. ij Spatial edge weight w s (i,j)=1 / (1+d ij This method strengthens the spatial association weight of adjacent nodes and weakens the interference of distant nodes.

[0076] Temporal edges are constructed for adjacent frames in a video sequence, connecting keypoints with the same number in different frames (e.g., the left shoulder node in frame t and the left shoulder node in frame t+1), thus representing the motion trajectory of the same keypoint in the time dimension. The weights of the temporal edges are set based on the motion amplitude of the keypoints between frames: let the coordinates of node k in frame t be (x... t,k ,y t,k The coordinates of node k in the (t+1)th frame are (x... t+1,k ,y t+1,k The distance traveled was... Time edge weight w t (k,t)=exp(-d t,k The smaller the amplitude of motion (d) t,k The smaller the value, the closer the time edge weight is to 1; conversely, the larger the value, the lower the weight, thus highlighting the continuity of the joint motion.

[0077] The final generated spatiotemporal graph structure data contains three core pieces of information: first, the node feature matrix F, with dimensions T×N×2 (T is the number of frames in the video sequence, N is the number of key points, and 2 corresponds to the x and y coordinates), which is expanded to T×N×3 if completion markers are included; second, the spatial adjacency matrix A. s The dimension is N×N; when nodes i and j have spatial edges, A s [i][j]=w s (i,j); when there are no spatial edges, A s [i][j] = 0, and this matrix remains fixed across all frames (due to unchanged anatomical relationships); thirdly, the temporal adjacency matrix A t The dimension is T×N×N, A t [t][k][k]=w t (k,t) (only diagonal elements are non-zero, corresponding to inter-frame connections at the same keypoint), and all off-diagonal elements are 0. These three elements together constitute the input data for the spatiotemporal graph model, ensuring that subsequent feature extraction can simultaneously capture spatial structure and temporal dynamics.

[0078] S12. Based on the spatiotemporal graph structure data, features are extracted hierarchically through a spatiotemporal graph convolutional network to obtain spatiotemporal fusion features.

[0079] Specifically, this step is based on the spatiotemporal graph structure data generated by S11. It uses multi-layer spatiotemporal convolutional blocks to gradually extract and fuse features. The network structure takes into account both the local aggregation of spatial features and the dynamic capture of temporal features.

[0080] The overall architecture of the spatiotemporal graph convolutional network can contain four cascaded spatiotemporal convolutional blocks. Each convolutional block consists of a "spatial graph convolutional layer - temporal convolutional layer - batch normalization layer - ReLU activation function". The parameters of each layer are set according to the feature dimension increment rule to ensure the evolution from low-dimensional local features to high-dimensional global features.

[0081] Firstly, regarding the implementation of spatial graph convolutional layers, Chebyshev multinomial expansion can be used to achieve feature aggregation on the graph. This method avoids the strict dependence of traditional graph convolution on graph structure and adapts to the fluctuations in node correlation caused by changes in the posture of downhole personnel. The calculation formula for spatial graph convolution is:

[0082]

[0083] In the formula, Y s This represents the output features of the spatial graph convolutional layer, where K represents the order of the Chebyshev polynomial (usually set to 2 to balance computational complexity and feature capture capability). Let T represent the trainable weight matrix corresponding to the k-th order polynomial. k (·) denotes the k-th order Chebyshev polynomial. This represents the normalized graph Laplacian matrix (derived from the spatial adjacency matrix A in S11). s Calculations show that Where I is the identity matrix, and D is the matrix of A. s The degree matrix, D[i][i] = ∑ j A s [i][j]), where X represents the input features of this layer (the first block input is the node feature matrix F of S11, and the subsequent blocks input are the output features of the previous convolutional block). Through this formula, the spatial graph convolutional layer can aggregate the features of each node with the features of adjacent nodes to extract the spatial structural features of human posture, such as the arm bending angle and the torso tilt state.

[0084] Next is the implementation of the temporal convolutional layer. A one-dimensional convolutional kernel can be used to filter the output features of the spatial graph convolutional layer in the temporal dimension, capturing the temporal dynamic features of joint motion, such as arm swing speed and leg step frequency. The input to the temporal convolutional layer is the T×N×C output of the spatial graph convolutional layer. s Features (C) s (where N is the number of output channels of the spatial graph convolutional layer), first reorganize its dimensions into T×(N×C). s Then, a convolution operation is performed using a one-dimensional convolution kernel. The calculation formula is as follows:

[0085] Y t =Conv1d(Y s ′,W t stride = 1, padding = P)

[0086] In the formula, Conv1d represents a 1D convolution operation, Y t Y represents the output feature of the temporal convolutional layer. s ′ represents the recombined input feature (dimension T×(N×C)). s W t This represents a one-dimensional convolutional kernel (e.g., size can be set to 3 to ensure coverage of a three-frame time window and capture short-term motion trends), stride=1 indicates a convolution stride of 1, and padding=P indicates padding length (e.g., set to 1 to ensure the time dimension T remains unchanged after convolution). The number of output channels C of the temporal convolutional layer. t It can be set to C s It is twice as large as that of the time and space features, achieving dimensional matching and fusion of time and space features.

[0087] The subsequent processing of each spatiotemporal convolutional block follows the sequence of "batch normalization - ReLU activation". The batch normalization layer accelerates network training convergence and avoids gradient vanishing by standardizing the features (mean = 0, variance = 1). The ReLU activation function introduces the model's nonlinearity through a nonlinear transformation (f(x) = max(0,x)) to adapt to the feature distribution of complex poses of underground personnel. After processing by the four spatiotemporal convolutional blocks in sequence, the final output spatiotemporal fusion feature dimension is T×N×C. final (C final The number of output channels for the last spatiotemporal convolutional block (usually set to 256) is a feature that includes both spatial structural information of human posture and temporal dynamic information of joint movement, providing comprehensive feature support for subsequent atomic behavior classification.

[0088] S13. Based on spatiotemporal fusion features, the confidence level of each frame action in the video data stream belonging to a predefined atomic behavior category is calculated through a fully connected layer and a Softmax classifier, thus obtaining the probability distribution of the atomic behavior category.

[0089] Specifically, this step, based on the spatiotemporal fusion features output by S12, obtains the probability distribution of atomic behavior categories corresponding to each frame action through the process of "feature dimensionality reduction - category mapping - probability normalization". The core is to realize the mapping of high-dimensional spatiotemporal features to a finite number of atomic behavior categories.

[0090] First, the spatiotemporal fusion features are subjected to global pooling processing, and T×N×C is... final The features are averaged pooled along the time dimension to obtain N×C. final The global spatial features are then subjected to max pooling along the node dimension to obtain a 1×C feature. final The global feature vector. The purpose of global pooling is to eliminate the difference between the number of nodes within a frame and the time length between frames, ensuring that the feature dimensions of the input fully connected layer are consistent and avoiding classification errors caused by changes in the length of the person's pose.

[0091] The implementation of the fully connected layer is divided into two levels. The first level of the fully connected layer will connect 1×C final Map the global feature vector to 1×C hid Hidden layer features (C hid The hidden layer dimension can usually be set to C. final The weight matrix W of this layer is twice that of the previous layer. fc1 Dimension C hid ×C final Bias vector b fc1 Dimension is 1×C hid The calculation process is as follows:

[0092] F hid =W fc1 ·Fglobal +b fc1

[0093] In the formula, F hid F represents the output hidden feature of the first fully connected layer. global This represents the feature vector after global pooling.

[0094] The first fully connected layer is followed by a ReLU activation function to enhance the model's ability to express complex features. The second fully connected layer uses a 1×C... hid The hidden features are mapped to 1×C class Category features (C) class Given a predefined number of atomic behavior categories, such as the 12 categories mentioned above, including "not wearing a safety helmet" and "working while bending over," then C... class =12), the weight matrix W of this layer fc2 Dimension C class ×C hid Bias vector b fc2 Dimension is 1×C class The calculation process is as follows:

[0095] F class =W fc2 ·F hid +b fc2

[0096] In the formula, F class This represents the output category feature of the second-level fully connected layer, which directly corresponds to the raw score of each type of atomic behavior.

[0097] The Softmax classifier is used to convert category features into a probability distribution, ensuring that the sum of the probabilities of each action in each frame is 1, facilitating subsequent action label determination. The formula for calculating Softmax is:

[0098]

[0099] In the formula, P(c) represents the confidence level (probability) of the c-th type of atomic behavior, and F class (c) represents the raw score of the c-th behavior output by the second-level fully connected layer, and c' represents all atomic behavior categories. After calculation using this formula, each frame's action will obtain a 1×C dimension score. class The probability distribution of atomic behavior categories, for example, if the probability distribution of a certain frame is [0.02, 0.95, 0.03, ..., 0.00] (corresponding to 12 behavior categories), it indicates that the action in that frame belongs to the second behavior category with the highest confidence.

[0100] In practice, a confidence threshold (usually 0.5) is set for the probability distribution of each frame. If the highest confidence is lower than the threshold, the frame is determined to be an "unidentified behavior" (which needs to be corrected later by the dynamic time warping algorithm of S14) to avoid misclassification caused by feature ambiguity. If the highest confidence is higher than the threshold, it is temporarily designated as the behavior label of the frame to provide an initial basis for sequence alignment in S14.

[0101] S14. Based on the probability distribution of atomic behavior categories, and combined with the dynamic time warping algorithm, align the behavior labels of consecutive frames in the video data stream to generate an initial atomic behavior sequence.

[0102] Specifically, this step addresses the temporal offset problem of consecutive frame action labels output by S13 using the Dynamic Time Warping (DTW) algorithm (such as differences in the duration of the same action in different frames, misclassification noise in local frames, etc.), achieving smooth alignment and sequence generation of action labels. The DTW algorithm is implemented in three steps: "distance matrix construction - optimal path search - label merging". The core is to find the optimal matching path between two frame sequences to ensure the temporal continuity of action labels.

[0103] First, the distance matrix is ​​constructed. Based on the M consecutive frames of action labels output by S13 (denoted as sequence A, length M, A[i] representing the action label of frame i), K consecutive frames (K can be set to 5-10, covering the shortest duration of a complete action) are selected as reference sequence B (length K, B[j] representing the action label of frame j in the reference sequence). The frame-level distance matrix D between the two sequences is calculated, with dimensions M×K. The frame-level distance is calculated using "category difference distance": if A[i] and B[j] belong to the same action category, then the distance d(i,j) = 0; if they belong to different categories, then d(i,j) = 1 - min(P A (i,A[i]),P B (j,B[j])), where P A (i, A[i]) represents the confidence level of the label corresponding to the i-th frame of sequence A, P B (j, B[j]) represents the confidence level of the label corresponding to the j-th frame of sequence B. This distance definition considers both the consistency of behavior categories and adjusts the distance weights through confidence levels to reduce interference from misclassified frames with high confidence levels.

[0104] The next step is to search for the optimal path. Dynamic programming can be used to find the optimal path from the top left corner (i=1, j=1) to the bottom right corner (i=M, j=K) of the distance matrix D. The path must satisfy the "monotonically increasing" constraint (i.e., i and j can only increase or remain unchanged, ensuring the time order is not reversed), and the cumulative distance of the path must be minimized. The formula for calculating the cumulative distance matrix is:

[0105] Dacc (i,j)=d(i,j)+min(D acc (i-1,j),D acc (i,j-1),D acc (i-1,j-1))

[0106] In the formula, D acc (i,j) represents the cumulative distance to position (i,j), d(i,j) represents the frame-level distance to position (i,j), and min(·) represents taking the minimum cumulative distance from the previous position (left, top, top left). The initial condition is D. acc (1,1)=d(1,1), D acc (i,1)=D acc (i-1,1)+d(i,1)(first column), D acc (1,j)=D acc (1,j-1)+d(1,j)(first row). After calculating the complete cumulative distance matrix using this formula, from D... acc (M,K) backtracks to D acc (1,1) can be used to obtain the optimal matching path, where (i,j) on the path corresponds to the frame-level matching relationship between sequence A and sequence B.

[0107] Finally, the behavior labels are merged and the initial atomic behavior sequence is generated. Based on the optimal matching path, consecutive frames in sequence A that match the same frame in sequence B are merged into a single behavior segment. For example, if more than 80% of the frames in the merged segment belong to the same behavior category, that category is used as the behavior label for that segment. If multiple categories are mixed and there is no dominant category (e.g., each category accounts for less than 60%), the behavior labels of the corresponding frames in sequence B are used for correction. After merging, duplicate consecutive behavior labels are removed in chronological order (e.g., "not wearing a helmet - not wearing a helmet" is merged into a single "not wearing a helmet" label), and short behavior segments with a duration of less than 0.5 seconds are deleted (determined as noise), ultimately generating the initial atomic behavior sequence. The sequence can be formatted as [{"frame range":[t1,t2],"behavior label":"category 1","average confidence":p1},{"frame range":[t3,t4],"behavior label":"category 2","average confidence":p2},...], where t1-t2 represent the start and end frames of the behavior segment, and p1 represents the average confidence of all frames within the segment.

[0108] S15. Based on the initial atomic behavior sequence, associate the corresponding timestamp, camera position and environmental parameters to generate an atomic behavior sequence with spatiotemporal markers.

[0109] Specifically, this step associates the initial atomic behavior sequence generated in S14 with multi-source auxiliary information, supplementing it with timestamps, spatial locations, and environmental parameters to form a complete atomic behavior sequence with spatiotemporal tags, providing contextual information for subsequent behavior chain analysis. The association process follows the logic of "time alignment - spatial mapping - environment matching" to ensure that each tag information corresponds precisely to the time range of the atomic behavior.

[0110] First, the timestamps are associated. Based on the individual trajectory data generated by multi-target tracking in S1, the timestamp sequence is extracted from the trajectory data (each timestamp corresponds to one frame of video data). According to the frame range [t_start, t_end] of each behavior segment in the initial atomic behavior sequence of S14, the corresponding timestamp range [T_start, T_end] is found (T_start is the timestamp of the t_start frame, and T_end is the timestamp of the t_end frame), and the duration ΔT = T_end - T_start of the behavior segment is calculated. The timestamp range and duration are used as "time stamps" to supplement the corresponding segments of the initial atomic behavior sequence. The association of timestamps ensures synchronization with the trajectory timestamps of S1 (using the same network time protocol), avoiding misalignment between behavior and scene caused by time deviation.

[0111] Secondly, there is the association of spatial locations. Spatial location information includes two categories: camera locations and real-time personnel locations. Camera location information comes from the deployment files of underground cameras in the coal mine. The files record the unique number, installation coordinates (underground 3D spatial coordinates), and monitoring coverage area of ​​each camera. Based on the camera number of the video data stream collected in S1, the corresponding camera location can be directly obtained as the device location marker. The real-time personnel location information comes from the spatial coordinates of individual trajectory data in S1. Based on the frame range [t_start, t_end] of the behavior segment, the average value of the personnel's spatial coordinates within that frame segment is calculated as the personnel location marker (if the coordinates within the frame segment fluctuate greatly, median filtering is used to remove outliers). The two types of spatial location information are merged into a "spatial marker" and added to the atomic behavior sequence.

[0112] Finally, the environmental parameters are correlated. These parameters come from an underground environmental monitoring sensor network (such as gas sensors, temperature sensors, and humidity sensors). The sensor network is divided into regions corresponding to the camera monitoring areas, with at least two environmental sensors deployed within each camera monitoring area to ensure data reliability. Based on the timestamp range [T_start, T_end] of the S14 behavior segment and the camera monitoring area in the spatial marker, the average environmental parameters (such as gas concentration, temperature, and humidity) of the sensors within that area during the [T_start, T_end] time period are retrieved. If a parameter is missing (e.g., due to temporary sensor failure), it is supplemented using the average parameter from adjacent time periods (e.g., the first 10 seconds and the last 10 seconds). These environmental parameters are then added to the atomic behavior sequence as environmental markers.

[0113] Through the above association process, the final generated atomic behavior sequence with spatiotemporal labels fully contains the category information, spatiotemporal context and environmental background of the atomic behavior, which can be directly used as input data for Attention-LSTM network to analyze the behavior chain, providing comprehensive behavioral feature support for subsequent risk reasoning.

[0114] In this embodiment, by constructing a spatiotemporal graph model containing nodes, spatial edges, and temporal edges, and combining dynamic time warping to align behavior labels with associated spatiotemporal environment parameters, the spatiotemporal features of atomic behaviors are accurately extracted, generating reliable atomic behavior sequences with spatiotemporal tags. This solves the limitations of traditional single behavior identification and provides high-quality basic data for subsequent behavior chain analysis.

[0115] In one optional embodiment, based on spatiotemporal graph structure data, features are extracted hierarchically through a spatiotemporal graph convolutional network to obtain spatiotemporal fusion features, including the following steps:

[0116] S21. Based on the spatiotemporal graph structure data, perform graph convolution operations based on anatomical constraints in the spatial dimension to obtain a spatial feature map.

[0117] Specifically, this step uses the spatiotemporal graph structure data generated by S11 as input. The core is to extract the spatial structural features of human joints (such as relative limb positions and joint flexion states) through spatial graph convolution incorporating anatomical constraints, ensuring that the feature extraction conforms to the physiological movement patterns of the human body. The input data consists of two parts: first, a node feature matrix F (dimension T×N×2, where T is the number of frames, N is the number of joints, and 2 represents the x / y coordinates); and second, a spatial adjacency matrix A. s (Dimensions N×N, constructed based on anatomical adjacency relationships).

[0118] First, we strengthen the role of anatomical constraints in graph convolution. Spatial adjacency matrix A sThe spatial association range has been defined by "connecting only anatomically adjacent joints" (e.g., left shoulder-left elbow, left hip-left knee, etc.). The graph convolution operation in this step implements "constrained feature aggregation" based on this matrix, meaning that the features of each joint are only fused with the features of its adjacent joints, avoiding feature interference from non-anatomically related nodes (e.g., left shoulder-right knee). Specifically, the Chebyshev polynomial expansion graph convolution method can be used, with the calculation formula as follows:

[0119]

[0120] In the formula, F s The output spatial feature map (dimension T×N×C) s C s (where K is the number of output channels for spatial graph convolution), K represents the order of the Chebyshev polynomial, and W is the number of output channels for spatial graph convolution. k This represents the trainable weight matrix (dimension C) corresponding to the k-th order polynomial. s ×2, mapping the input 2D coordinate features to C s 2D spatial features), T k (·) denotes the k-th order Chebyshev polynomial (recursively defined as T0(x) = 1, T1(x) = x, T k (x)=2xT k-1 (x)-T k-2 (x)), The normalized graph Laplace matrix (derived from the spatial adjacency matrix A) represents the normalized graph Laplace matrix. s Calculated, it is the core carrier of anatomical constraints), F node This represents the input node feature matrix (dimension T×N×2, taken from the spatiotemporal graph structure data of S11).

[0121] Among them, the normalized graph Laplacian matrix The calculation is strictly related to anatomical constraints, and the formula is:

[0122]

[0123] In the formula, I represents the N×N identity matrix, and D represents the spatial adjacency matrix A. s The degree matrix (dimension N×N, That is, the sum of the number of adjacent nodes of the i-th joint), D -1 / 2 This represents the inverse square root of the degree matrix (used to normalize the adjacency matrix and avoid feature aggregation bias caused by differences in the number of adjacent nodes at different joints). Using this matrix, graph convolution operations can only transfer features between anatomically adjacent joints. For example, the features of the left elbow node can only be aggregated with the features of the left shoulder and left wrist nodes, ensuring that the extracted spatial features conform to the movement patterns of human physiological structures.

[0124] The specific implementation process of spatial graph convolution is as follows: First, the node feature matrix F node The time dimension is split into T N×2 single-frame node features. The above graph convolution calculation is performed on each single-frame feature to obtain T N×C features. s The single-frame spatial features are then reassembled in chronological order to form a T×N×C spatial feature set. s Spatial feature map F s This feature map has captured the spatial structural information of human posture (such as arm extension / bending, torso upright / tilted, etc.), laying the foundation for subsequent dynamic feature extraction in the time dimension.

[0125] S22. Based on the spatial feature map, perform causal convolution operation in the time dimension to obtain the spatiotemporal feature map.

[0126] Specifically, this step uses the spatial feature map F output by S21. s Using causal convolution operations as input, it captures the temporal dynamic features of joint motion (such as arm swing speed, leg step frequency, etc.). Its core advantage lies in avoiding reliance on "future frame" features and ensuring that feature extraction is real-time (meeting the real-time early warning requirements of underground coal mine safety monitoring).

[0127] First, the spatial feature map is reshaped in dimensions. Spatial feature map F s The dimensions are T×N×C s This is converted into a time-dimension-first feature format, i.e., T×(N×C) is obtained through dimension permutation. s The characteristic matrix F s ′, where N×C s This represents the sum of spatial features of all relevant nodes within a single frame, and T represents the length of the time series. This reorganization method ensures that causal convolution can completely capture the motion changes of all relevant nodes along the time axis.

[0128] The core design of causal convolution is to "utilize only the features of the current frame and historical frames", and its calculation formula is as follows:

[0129] F st =Conv1d causal (F s ′,W t stride = 1, padding = P)

[0130] In the formula, F st The output spatiotemporal feature map (dimension T×(N×C)) represents the spatiotemporal feature map. t ), C t (Number of output channels for causal convolution), Conv1d causal W represents a causal one-dimensional convolution operation. t Represents the causal convolution kernel (dimension K)t ×(N×C s )×(N×C t ), K t The convolution kernel's time window size can be 3-5 to cover short-term motion trends. `stride = 1` indicates a convolution stride of 1 (ensuring no information loss in the time dimension). `P` represents the causal padding length, calculated as P = (K... t -1) is used to keep the time dimension length T after convolution and only fill in the features in the direction of historical frames to avoid introducing information from future frames.

[0131] It's worth noting the difference between causal convolution and ordinary one-dimensional convolution. Ordinary one-dimensional convolution utilizes features from the current frame, historical frames, and future frames simultaneously, resulting in a "time delay" in feature extraction (waiting for future frames to be captured). Causal convolution, however, through its design of "left-padding with zeros + kernel covering only the current frame and the left-hand historical frames," ensures that when processing frame t, it only relies on features from frames 1 to t, perfectly adapting to real-time monitoring scenarios in coal mines (requiring immediate analysis of personnel behavior to avoid early warning delays). For example, when K... t When the value is 3, when processing the features of frame t, the convolution kernel only covers the features of frames t-2, t-1, and t. When filling, zero values ​​are only filled to the left of frames 1 and 2 to ensure that no information from future frames is involved.

[0132] The feature dimension transformation after causal convolution maintains the correlation with spatial features, and the output F st The dimension is T×(N×C) t ), and re-split it into T×N×C t The spatiotemporal feature map, where C t Let's set it as C s This spatiotemporal feature map is twice the size of the original map, achieving dimensional matching between temporal dynamic features and spatial structural features, ensuring a balanced weighting of the two types of features during subsequent fusion. It includes both the spatial positional relationships of human joints and the temporal trends of joint movements, such as the dynamic process of "the arm going from hanging down naturally to being raised."

[0133] S23. Based on the spatiotemporal feature map, multi-scale features are fused through hierarchical residual connections to obtain multi-scale fused features.

[0134] Specifically, this step targets the spatiotemporal feature map F output by S22. st By fusing multi-scale features of different network depths through a hierarchical residual connection structure, the gradient vanishing problem in deep network training is solved. At the same time, the complementary fusion of local detail features and global semantic features is achieved. Shallow features capture local motion details of joints (such as finger bending), while deep features capture the global semantics of human body behavior (such as limb coordination).

[0135] First, a hierarchical residual network structure is constructed, consisting of four residual blocks (the same number as the spatiotemporal convolutional blocks in S12 above, ensuring network depth matching). Each residual block employs a bottleneck structure design, comprising three steps: "1×1 convolution dimensionality reduction - 3×3 convolution feature processing - 1×1 convolution dimensionality increase," while also introducing residual connections with identity mappings. The calculation formula for a single residual block is:

[0136] F res (l)=ReLU(BN(W3·ReLU(BN(W2·F res (l-1)))))+W1·F res (l-1)

[0137] In the formula, F res (l) represents the output feature of the l-th residual block (dimension T×N×C) l ), F res (l-1) represents the input features of the (l-1)th residual block (the input of the first residual block is the spatiotemporal feature map F of S22). st W1 represents the weight matrix (dimension C) of a 1×1 up-dimensional convolution. l ×C l-1 C l Let C be the number of output channels for the l-th block. l-1 W1 represents the number of output channels in the (l-1)th block, and W2 represents the weight matrix of the 3×3 feature processing convolution (dimension C). m ×C m C m The number of intermediate channels after dimensionality reduction can be set as C. l-1 / 4 to reduce computational cost), W3 represents the weight matrix of the 1×1 dimensionality-reduced convolution (dimension C). m ×C l-1 BN stands for Batch Normalization (standardizing features to accelerate training convergence), ReLU stands for Activation Function (introducing non-linear feature representation), and "+" represents Residual Connection, which directly superimposes input features onto the output through identity mapping to avoid gradient vanishing.

[0138] The fusion logic of multi-scale features is reflected in hierarchical extraction and cross-layer connections: the first and second residual blocks (shallow layer) mainly extract local detail features (such as minute movements of joints and changes in the relative positions of local limbs), with the output feature dimension set to a low value (e.g., C1 = 128, C2 = 256); the third and fourth residual blocks (deep layer) mainly extract global semantic features (such as changes in the overall posture of the human body and multi-joint collaborative movement patterns), with the output feature dimension set to a higher value (e.g., C3 = 512, C4 = 1024). Simultaneously, a cross-layer residual connection is added between the second and fourth residual blocks to directly transfer the local features of the shallow layer to the deep layer, as shown in the formula:

[0139] Fms =F res (4) +UpSample(F res (2))

[0140] In the formula, F ms This represents multi-scale fusion features with dimensions T×N×C. final C final =C4; UpSample indicates an upsampling operation, using bilinear interpolation to upsample shallow features F res (2) The number of channels is increased from C2 to C4 to ensure compatibility with deep feature F. res (4) Dimensional consistency; "+" indicates the addition of feature elements, realizing the direct fusion of local details and global semantics.

[0141] Through this hierarchical residual connection, the multi-scale fusion features retain the ability of shallow networks to capture local motion details of joints (such as identifying head posture details when "not wearing a helmet"), and also have the ability of deep networks to understand the semantics of human body behavior as a whole (such as identifying limb coordination movements when "climbing equipment"), providing more comprehensive feature support for subsequent behavior classification.

[0142] S24. Based on multi-scale fusion features, the spatial attention mechanism is applied to enhance the features of key nodes, resulting in spatiotemporal fusion features enhanced by spatial attention.

[0143] Specifically, this step is based on the multi-scale fusion feature F output by S23. ms By dynamically allocating the weights of key features through a spatial attention mechanism, the system enhances the response of key features that play a crucial role in atomic behavior recognition (such as recognizing the waist and leg joints when "bending over"), suppresses feature interference from irrelevant key points (such as the head), and improves the discriminativeness of features.

[0144] The implementation of the spatial attention mechanism consists of two steps: “attention weight generation - feature weight enhancement”. First, global spatial pooling is performed on the multi-scale fusion features to generate global features at the key point level. Then, attention weight maps are generated through a fully connected layer and an activation function. Subsequently, the weight maps are multiplied element-wise with the original multi-scale fusion features to enhance the features at the key points.

[0145] The first step, the generation process of the attention weight map, is calculated using the following formula:

[0146] W att =Sigmoid(W5·ReLU(W4·GAP(F)) ms )))

[0147] In the formula, W attW1 represents the spatial attention weight map (dimension T×N×1, each keypoint corresponds to a weight value of 0 to 1 in each frame), W2 represents the activation function (maps the weight values ​​to the 0-1 interval, achieving weight normalization), W3 represents the weight matrix of the second-level fully connected layer (dimension N×N, maps hidden layer features to keypoint-level weights), and W4 represents the weight matrix of the first-level fully connected layer (dimension N×(N×C)). final (This maps the globally pooled features to the hidden layer), where GAP represents the global average pooling operation (for multi-scale fused features F). ms Average pooling is performed on the channel dimension, with input dimensions T×N×C. final The output dimension is T×N×1, which is the average feature value of each key point in each frame. ReLU represents the activation function and can introduce non-linear weight mapping capabilities.

[0148] The logic for allocating attention weights is as follows: W att Weights with values ​​close to 1 correspond to "critical joints," while weights with values ​​close to 0 correspond to "irrelevant joints." The magnitude of the weights is adaptively learned by the fully connected layer during training. For example, when training on behaviors such as "climbing equipment," the network automatically adjusts the weights of the hand and foot joints to 0.8–0.9 and the weights of the head joints to 0.1–0.2, ensuring that the motion features of these critical joints are captured.

[0149] The second step, the feature weighting enhancement process, is calculated using the following formula:

[0150] F final =F ms ⊙W att

[0151] In the formula, F final This represents the spatiotemporal fusion feature of the final output (dimension T×N×C). final ), ⊙ represents element-wise multiplication, which multiplies the multi-scale fused features with the attention weight map element by element. The feature values ​​of key nodes are amplified, while the feature values ​​of irrelevant nodes are reduced, thereby achieving targeted enhancement of key features.

[0152] This spatiotemporal fusion feature possesses three core pieces of information: In the spatial dimension, it ensures correct joint associations through anatomical constraint graph convolution and highlights key nodes through spatial attention; in the temporal dimension, it ensures real-time performance and captures dynamic motion trends through causal convolution; and in the multi-scale dimension, it fuses local and global features through residual connections, ultimately providing highly discriminative feature inputs for atomic behavior classification.

[0153] In this embodiment, relying on anatomically constrained spatial convolution (ensuring spatial features conform to human body structure), causal convolution (ensuring real-time performance), hierarchical residual connections (solving gradient vanishing), and spatial attention (highlighting key joints), the accuracy and relevance of spatiotemporal fusion feature extraction are optimized, the accuracy of atomic behavior recognition is enhanced, and more precise feature support is provided for behavior chain analysis.

[0154] In one optional embodiment, based on spatiotemporal graph structure data, an anatomically constrained graph convolution operation is performed in the spatial dimension to obtain a spatial feature map, including the following steps:

[0155] S31. For each key point in each frame of the video data stream, assign a learnable weight matrix based on the corresponding anatomical neighborhood using a label mapping function.

[0156] Specifically, this step assigns a unique weight matrix to each joint point in each frame of the video data stream based on anatomical neighborhood and joint type label through a trainable label mapping function. The core is to make the feature aggregation between different types of joint points anatomically targeted.

[0157] Anatomical neighborhood refers to the set of joints that are directly connected to the target joint in human anatomy (e.g., the anatomical neighborhood of the left elbow joint is the left shoulder joint and left wrist joint; the anatomical neighborhood of the left hip joint is the left knee joint and trunk joints, etc.). Based on the human anatomy atlas, a fixed list of neighborhood joints is defined in advance for each joint (e.g., the 17 common joints such as nose, shoulder, elbow, wrist, hip, knee, and ankle). This list remains consistent across all frames (because the human anatomy is fixed).

[0158] Each joint is assigned a unique type label to identify its anatomical location in the human body, such as using the number "1" to label "left shoulder joint", "2" to label "left elbow joint", and "3" to label "left wrist joint". The type label is the input to the subsequent "label mapping function" to ensure that feature aggregation rules between joints of the same type can be reused.

[0159] The label mapping function is a trainable parameterized function that takes as input a type-label pair of two key points (denoted as ). and in It is the target key point. (These are the key points within its neighborhood), and the output is a learnable weight matrix.

[0160] The dimension of the weight matrix is ​​determined by the dimensions of the input features and the target features. Let's assume the spatial coordinate features of the key points... It is a two-dimensional vector (containing x and y coordinates), i.e., the input dimension C. in =2; if the desired feature dimension after aggregation is Cout =64 (can be adjusted according to network depth), then the weight matrix The dimension is C out ×C in (i.e., 64×2).

[0161] This function learns adaptively through network training (such as backpropagation). For anatomically closely related joint pairs (such as left shoulder-left elbow), training will allow the corresponding weight matrix to more accurately capture the motion relationship features between the two; for weakly related joint pairs (such as left shoulder-right knee), since they are not in the anatomical neighborhood, the assignment of this weight matrix will not be triggered (or a very small weight will be assigned).

[0162] S32. Based on the weight matrix, aggregate the neighbor features of each node in the spatiotemporal graph structure data in the spatial dimension to obtain the aggregated features of each node; the formula for calculating the aggregated features is:

[0163]

[0164] in, Represents the i-th key point in frame t. The aggregation characteristics, Indicates key points The corresponding anatomical neighborhood, This represents the j-th joint point in the anatomical neighborhood at frame t. Indicates key points Spatial coordinate characteristics, Indicates key points Type tags, Indicates key points Type tags, Represents a node Type tags and nodes The weight matrix corresponding to the type label, Z i denoted as the number of joints within the anatomical neighborhood corresponding to the i-th joint.

[0165] Specifically, Represents the i-th key point in frame t. The aggregated features after spatial neighborhood aggregation have a dimension of C. out This is the core feature of the node after integrating neighborhood information. This indicates the target key point during the traversal of frame t. All joints within the corresponding anatomical neighborhood For example, if It's the left elbow joint. Including the left shoulder joint and left wrist joint, at this time Take the left shoulder joint and then the left wrist joint in sequence. Indicates key points The spatial coordinate features are the two-dimensional coordinate vectors [x, y] of the joint point in the image coordinate system, with dimensions C. in =2. and Target joints and neighboring nodes The type labels (e.g., "left shoulder joint" corresponds to label "1", "left elbow joint" corresponds to label "2") are used to index the weight matrix assigned in S31. The learnable weight matrix in S31, assigned via the label mapping function, has dimension C. out ×C in .right Perform linear transformations (matrix multiplication W·f) in ), mapping two-dimensional coordinate features to C out The high-dimensional features of Z are combined with weighted information on anatomical associations. i For the i-th joint The number of joints in the corresponding anatomical neighborhood (i.e.) (The number of elements). For example, if the neighborhood of the left elbow joint contains two joints (left shoulder, left wrist), then Z... i =2. Introducing Z i This is to normalize the aggregated features and avoid feature scale differences caused by different numbers of neighbors at different joints (e.g., trunk joints have more neighbors than limb joints), thus ensuring stable network training.

[0166] The specific process of the aggregation operation is as follows: for the i-th key point of the t-th frame... Traversing its anatomical neighborhood Each joint within take out Spatial coordinate characteristics Use the corresponding weight matrix Perform a linear transformation (matrix multiplication) on it; sum the transformation results of all neighborhood key points, and then divide by Z. i After normalization, we get aggregation features

[0167] Through this process, the features of each joint retain their own coordinate information and are integrated with the features of other joints in the anatomical neighborhood. The integration weights are adaptively adjusted by a learnable matrix to ensure that they conform to the laws of human anatomical movement. For example, the feature aggregation of upper limb joints pays more attention to the movement correlation of upper limb neighborhood joints.

[0168] S33. Integrate the aggregated features of all nodes in each frame and output the spatial feature map of each frame.

[0169] Specifically, this step integrates the aggregated features of all keypoints within each frame to form a spatial feature map, providing input for subsequent temporal feature extraction. The integration logic is as follows: for a single frame (frame t), the frame contains N keypoints (e.g., the common 17 keypoints), and each keypoint corresponds to a dimension C. out aggregation features Arrange these N aggregated features in keypoint order to form an N×C array. out The matrix is ​​the spatial feature matrix of that frame. For a continuous T frames of a video data stream, each frame generates an N×C matrix. out The spatial feature matrices, stacked in chronological order, form a dimension of T×N×C. out The spatial feature map preserves both the aggregated features of each joint within each frame (reflecting spatial structural relationships) and the sequential nature of the temporal dimension (reflecting the temporal continuity of joint movements), providing spatial feature input with anatomical constraints for the subsequent "causal convolution operation in the temporal dimension".

[0170] In this embodiment, the spatiotemporal correlation of atomic behavior chains is accurately captured through the attention mechanism LSTM network, generating feature vectors that represent the inherent patterns of the behavior chains. This breaks through the bottleneck of traditional methods that cannot deeply understand continuous related behaviors, and provides core feature inputs for subsequent risk reasoning.

[0171] In one optional embodiment, risk reasoning is performed based on feature vectors and combined with a pre-built coal mine safety knowledge graph to obtain risk prediction results, including the following steps:

[0172] S41. Calculate the similarity between the feature vector and the embedding vector of each entity in the coal mine safety knowledge graph. For feature vectors with similarity higher than the threshold, establish mappings between the atomic behaviors and entities corresponding to the embedding vectors to obtain behavior-knowledge association data. Entities include equipment, regions, and rules.

[0173] Specifically, this step first completes the pre-construction of entity embedding vectors in the coal mine safety knowledge graph, and then establishes the mapping relationship between the feature vectors output by S3 and the entity embedding vectors through similarity calculation. The core is to realize the semantic association between atomic behavior features and knowledge graph entities, so as to provide knowledge support for subsequent risk reasoning.

[0174] The first step is the construction of entity embedding vectors in the knowledge graph. For the three core entity types (equipment, region, and rule) in the knowledge graph, a graph embedding algorithm is used to map entities into fixed-dimensional vectors, ensuring that the embedding vectors of different entity types reside in the same semantic space. For equipment entities (such as "fully mechanized mining support" and "high-voltage switch") and region entities (such as "fully mechanized mining face" and "return airway"), the Node2Vec algorithm is used for embedding learning. This algorithm generates contextual neighbor sequences for entities through random walks (e.g., the neighbor sequence for "high-voltage switch" is "fully mechanized mining face - violation operation rule - cable"), and then encodes the entities into d-dimensional embedding vectors (denoted as e) using a Skip-gram model. ent d has the same dimension as the feature vector output by S3 to ensure that the similarity calculation dimension matches; for rule entities (such as "No entry to the work area without a safety helmet" or "Power off and test for voltage when operating a high-voltage switch"), the TransE algorithm is used to transform the "premise-conclusion" logic of the rule into a vector relationship (denoted as e). rule That is, by optimizing the objective function of "premise entity embedding + relation embedding ≈ conclusion entity embedding", the rule embedding vector contains logical constraint information. The embedding vectors of the three types of entities are uniformly stored in the vector database of the knowledge graph, forming a mapping table of "entity ID - embedding vector - entity type".

[0175] The next step is similarity calculation and mapping establishment. The feature vector output by S3 (denoted as f) seq Taking the same dimension as the entity embedding vector as the input, we iterate through the embedding vectors of all entities in the knowledge graph vector database and calculate the semantic relevance between them using cosine similarity. The calculation formula is as follows:

[0176]

[0177] In the formula, sim(·) represents the cosine similarity value, which ranges from [-1, 1]; f seq ·e ent The dot product of the feature vector and the entity embedding vector is represented by ||f||. seq ‖ and ‖e ent ‖ represents the L2 norm of the two vectors. The similarity threshold is set based on the distribution characteristics of entities in the knowledge graph and is determined through optimization using an offline validation set. For example, this threshold ensures that the mapping accuracy between "atomic behavior and associated entity" is ≥90%. When the similarity value is higher than the threshold, it is determined that the atomic behavior corresponding to the feature vector has a semantic association with the entity.

[0178] Finally, the generation of behavior-knowledge association data is performed. For feature vectors and entity embedding vectors with similarity exceeding a threshold, their corresponding atomic behavior information (e.g., "climbing equipment") and entity information (e.g., equipment entity "fully mechanized mining support", regional entity "fully mechanized mining face", rule entity "climbing equipment prohibited") are extracted to establish multi-dimensional associations. The format of the association data can include "atomic behavior ID-entity ID-entity type-similarity value-association timestamp". This association data realizes the semantic binding between atomic behaviors and knowledge graph entities, providing an association index for subsequent aggregation of risk information from neighboring nodes.

[0179] S42. Based on behavior-knowledge association data, the risk information of each entity's neighbor nodes in the coal mine safety knowledge graph is aggregated through the message passing mechanism of graph neural network to obtain knowledge-enhanced risk propagation characteristics.

[0180] Specifically, this step is based on the behavior-knowledge association data generated by S41. Using a knowledge graph as the topology, it aggregates the risk information of the neighboring nodes of the target entity through the message passing mechanism of a graph neural network (GNN). The core is to capture the risk propagation path between entities (such as "climbing equipment → fully mechanized mining support → high-voltage cable → electric shock risk") and generate propagation features containing associated risks.

[0181] First, the selection and topology construction of the graph neural network. A Graph Attention Network (GAT) was chosen as the message passing carrier because it can adaptively distinguish the importance of neighboring nodes through attention weights, adapting to the heterogeneity of entity associations in the coal mine knowledge graph (e.g., the association strength between equipment and rules is higher than that between equipment and regions). The input topology of GAT is constructed based on the knowledge graph. Nodes are entities (equipment, regions, rules) in the knowledge graph, and edges are the associations between entities (e.g., "equipment-located-region" and "rule-constraint-equipment"). The edge weights are set by the association confidence between entities (taken from the relationship attributes of the knowledge graph). Simultaneously, based on the behavior-knowledge association data of S41, entities "associated with atomic behaviors" are marked as "seed nodes" in the topology. Subsequent message passing will start from the seed node, aggregating information from its 1-3 hop neighbor nodes (the number of hops is set by balancing the risk propagation range and computational complexity).

[0182] Secondly, there's message passing and aggregation. GAT's message passing process consists of three steps: "message generation - attention weight calculation - message aggregation".

[0183] The first step is to generate a message. For each neighboring node u of the seed node v, the features of the neighboring nodes (including the node's risk attribute values, such as "equipment aging level", "regional gas concentration threshold", and "number of rule violations") are concatenated with the edge features (confidence of inter-entity association). A message vector is generated through a linear transformation, and the calculation formula is as follows:

[0184] m u→v =W m ·[h u ,e u→v ]+b m

[0185] In the formula, m u→v W represents the message vector passed from neighbor node u to seed node v. m The weight matrix h represents the linear transformation. u Let e ​​represent the feature vector of neighbor node u. u→v Let b represent the eigenvector of the edge u→v. m The symbol represents the bias vector, and "[·,·]" represents the vector concatenation operation.

[0186] The second step is to calculate the attention weights. A multilayer perceptron (MLP) is used to calculate the attention weights of neighbor node u on seed node v, highlighting the message contributions of important neighbors. The calculation formula is as follows:

[0187]

[0188] In the formula, α u→v W represents the attention weight from neighbor node u to seed node v (values ​​range [0,1], the sum of all neighbor weights is 1). a h represents the weight matrix of the MLP. v This represents the initial feature vector of the seed node v. Let v represent the set of all neighboring nodes of the seed node v, LeakyReLU is the activation function (used to avoid gradient vanishing), and exp is the exponential function.

[0189] The third step is message aggregation, which involves summing the message vectors of each neighboring node according to their attention weights and fusing them with the initial feature vector of the seed node to obtain the updated features of the seed node (i.e., risk propagation features). The calculation formula is as follows:

[0190]

[0191] In the formula, W represents the risk propagation characteristics of seed node v after one round of message passing. self This represents the weight matrix of the seed node's self-features, and ReLU is the activation function.

[0192] For multi-hop neighbor aggregation, repeat the above three steps (the number of hops is determined by the risk propagation range). The final output risk propagation feature dimension is consistent with the initial feature dimension of the seed node and contains risk association information of neighbor nodes. For example, the risk propagation feature of "fully mechanized mining support" includes information such as "high voltage cable aging" and "violation of operating rules".

[0193] S43. Based on the characteristics of risk propagation, perform security constraint checks through rule reasoning to obtain the check results; calculate the behavioral chain risk violation score based on the check results.

[0194] Specifically, this step, based on the risk propagation characteristics generated by S42 and combined with predefined coal mine safety constraint rules, determines whether the atomic behavior chain violates safety regulations through rule reasoning, and then quantifies and calculates the risk violation score. The core is to transform qualitative safety rules into quantitative risk indicators, providing a basis for risk intensity for subsequent probabilistic prediction.

[0195] The first step is the construction and formalization of safety constraint rules. Safety constraint rules can originate from relevant regulations and internal safety management systems of coal mining enterprises, and then be transformed into machine-executable formal rules. Rules can be divided into three categories: first, "equipment-behavior" constraint rules, such as "When the high-voltage switch is not de-energized, it is forbidden to touch the operating handle"; second, "area-behavior" constraint rules, such as "When the gas concentration exceeds the limit, hot work is prohibited in this area"; and third, "behavior-behavior" sequential constraint rules, such as "Check for voltage before operating high-voltage equipment; the order must not be reversed." The formal rule expression can include "preconditions-constrained behavior-violation consequences-severity weight," for example, a production rule: IF(entity type = high-voltage switch AND equipment state = not de-energized) AND(atomic behavior = touching the operating handle); THEN violation rule = RL023, severity weight = 0.9.

[0196] Secondly, rule reasoning and safety constraint checks are performed. The Rete algorithm can be used for efficient rule matching. This algorithm constructs a "rule network" to decompose formal rules into atomic condition nodes. The risk propagation features of S42 (including entity state and atomic behavior information) are then input into the rule network, proceeding through three steps: matching, conflict resolution, and execution. The first step, matching, matches the entity state (e.g., "high-voltage switch not de-energized") and atomic behavior (e.g., "touch operating handle") in the risk propagation features with the atomic condition nodes of the rule network, generating partial matching results. The second step, conflict resolution, sorts multiple matching rules in descending order of severity weight, prioritizing rules with higher weights. The third step, execution, outputs the check result for each matching rule, categorized as "compliant" (feature did not trigger the rule's premise) and "violated" (feature triggered the rule's premise), while recording the ID, severity weight, and associated atomic behavior chain of the violated rule.

[0197] Finally, the risk violation score of the behavioral chain is calculated. Based on the constraint check results, a weighted summation model is used to calculate the overall risk violation score of the atomic behavioral chain. The calculation formula is as follows:

[0198]

[0199] In the formula, S represents the risk violation score of the behavioral chain, ranging from [0,1], with higher scores indicating greater risk; M represents the number of rules violated; w i k represents the severity weight of the rule violation of the i-th rule, with a value range of [0,1]; i The number of violations of the i-th rule is represented by r, where multiple violations of a rule by the same row are cumulative; L represents the number of entities associated with the i-th rule, and r represents the number of entities associated with the i-th rule. i,j This represents the risk attribute value of the j-th entity associated with the i-th rule, with a value range of [0,1]. Before calculation, all parameters are normalized to the [0,1] interval to ensure the reasonableness of the score calculation.

[0200] S44. Based on the risk violation score and risk propagation characteristics, the probabilistic risk events for each time window are obtained through a time-series risk prediction model, which serve as the risk prediction results.

[0201] Specifically, this step, based on the risk violation score of S43 and the risk propagation characteristics of S42, captures the temporal evolution trend of behavioral chain risks through a time-series risk prediction model, and outputs probabilistic risk events for each time window. The core is to realize the mapping of "risk characteristics - time window - probability distribution" to provide quantitative basis for subsequent multi-level early warning.

[0202] The first step involves selecting and constructing the input for the time-series risk prediction model. A Temporal Convolutional Network (TCN) can be chosen as the prediction model due to its parallel computing capabilities and ability to effectively capture long-term and short-term time dependencies, making it suitable for the dynamic evolution of risk events in coal mine operations (e.g., the time chain of "violation → equipment malfunction → accident occurrence"). The model input is a "time-series feature sequence," which is composed of two parts: first, risk propagation features arranged in chronological order (each time step corresponds to the risk propagation features of a behavioral segment, with a dimension of d1); and second, the risk violation score for the corresponding time step (with a dimension of 1), ultimately forming a time-series input matrix with a dimension of "number of time steps × (d1+1)". The division of time windows is combined with the coal mine operation cycle and risk response time. For example, the time window can be set as "the current moment + 3 future time windows" to predict the probability of occurrence of various risk events within each window.

[0203] Secondly, regarding the model's network structure and training, the TCN network structure contains multiple causal convolutional blocks. Each convolutional block consists of a "causal convolutional layer - batch normalization layer - ReLU activation layer - dropout layer," and residual connections mitigate the gradient vanishing problem. The kernel size of the causal convolutional layer can be set to 3-5 (covering features across 3-5 time steps), with padding set to "kernel size - 1" to ensure the output time step matches the input. The batch normalization layer standardizes the features, accelerating model convergence. The dropout layer prevents overfitting, and the dropout probability is determined through validation set optimization. The model's training dataset contains "temporal feature sequences - risk event labels" samples from historical behavior chains. The labels represent actual risk events occurring within each time window (e.g., "fall from height," "mechanical injury"). The cross-entropy loss function is used to optimize the model parameters, ensuring that the accuracy and F1 score for risk event prediction on the validation set exceed preset values.

[0204] Finally, the probabilistic risk event generation is performed. The constructed time-series input matrix is ​​input into the trained TCN model. The model output layer uses the softmax activation function to map the features of each time window to the probability distribution of various risk events. The calculation formula is as follows:

[0205]

[0206] In the formula, P(t,c) represents the probability of the c-th risk event occurring within the t-th time window, with a value ranging from [0,1], and the sum of the probabilities of all events is 1; z(t,c) represents the raw score of the c-th event in the t-th time window from the model output layer, and C represents the number of risk event categories. The output probabilistic risk events are sorted by time window, and the format includes "time window number - risk event type - probability of occurrence - risk level".

[0207] In this embodiment, behavioral features are integrated with a coal mine safety knowledge graph. Through entity embedding matching, GNN aggregation of risk information, rule reasoning to quantify violation scores and time-series prediction, probabilistic risk events are generated, achieving a leap from behavioral analysis to risk prediction. This improves the accuracy, foresight, and interpretability of risk prediction, and directly supports the formulation of multi-level early warning strategies.

[0208] The aforementioned method for intelligent identification of unsafe behaviors of coal mine workers extracts individual trajectory data from underground video streams using a multi-target tracking algorithm. It then decomposes continuous actions into spatiotemporally labeled atomic behavior sequences using pose estimation and a spatiotemporally graph convolutional network. Next, an attention-enhanced long short-term memory network captures the spatiotemporal correlation of the behavior chains to generate feature vectors. Finally, it combines a coal mine safety knowledge graph to perform risk reasoning and predict the types and probabilities of risk events. Based on a multi-level early warning strategy, it generates early warning information. This scheme achieves a leap from isolated action identification to behavior chain risk prediction. Through spatiotemporal correlation analysis of behavior sequences and domain knowledge fusion, it significantly improves the early warning capability for potential safety risks in coal mine operations.

[0209] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0210] Based on the same inventive concept, this application also provides a system for implementing the method for intelligent identification of unsafe behaviors of coal mine workers as described above. The solution provided by this system is similar to the solution described in the above method; therefore, the specific limitations in one or more system embodiments for intelligent identification of unsafe behaviors of coal mine workers provided below can be found in the limitations of the method for intelligent identification of unsafe behaviors of coal mine workers described above, and will not be repeated here.

[0211] In one exemplary embodiment, such as Figure 3 As shown, a system 30 for intelligent identification of unsafe behaviors of coal mine workers is provided to implement the methods in the above-described embodiments. The system includes:

[0212] The real-time trajectory perception module 31 is used to obtain individual trajectory data, including personnel ID, timestamp and spatial coordinates, based on the video data stream collected by cameras deployed in underground coal mines and through a multi-target tracking algorithm.

[0213] The behavior feature modeling module 32 is used to process the human image segments in the video sequence of the video data stream using a pose estimation algorithm based on individual trajectory data to obtain the human skeleton key point sequence; based on the human skeleton key point sequence, atomic behaviors are identified through a spatiotemporal graph convolutional network to generate a spatiotemporally labeled atomic behavior sequence.

[0214] The behavior chain correlation analysis module 33 is used to analyze atomic behavior sequences through a long short-term memory network based on an attention mechanism, capture the spatiotemporal correlation of the behavior chain, and generate feature vectors that characterize the intrinsic patterns of the behavior chain.

[0215] The risk reasoning and decision-making module 34 is used to perform risk reasoning based on feature vectors and combined with a pre-built coal mine safety knowledge graph to obtain risk prediction results; the risk prediction results include risk event types and occurrence probabilities.

[0216] The intelligent early warning execution module 35 is used to generate early warning information based on risk prediction results through a multi-level early warning strategy, and send the early warning information to operators and safety supervisors.

[0217] Embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the aforementioned method embodiments.

[0218] Embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0219] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0220] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for intelligent identification of unsafe behaviors of coal mine workers, characterized in that, The method includes: S1. Based on the video data stream collected by cameras deployed underground in the coal mine, individual trajectory data including personnel ID, timestamp and spatial coordinates are obtained through a multi-target tracking algorithm; S2. Based on the individual trajectory data, the person image segments in the video sequence of the video data stream are processed using a pose estimation algorithm to obtain a human skeleton key point sequence; based on the human skeleton key point sequence, atomic behaviors are identified through a spatiotemporal graph convolutional network to generate a spatiotemporally labeled atomic behavior sequence. S3. The atomic behavior sequence is analyzed by an attention-based long short-term memory network to capture the spatiotemporal correlation of the behavior chain and generate feature vectors that characterize the intrinsic patterns of the behavior chain. S4. Based on the feature vector, risk reasoning is performed in conjunction with a pre-constructed coal mine safety knowledge graph to obtain risk prediction results; the risk prediction results include risk event types and occurrence probabilities. S5. Based on the risk prediction results, generate early warning information through a multi-level early warning strategy and send the early warning information to the operators and safety supervisors.

2. The method according to claim 1, characterized in that, The step of identifying atomic behaviors and generating a sequence of atomic behaviors with spatiotemporal labels based on the sequence of key points of the human skeleton using a spatiotemporal graph convolutional network includes: S11. Construct a spatiotemporal graph model based on the sequence of key points of the human skeleton to obtain spatiotemporal graph structure data; wherein, the spatiotemporal graph model includes nodes, spatial edges and temporal edges, the nodes represent human joints, the spatial edges connect adjacent anatomical joints in the same frame of the video data stream, and the temporal edges connect the same joints in adjacent frames of the video data stream. S12. Based on the spatiotemporal graph structure data, features are extracted hierarchically through a spatiotemporal graph convolutional network to obtain spatiotemporal fusion features; S13. Based on the spatiotemporal fusion features, the confidence level of each frame action in the video data stream belonging to a predefined atomic behavior category is calculated through a fully connected layer and a Softmax classifier, and the probability distribution of the atomic behavior category is obtained. S14. Based on the atomic behavior category probability distribution, and combined with the dynamic time warping algorithm, align the behavior labels of consecutive frames in the video data stream to generate an initial atomic behavior sequence. S15. Based on the initial atomic behavior sequence, associate the corresponding timestamp, camera position and environmental parameters to generate the atomic behavior sequence with spatiotemporal markers.

3. The method according to claim 2, characterized in that, The spatiotemporal fusion features are obtained by performing hierarchical feature extraction using a spatiotemporal graph convolutional network based on the spatiotemporal graph structure data, including: S21. Based on the spatiotemporal graph structure data, perform a graph convolution operation based on anatomical constraints in the spatial dimension to obtain a spatial feature map; S22. Based on the spatial feature map, perform a causal convolution operation in the time dimension to obtain a spatiotemporal feature map; S23. Based on the spatiotemporal feature map, multi-scale features are fused through hierarchical residual connections to obtain multi-scale fused features; S24. Based on the multi-scale fusion features, the spatial attention mechanism is applied to enhance the features of key nodes, resulting in spatial attention-enhanced spatiotemporal fusion features.

4. The method according to claim 3, characterized in that, The process of performing anatomically constrained graph convolution operations in the spatial dimension based on the spatiotemporal graph structure data to obtain a spatial feature map includes: S31. For each key point in each frame of the video data stream, a learnable weight matrix is ​​assigned according to the corresponding anatomical neighborhood using a label mapping function; S32. Based on the weight matrix, aggregate the neighbor features of each node in the spatiotemporal graph structure data in the spatial dimension to obtain the aggregated features of each node; the calculation formula for the aggregated features is: in, Represents the i-th key point in frame t. The aforementioned aggregation feature, Indicates key points The corresponding anatomical neighborhood, This represents the j-th joint point within the anatomical neighborhood in frame t. Indicates key points Spatial coordinate characteristics, Indicates key points Type tags, Indicates key points Type tags, Represents a node Type tags and nodes The weight matrix Z corresponding to the type label i The number of joints in the anatomical neighborhood corresponding to the i-th joint; S33. Integrate the aggregated features of all nodes in each frame and output the spatial feature map of each frame.

5. The method according to any one of claims 1 to 4, characterized in that, The risk prediction result obtained by performing risk reasoning based on the feature vector and combining it with a pre-constructed coal mine safety knowledge graph includes: S41. Calculate the similarity between the feature vector and the embedding vector of each entity in the coal mine safety knowledge graph, and establish a mapping between the atomic behaviors and entities corresponding to the feature vectors and embedding vectors with similarity higher than a threshold, respectively, to obtain behavior-knowledge association data; the entities include equipment, regions and rules; S42. Based on the behavior-knowledge association data, the risk information of each entity's neighbor nodes in the coal mine safety knowledge graph is aggregated through the message passing mechanism of the graph neural network to obtain knowledge-enhanced risk propagation features. S43. Based on the risk propagation characteristics, perform security constraint checks through rule reasoning to obtain the check results; calculate the behavioral chain risk violation score based on the check results; S44. Based on the risk violation score and the risk propagation characteristics, the probabilistic risk events for each time window are obtained through a time-series risk prediction model, which are used as the risk prediction results.

6. A system for intelligent identification of unsafe behaviors of coal mine workers, used to implement the method according to any one of claims 1 to 5, characterized in that, The system includes: The real-time trajectory perception module is used to obtain individual trajectory data, including personnel ID, timestamp, and spatial coordinates, based on video data streams collected by cameras deployed underground in coal mines and through multi-target tracking algorithms. The behavior feature modeling module is used to process the human image segments in the video sequence of the video data stream using a pose estimation algorithm based on the individual trajectory data to obtain a sequence of human skeleton key points; based on the sequence of human skeleton key points, a spatiotemporal graph convolutional network is used to identify atomic behaviors and generate a sequence of atomic behaviors with spatiotemporal labels. The behavior chain correlation analysis module is used to analyze the atomic behavior sequence through a long short-term memory network based on an attention mechanism, capture the spatiotemporal correlation of the behavior chain, and generate feature vectors that characterize the intrinsic patterns of the behavior chain. The risk reasoning and decision-making module is used to perform risk reasoning based on the feature vector and in combination with a pre-constructed coal mine safety knowledge graph to obtain risk prediction results; the risk prediction results include risk event types and occurrence probabilities. The intelligent early warning execution module is used to generate early warning information based on the risk prediction results through a multi-level early warning strategy, and send the early warning information to operators and safety supervisors.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images