Field personnel behavior recognition method based on human skeleton key points

Abstract

The invention discloses a field personnel behavior identification method based on human skeleton key points. The method is suitable for behavior monitoring of operators in complex production environments such as hydraulic power plants. The method comprises the following steps: performing feature extraction and modeling on skeleton data acquired by camera equipment, and performing multi-scale space-time modeling by adopting a time sequence guide space topology modeling module and a space-time motion modeling network; according to the method, the capturing capability of the dynamic relation between human skeleton nodes is enhanced, the behavior recognition precision of long time span and complex postures is improved, and the robustness and generalization capability of the model in a complex environment are improved; on the whole, the risk behavior of the operating personnel can be effectively identified, the production safety is guaranteed, and the risk caused by human errors is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, in particular to an on-site personnel behavior recognition method based on key points of human skeleton. Background Art

[0002] With the development of intelligent technology, human behavior recognition is increasingly playing a vital role in many practical applications, particularly in industrial production and safety supervision. In environments such as power plants, identifying operator behavior is crucial for preventing safety accidents and improving production efficiency. For example, in high-risk environments like generator sets, operator errors can have serious consequences. Therefore, real-time monitoring and identifying their behavior can promptly detect and prevent potential hazards. Currently, image- and video-based behavior recognition technology has begun to be gradually applied in production and safety fields, not only reducing labor costs but also providing timely and accurate warnings to ensure safe production. Skeleton data—information about human joints extracted from images or videos—is an effective way to describe human behavior. Its significant advantages in eliminating background interference and coping with complex lighting conditions have made skeleton-based behavior recognition a popular technology in many applications.

[0003] Existing methods for human action recognition based on skeleton data typically employ deep learning models, such as graph convolutional networks, to model the connections between the joints of the human skeleton and achieve action recognition in both time and space. These methods construct a skeleton topology, modeling the physical connection structure and motion information of the joints to improve action recognition accuracy and adaptability to different actions.

[0004] However, existing technologies still have shortcomings when dealing with complex scenes. First, although these methods can model the relationship between different joints, the interactive processing of spatiotemporal information is still relatively limited, making it difficult to fully capture dynamic changes over long time spans. Secondly, when it comes to diverse actions and skeleton changes in complex scenes, the generalization ability of the model is still insufficient and may be affected by different skeleton postures or environmental noise, resulting in a decrease in recognition accuracy. Therefore, how to effectively improve the model's ability to process dynamic information and enhance its robustness and generalization ability in complex scenes is a problem that existing technologies need to further address. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a field personnel behavior recognition method based on key points of the human skeleton, which can improve the efficiency and accuracy of identifying the behavior of operators in complex production scenarios such as hydropower plants, ensure production safety, and reduce the risk of human errors.

[0006] To solve the above technical problems, the technical solution adopted in the present invention is: a field personnel behavior recognition method based on human skeleton key points, which comprises the following steps:

[0007] Step 1: Use cameras to collect video data of common high-risk behaviors in hydropower plant scenes, preprocess the data, extract skeleton data using pose estimation algorithms, and construct a field personnel behavior dataset;

[0008] Step 2: Construct a temporal guided spatial topology modeling module (DMSG): extract the rhythmic features between skeleton frames, fuse structural connections and motion responses to construct multiple sets of spatial topology graphs, and guide graph convolution to group channels and model dynamic dependencies at different time periods.

[0009] Step 3: Construct a spatiotemporal motion modeling network (TMSN): Combine DMSG with the multi-scale temporal modeling module (ATSC) to form a stacked spatiotemporal hybrid block. This extracts the correlation features of the skeleton in spatial structure and temporal dynamics, and ultimately completes action recognition through global pooling and a softmax classifier.

[0010] Step 4: Input the data set in step 1 into the spatiotemporal motion modeling network described in step 3 for training, and adjust and optimize the network weights based on the forward and backward propagation of the loss function to obtain the optimal behavior recognition model.

[0011] Preferably, step 1 includes the following process:

[0012] Step 1.1: Use industrial-grade video equipment to record on-site personnel activities and select several common risk behaviors of hydropower plant operators;

[0013] Step 1.2: Annotate the human skeleton data;

[0014] Step 1.3: Divide the skeleton data into training and validation sets, and randomly move and scale the extracted skeleton joints to enhance the diversity of the dataset and the robustness of the model.

[0015] Preferably, in step 1.1, several common risk behaviors of hydropower plant operators include climbing heights, operating high-voltage equipment, carrying heavy objects, leaning on equipment, falling or slipping, and using tools for maintenance.

[0016] Preferably, the specific process of annotating the human skeleton data in step 1.2 is: first, the collected video of each type of activity is cropped, and the original video is re-edited into 10-second segments at a frame rate of 20 frames per second, and the original video resolution is compressed to 240×320. Then, the human skeleton data is extracted using a pose estimation algorithm to generate C×T×V tensor format data; wherein C represents the 2D pixel coordinates of the skeleton node, T represents the number of skeleton frames, and V represents the number of skeleton joint points.

[0017] Preferably, in step 1.3, the skeleton data is divided into a training set and a validation set in a ratio of 8:2.

[0018] Preferably, the step 2 comprises:

[0019] Step 2.1, rhythm-aware encoding: Represent the skeleton time series data as the input feature tensor X in , whose dimension is (C, T, J), where C represents the initial number of channels, T represents the number of time steps, and J represents the number of skeleton joints. To reduce the subsequent computational overhead, the original features are firstly channel compressed and normalized to obtain the feature tensor Q:

[0020] Q=Norm(W q ·X in )

[0021] Among them, W q is a trainable projection matrix, and Norm represents the normalization operation performed along the time dimension, which is used to unify the distribution differences between frames.

[0022] In order to characterize the dynamic response between frames and calculate the structural interaction relationship between adjacent time frames, the motion activation tensor R is defined as:

[0023] R t =σ(Q t ⊙Q t+1 ),t=1,2,…,T-1

[0024] Among them, ⊙ represents the corresponding element multiplication, σ is the Sigmoid activation function, which is used to compress the response value to the (0,1) interval to enhance the sensitivity to the action boundary; the resulting tensor Used to represent significant motion information between consecutive frames.

[0025] Preferably, the step 2 further comprises:

[0026] Step 2.2, topology guided graph construction: The joint dependency of the skeleton is determined by the structure response graph G s and action response graph G a Construct multiple graph structures (K in total), used to guide spatial graph convolution operations and enhance the ability to model static and dynamic connections between nodes;

[0027] Structural response diagram G s :The static topological relationship of each joint in the skeleton is obtained through the adjacency matrix Indicates that each element Defined as:

[0028]

[0029] in, Represents the average geometric position of joint point i, which is calculated from the three-dimensional coordinates of the node in frame T in the skeleton sequence:

[0030]

[0031] Where, is the 3D coordinate information of joint point i in the t-th frame of the original skeleton input, which can be provided by the pose estimation model or sensor system. The parameter σ>0 is the kernel width that controls the range of geometric similarity attenuation, and δ is the upper limit of the distance that defines the geometric connection, which is used to limit the construction range of non-structural edges and prevent misconnection between unrelated node pairs.

[0032] To make the computation of graph convolution more numerically stable, the adjacency matrix Normalization is required to avoid feature scale shift caused by uneven node degrees. Symmetric normalization is used for normalization:

[0033]

[0034] That is the degree matrix, and the diagonal elements are:

[0035]

[0036] Action response graph G a :This graph is used to model the collaborative response relationship between non-structurally connected nodes during movement; First, the motion activation tensor obtained in step 2.1 is Perform average pooling along the time dimension to obtain the response vector of each joint point:

[0037]

[0038] In order to express the response difference between nodes, the average L1 distance metric is introduced, and the response difference between any two joint points i and j is defined as:

[0039]

[0040] In order to map the difference value into the edge weight form, an exponential decay mapping function is introduced to convert the response difference into the connection strength:

[0041]

[0042] Where τ is an adjustable temperature coefficient, which is used to control the influence range of the response difference on the connection strength;

[0043] Graph Structure Set In order to enhance the model's ability to perceive multiple structural dependencies, K spatial graph structure subsets are introduced. Each subgraph structure is constructed by a different structure-action fusion ratio to reflect the model's attention pattern on different scales or different regions. It is obtained by weighted combination of structure graph and action graph, and the expression is:

[0044]

[0045] in, They represent the weight ratios of the structure graph and the action graph in the k graph structures respectively. The weight parameters are learnable variables used to adapt to the dependency preferences during network training.

[0046] The final graph structure set is:

[0047] K = number of spatial subsets, preferably 3 or 4.

[0048] Preferably, the step 2 further comprises:

[0049] Step 2.3, Implementation of dynamic graph structure convolution module: Given input feature X in And the K graph structures generated in step 2.2 First, the input features are divided into S equal-length time periods in the time dimension, and the length of each period is The feature of the i-th segment is expressed as:

[0050]

[0051] Next, each segment feature is divided into K sub-channel groups in the channel dimension, and each group of features is recorded as:

[0052]

[0053] In each time period i, using the graph structure Guide the corresponding channel group to perform spatial graph convolution operation, and the convolution result is:

[0054]

[0055] Concatenate the K-way convolution results in the channel dimension to get the i-th segment output:

[0056]

[0057] Finally, the output features of S time periods are spliced ​​in the time dimension to obtain the output of the module:

[0058]

[0059] Preferably, in the step 3, the spatiotemporal motion modeling network TMSN includes 10 spatiotemporal hybrid blocks TSMBlock, each block consisting of a dynamic motion spatial graph structure DMSG and an adaptive multi-scale temporal modeling module ATSC; each ATSC module contains 4 branches: the first and second branches use time dilation convolution with a convolution kernel size of 5×1, and set two different dilation factors respectively; the third branch is constructed by first dimensionality reduction and then pooling: first, channel compression is performed through a 1×1 convolution layer, and then a maximum pooling operation is performed along the time dimension, and finally, the channel dimension is restored through another 1×1 convolution to extract local time-invariant features; the fourth branch is a self-attention mechanism based on the time dimension, which first maps the input features into three tensors of query, key and value, calculates the global time dependency within each frame, and fuses them in the form of attention weighting to enhance the global perception ability of time series modeling; the outputs of the above four branches are leveled in the channel dimension, and then the channel attention mechanism is introduced to generate fusion weights, and the features of each channel are weighted and adjusted, and finally, channel compression is performed through convolution as the final output of the module.

[0060] Preferably, in the spatiotemporal motion modeling network TMSN, the number of output channels of the first four blocks is 64; in the fifth and eighth spatiotemporal blocks, the number of output channels is multiplied by 2, and downsampled in the time dimension; in each block, batch normalization BatchNorm and ReLU activation function are combined to improve the training efficiency and stability of the network; the extracted features are aggregated through the global average pooling layer to compress the time and space dimensions; finally, the Softmax classifier is used to predict the action category, and the classification results are used for on-site personnel behavior recognition.

[0061] Beneficial effects of the present invention:

[0062] 1. This paper enhances the modeling capabilities of dynamic responses between skeleton frames and dependencies between joints by introducing rhythm-aware encoding and multi-graph structure guidance. Through channel compression and structural interaction modeling, more discriminative motion activation features are extracted, providing effective timing guidance for subsequent topology construction and graph convolution operations.

[0063] 2. This invention improves the model's recognition accuracy for complex actions and long-term behavioral patterns through the combined design of a temporally guided graph structure modeling module (DMSG) and a temporal motion modeling network (TMSN). The DMSG module integrates physical structure and motion response information to construct multiple topological graphs. The ATSC module in the temporal motion modeling network exploits multi-level temporal features through multi-scale convolution and channel attention mechanisms, achieving enhanced spatiotemporal collaborative perception capabilities.

[0064] 3. This invention introduces a graph structure optimization strategy and motion-guided convolution mechanism, effectively improving the model's anti-interference ability and generalization performance in complex scenarios. This overall approach can be widely applied in high-risk operating environments such as hydropower plants and substations, enabling efficient and accurate identification of operator behavior, enhancing the safety and management efficiency of production processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of the temporal guided spatial topology modeling module;

[0066] Figure 2 Schematic diagram of the network structure for modeling spatiotemporal motion;

[0067] Figure 3 This is a schematic diagram of the skeleton structure of the modeler;

[0068] Figure 4 This is the position attention structure diagram in Example 3;

[0069] Figure 5 This is a diagram of the space and channel information processing module in Example 4. DETAILED DESCRIPTION

[0070] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0071] Example 1: Figure 1-3 As shown, a method for on-site personnel behavior recognition based on human skeleton key points includes the following steps:

[0072] Step 1: Use cameras to collect video data of common high-risk behaviors in hydropower plant scenes, preprocess the data, extract skeleton data using pose estimation algorithms, and construct a field personnel behavior dataset;

[0073] Step 2: Construct a temporally guided spatial topology modeling module (DMSG): extract the rhythmic features between skeleton frames, fuse structural connections and motion responses to construct multiple sets of spatial topology graphs, and guide graph convolution to group channels and model dynamic dependencies at different time periods.

[0074] Step 3: Construct a spatiotemporal motion modeling network (TMSN): Combine DMSG with the multi-scale temporal modeling module (ATSC) to form a stacked spatiotemporal hybrid block. This extracts the correlation features of the skeleton in spatial structure and temporal dynamics, and ultimately completes action recognition through global pooling and a softmax classifier.

[0075] Step 4: Input the data set in step 1 into the network described in step 2 and step 3 for training, and adjust and optimize the network weights based on the forward and backward propagation of the loss function to obtain the optimal behavior recognition model.

[0076] Furthermore, the step 1 includes the following process:

[0077] Step 1.1. Video Capture: Use industrial-grade video equipment to record on-site personnel activities. Select several common risk behaviors of hydropower plant operators, such as climbing high (CH), operating high-voltage equipment (OE), carrying heavy objects (CO), leaning on equipment (LE), slipping / falling (SF), and using tools for repair (TR).

[0078] Step 1.2: Annotate the human skeleton data. First, crop the collected videos of each type of activity and re-edit the original videos into 10-second segments at a frame rate of 20 frames per second. In order to reduce the amount of computation, compress the original video resolution to 240×320, and then use the pose estimation algorithm (such as OpenPose) to extract the human skeleton data and generate C×T×V tensor format data. Among them, C represents the 2D pixel coordinates of the skeleton node, T represents the number of skeleton frames, and V represents the number of skeleton joints;

[0079] Step 1.3: Split the skeleton data into a training set and a validation set with a ratio of 8:2. To enhance the generalization ability of the model, the extracted skeleton joints are further randomly moved and scaled to increase the diversity of the dataset and the robustness of the model.

[0080] Furthermore, the step 2 includes the following process:

[0081] Step 2.1, rhythm-aware encoding: Represent the skeleton time series data as the input feature tensor X in , whose dimension is (C, T, J), where C represents the initial number of channels, T represents the number of time steps, and J represents the number of skeleton joints. To reduce the subsequent computational overhead, the original features are first channel compressed and normalized to obtain the feature tensor Q:

[0082] Q=Norm(W q ·X in )

[0083] Among them, W q is a trainable projection matrix, and Norm represents the normalization operation performed along the time dimension, which is used to unify the distribution differences between frames.

[0084] In order to characterize the dynamic response between frames and calculate the structural interaction relationship between adjacent time frames, the motion activation tensor R is defined as:

[0085] R t =σ(Q t ⊙Q t+1 ),t=1,2,…,T-1

[0086] Among them, ⊙ represents the corresponding element multiplication, σ is the Sigmoid activation function, which is used to compress the response value to the (0,1) interval to enhance the sensitivity to the action boundary. The resulting tensor Used to represent significant motion information between consecutive frames.

[0087] Step 2.2, topology guided graph construction: The joint dependency of the skeleton is determined by the structure response graph G s and action response graph G a Construct multiple graph structures (K in total), used to guide spatial graph convolution operations and enhance the ability to model static and dynamic connections between nodes.

[0088] Structural response diagram G s :The static topological relationship of each joint in the skeleton is obtained through the adjacency matrix Indicates that each element Defined as:

[0089]

[0090] in, Represents the average geometric position of joint point i, which is calculated from the three-dimensional coordinates of the node in frame T in the skeleton sequence:

[0091]

[0092] Where, is the 3D coordinate information of joint point i in frame t in the original skeleton input. This coordinate information can be provided by the pose estimation model or sensor system. The parameter σ>0 is the kernel width that controls the range of geometric similarity decay. δ is the upper limit of the distance that defines geometric connectivity, which is used to limit the construction range of non-structural edges and prevent misconnections between unrelated node pairs.

[0093] To make the computation of graph convolution more numerically stable, the adjacency matrix Normalization is required to avoid feature scale shift caused by uneven node degrees. The normalization method uses symmetric normalization:

[0094]

[0095] That is the degree matrix, and the diagonal elements are:

[0096]

[0097] Action response graph G a :This graph is used to model the collaborative response relationship between non-structurally connected nodes during movement. First, the motion activation tensor obtained in step 2.1 is Perform average pooling along the time dimension to obtain the response vector of each joint point:

[0098]

[0099] In order to express the response difference between nodes, the average L1 distance metric is introduced, and the response difference between any two joint points i and j is defined as:

[0100]

[0101] In order to map the difference value into the edge weight form, an exponential decay mapping function is introduced to convert the response difference into the connection strength:

[0102]

[0103] Where τ is an adjustable temperature coefficient, which is used to control the influence range of the response difference on the connection strength.

[0104] Graph Structure Set In order to enhance the model's ability to perceive multiple structural dependencies, K spatial graph structure subsets are introduced. Each subgraph structure is constructed by a different structure-action fusion ratio to reflect the model's attention mode to different scales or different regions. It is obtained by weighted combination of structure graph and action graph, and the expression is:

[0105]

[0106] in, They represent the weight ratios of the structure graph and the action graph in the k graph structures respectively. The weight parameters are learnable variables used to adapt to the dependency preferences during network training.

[0107] The final graph structure set is:

[0108] K = number of spatial subsets, preferably 3 or 4.

[0109] Step 2.3, Implementation of dynamic graph structure convolution module: Given input feature X in And the K graph structures generated in step 2.2 First, the input features are divided into S equal-length time periods in the time dimension, and the length of each period is The feature of the i-th segment is expressed as:

[0110]

[0111] Next, each segment feature is divided into K sub-channel groups in the channel dimension, and each group of features is recorded as:

[0112]

[0113] In each time period i, using the graph structure Guide the corresponding channel group to perform spatial graph convolution operation, and the convolution result is:

[0114]

[0115] Concatenate the K-way convolution results in the channel dimension to get the i-th segment output:

[0116]

[0117] Finally, the output features of S time periods are spliced ​​in the time dimension to obtain the output of the module:

[0118]

[0119] Furthermore, step 3 includes the following process:

[0120] The spatiotemporal motion modeling network (TMSN) consists of 10 spatiotemporal hybrid blocks (TSMBlocks), each composed of a dynamic motion spatial graph (DMSG) and an adaptive multi-scale temporal modeling module (ATSC). Each multi-ATSC module contains four branches: the first and second branches use time-dilated convolutions with a kernel size of 5×1 and two different dilation factors, respectively. The third branch is constructed using a dimensionality reduction-then-pooling approach: it first performs channel compression through a 1×1 convolution layer, then performs a maximum pooling operation along the temporal dimension, and finally restores the channel dimension through another 1×1 convolution to extract local time-invariant features. The fourth branch is a self-attention mechanism based on the time dimension. It first maps the input features into three tensors: query, key, and value. Global temporal dependencies are calculated within each frame and fused in an attention-weighted form to enhance the global perception capability of temporal series modeling. The outputs of the above four branches are cascaded in the channel dimension, and then a channel attention mechanism is introduced to generate fusion weights. The features of each channel are weighted and adjusted, and finally channel compression is performed through convolution as the final output of the module.

[0121] In the temporal motion modeling network (TMSN), the first four blocks all have 64 output channels. In the fifth and eighth blocks, the number of output channels is multiplied by 2, and downsampling is performed in the temporal dimension. Within each block, batch normalization (BatchNorm) and the ReLU activation function are combined to improve network training efficiency and stability. Each extracted feature is aggregated through a global average pooling (GAP) layer to compress the temporal and spatial dimensions. Finally, a softmax classifier is used to predict the action category, and the classification results are used for on-site personnel behavior recognition.

[0122] Example 2:

[0123] This embodiment, based on the human skeleton keypoint-based behavior recognition method described in Example 1, also provides a feature extraction method for skeleton keypoint data, employing multi-branch feature extraction and a densely connected structure to achieve lightweight human pose estimation. This embodiment first uses a multi-branch feature extraction module to convolve the input image data through multiple parallel branches at different scales to extract diverse features. These branches use convolution kernels of varying sizes to capture features from different receptive fields, enhancing the model's ability to perceive both details and global information. Then, a densely connected structure is used to fuse the features extracted by all branches layer by layer, ensuring full utilization of feature information and effectively minimizing information loss. By concatenating the feature outputs of previous layers at each feature extraction layer, the densely connected structure enables each node to access features from all previous layers. This design enhances the effectiveness of feature transfer and reduces the risk of vanishing gradients. Finally, spatial graph convolution is performed on the fused features to achieve spatiotemporal modeling of human skeleton nodes. This embodiment has low computational complexity while maintaining high recognition accuracy even in complex scenarios, making it suitable for applications requiring efficient real-time monitoring, such as hydropower plants.

[0124] Example 3:

[0125] This embodiment is based on the behavior recognition method based on human skeleton key points in Example 1, and also provides a feature extraction method of position attention, which enhances the modeling capabilities of some steps.

[0126] In the skeleton feature extraction process, the relative position information of each skeleton node is calculated and quantified to capture the global correlation between different nodes, such as Figure 4 The specific operation includes: fusing the horizontal and vertical features of each node's position information. First, the input features are divided into horizontal and vertical features, and then these features are averaged to obtain the feature vectors of each node in these two directions. Next, these feature vectors are fused to generate a global feature representation. This approach enables the model to more comprehensively capture the detailed relationships between various nodes in the human body, thereby enhancing the perception of the human body's overall posture and dynamic behavior, especially when complex posture changes occur.

[0127] Example 4:

[0128] This embodiment is based on the behavior recognition method based on human skeleton key points in embodiment 1, and also provides a method for processing spatial and channel motion information. In the motion information processing step, a feature optimization process is introduced to reduce feature redundancy and improve feature effectiveness, such as Figure 5 The specific optimization steps are as follows:

[0129] The input features are first divided into multiple parts through the "Split" operation. The features of each part are cross-fused to extract more comprehensive motion information. The purpose of this step is to enhance the model's understanding of spatial features, retain feature information that has a significant impact on the action, and reduce irrelevant redundancy; channel optimization processing, after spatial feature processing, the feature map is again divided into two parts, and global pooling is performed separately. The pooled features are calculated using Softmax to calculate the importance weight of each channel, and then weighted reconstruction is performed. This step ensures that key features are focused on, thereby improving the accuracy of behavior recognition; the weighted feature parts are fused through addition operations to generate the final output features. This feature is further modeled for behavior recognition, which effectively improves the robustness and recognition accuracy of the model.

[0130] The above embodiments are merely preferred technical solutions of the present invention and should not be construed as limiting the present invention. The scope of protection of the present invention shall be the technical solutions set forth in the claims, including equivalent alternatives to the technical features of the technical solutions set forth in the claims. In other words, equivalent alternatives and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A method for on-site personnel behavior recognition based on key points of human skeleton, characterized by: It includes the following steps: Step 1: Use cameras to collect video data of common high-risk behaviors in hydropower plant scenes, preprocess the data, extract skeleton data using pose estimation algorithms, and construct a field personnel behavior dataset; Step 2: Construct a temporal guided spatial topology modeling module (DMSG): extract the rhythmic features between skeleton frames, fuse structural connections and motion responses to construct multiple sets of spatial topology graphs, and guide graph convolution to group channels and model dynamic dependencies at different time periods. Step 3: Construct a spatiotemporal motion modeling network (TMSN): Combine DMSG with the multi-scale temporal modeling module (ATSC) to form a stacked spatiotemporal hybrid block. This extracts the correlation features of the skeleton in spatial structure and temporal dynamics, and ultimately completes action recognition through global pooling and a softmax classifier. Step 4: Input the data set in step 1 into the spatiotemporal motion modeling network described in step 3 for training, and adjust and optimize the network weights based on the forward and backward propagation of the loss function to obtain the optimal behavior recognition model.

2. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 1, characterized in that: Step 1 includes the following process: Step 1.1: Use industrial-grade video equipment to record on-site personnel activities and select several common risk behaviors of hydropower plant operators; Step 1.2: Annotate the human skeleton data; Step 1.3: Divide the skeleton data into training and validation sets, and randomly move and scale the extracted skeleton joints to enhance the diversity of the dataset and the robustness of the model.

3. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 2, characterized in that: In step 1.1, several common risk behaviors of hydropower plant operators include climbing heights, operating high-voltage equipment, carrying heavy objects, leaning on equipment, falling or slipping, and using tools for maintenance.

4. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 2, characterized in that: The specific process of annotating the human skeleton data in step 1.2 is as follows: first, the collected video of each type of activity is cropped, and the original video is re-edited into 10-second segments at a frame rate of 20 frames per second, and the original video resolution is compressed to 240×320. Then, the human skeleton data is extracted using a pose estimation algorithm to generate C×T×V tensor format data; where C represents the 2D pixel coordinates of the skeleton nodes, T represents the number of skeleton frames, and V represents the number of skeleton joint points.

5. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 2, characterized in that: In step 1.3, the skeleton data is divided into a training set and a validation set in a ratio of 8:

2.

6. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 1, characterized in that: The step 2 includes: Step 2.1, rhythm-aware encoding: Represent the skeleton time series data as the input feature tensor X in , whose dimension is (C, T, J), where C represents the initial number of channels, T represents the number of time steps, and J represents the number of skeleton joints. To reduce the subsequent computational overhead, the original features are firstly channel compressed and normalized to obtain the feature tensor Q: Q=Norm(W q ·X in ) Among them, W q is a trainable projection matrix, and Norm represents the normalization operation performed along the time dimension, which is used to unify the distribution differences between frames. In order to characterize the dynamic response between frames and calculate the structural interaction relationship between adjacent time frames, the motion activation tensor R is defined as: R t =σ(Q t ⊙Q t+1 ),t=1,2,…,T-1 Among them, ⊙ represents the corresponding element multiplication, σ is the Sigmoid activation function, which is used to compress the response value to the (0,1) interval to enhance the sensitivity to the action boundary; the resulting tensor Used to represent significant motion information between consecutive frames.

7. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 6, characterized in that: The step 2 further comprises: Step 2.2, topology guided graph construction: The joint dependency of the skeleton is determined by the structure response graph G s and action response graph G a Construct multiple graph structures (K in total), used to guide spatial graph convolution operations and enhance the ability to model static and dynamic connections between nodes; Structural response diagram G s :The static topological relationship of each joint in the skeleton is obtained through the adjacency matrix Indicates that each element Defined as: in, Represents the average geometric position of joint point i, which is calculated from the three-dimensional coordinates of the node in frame T in the skeleton sequence: Where, is the 3D coordinate information of joint point i in the t-th frame of the original skeleton input, which can be provided by the pose estimation model or sensor system. The parameter σ>0 is the kernel width that controls the range of geometric similarity attenuation, and δ is the upper limit of the distance that defines the geometric connection, which is used to limit the construction range of non-structural edges and prevent misconnection between unrelated node pairs. To make the computation of graph convolution more numerically stable, the adjacency matrix Normalization is required to avoid feature scale shift caused by uneven node degrees. Symmetric normalization is used for normalization: That is the degree matrix, and the diagonal elements are: Action response graph G a :This graph is used to model the collaborative response relationship between non-structurally connected nodes during movement; First, the motion activation tensor obtained in step 2.1 is Perform average pooling along the time dimension to obtain the response vector of each joint point: In order to express the response difference between nodes, the average L1 distance metric is introduced, and the response difference between any two joint points i and j is defined as: In order to map the difference value into the edge weight form, an exponential decay mapping function is introduced to convert the response difference into the connection strength: Where τ is an adjustable temperature coefficient, which is used to control the influence range of the response difference on the connection strength; Graph Structure Set In order to enhance the model's ability to perceive multiple structural dependencies, K spatial graph structure subsets are introduced. Each subgraph structure is constructed by a different structure-action fusion ratio to reflect the model's attention pattern on different scales or different regions. It is obtained by weighted combination of structure graph and action graph, and the expression is: in, They represent the weight ratios of the structure graph and the action graph in the k graph structures respectively. The weight parameters are learnable variables used to adapt to the dependency preferences during network training. The final graph structure set is: K = number of spatial subsets, preferably 3 or 4.

8. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 7, characterized in that: The step 2 further comprises: Step 2.3, Implementation of dynamic graph structure convolution module: Given input feature X in And the K graph structures generated in step 2.2 First, the input features are divided into S equal-length time periods in the time dimension, and the length of each period is The feature of the i-th segment is expressed as: Next, each segment feature is divided into K sub-channel groups in the channel dimension, and each group of features is recorded as: In each time period i, using the graph structure Guide the corresponding channel group to perform spatial graph convolution operation, and the convolution result is: Concatenate the K-way convolution results in the channel dimension to get the i-th segment output: Finally, the output features of S time periods are spliced ​​in the time dimension to obtain the output of the module:

9. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 1, characterized in that: In step 3, the spatiotemporal motion modeling network TMSN includes 10 spatiotemporal hybrid blocks TSMBlock, each of which is composed of a dynamic motion spatial graph structure DMSG and an adaptive multi-scale temporal modeling module ATSC; each ATSC module includes 4 branches: the first and second branches use time dilation convolution with a convolution kernel size of 5×1, and set two different dilation factors respectively; the third branch is constructed by first dimensionality reduction and then pooling: first, channel compression is performed through a 1×1 convolution layer, then a maximum pooling operation is performed along the time dimension, and finally, another 1×1 convolution is used to restore the channel dimension for extracting local time-invariant features; the fourth branch is a self-attention mechanism based on the time dimension, which first maps the input features into three tensors: query, key, and value, calculates the global time dependency within each frame, and fuses them in an attention-weighted form to enhance the global perception ability of temporal modeling; The outputs of the above four branches are leveled in the channel dimension, and then the channel attention mechanism is introduced to generate fusion weights, and the features of each channel are weighted and adjusted. Finally, convolution is performed to perform channel compression as the final output of the module.

10. The method for on-site personnel behavior recognition based on human skeleton key points according to claim 9, characterized in that: In the spatiotemporal motion modeling network (TMSN), the number of output channels in the first four blocks is 64. In the fifth and eighth spatiotemporal blocks, the number of output channels is multiplied by 2, and downsampled in the temporal dimension. In each block, batch normalization (BatchNorm) and ReLU activation functions are combined to improve the training efficiency and stability of the network. The extracted features are aggregated through a global average pooling layer to compress the temporal and spatial dimensions. Finally, a Softmax classifier is used to predict the action category, and the classification results are used for on-site personnel behavior recognition.

Images