Method for recognizing personnel action in industrial scene by improving time dynamic graph neural network

Abstract

The present application relates to the technical field of industrial production site safety management, in particular to an improved personnel action recognition method in an industrial scene based on a time dynamic graph neural network, comprising: obtaining body key position data of a to-be-recognized personnel in an industrial background; inputting the body key position data into a time dynamic graph neural network action recognition model based on a multi-scale collaborative attention, and outputting an action recognition result of the to-be-recognized personnel, wherein the action recognition model is obtained by training a training set, the training set comprises body key position data and corresponding action category labels of a plurality of personnel in an industrial background, and the action recognition model is constructed based on a multi-scale collaborative attention convolution network, a dynamic graph neural network, a differentiable layer graph pooling network and a combined loss function. The present application can improve the learning ability of complex spatio-temporal patterns in time series data, and realize accurate recognition of the actions of operators in the industrial production process.

Description

Technical Field

[0001] This invention relates to the field of industrial production site safety management technology, and in particular to an improved time-dynamic graph neural network method for personnel action recognition in industrial scenarios. Background Technology

[0002] Industrial production holds a vital position in the national economic industrial chain, serving as a key pillar for promoting high-quality economic development in my country and playing a crucial role in stabilizing economic growth and meeting the growing needs of residents for a better life. However, some raw materials used in industrial production possess dangerous properties such as flammability, explosiveness, and high toxicity. Safety issues during processing or storage can lead to incalculable economic and property losses, and even casualties.

[0003] Industrial production sites typically present complex working environments, dense personnel flow, and diverse safety risks. Human factors are one of the main causes of safety accidents in industrial production sites because industrial processes are generally complex, such as controlling the temperature and flow rate of inlets / outlets, and controlling the temperature, pressure, and flow rate inside reactors. This places high demands on employee operations. Traditional personnel safety management relies on manual inspections and fixed-rule monitoring, which is insufficient for real-time and comprehensive detection and response to various safety threats, especially in densely populated and dynamically changing environments. Therefore, there is an urgent need for methods that can dynamically sense personnel behavior, predict risks, and issue timely warnings. Summary of the Invention

[0004] The purpose of this invention is to provide an improved method for human action recognition in industrial scenarios using a time-dynamic graph neural network. By employing a multi-scale collaborative attention convolutional network, a dynamic graph neural network, differentiable hierarchical graph pooling, and a combined loss function method, the method enhances the ability to learn complex spatiotemporal patterns in time series data, thereby achieving accurate recognition of operator actions in industrial production processes.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An improved method for human action recognition in industrial scenarios using time-dynamic graph neural networks includes:

[0007] Acquire key body location data of individuals to be identified in an industrial setting;

[0008] The key body position data is input into a time-dynamic graph neural network action recognition model with multi-scale collaborative attention, and the action recognition result of the person to be identified is output. The action recognition model is obtained by training a training set, which includes key body position data of several people in an industrial background and corresponding action category labels. The action recognition model is constructed based on a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable hierarchical graph pooling network, and a combined loss function.

[0009] Optionally, before training the action recognition model using the training set, the method further includes preprocessing the training set, wherein the preprocessing includes:

[0010] The key body location data of several individuals in the industrial background of the training set are filtered and segmented using windowing.

[0011] Optionally, the action recognition model includes: a multi-scale collaborative attention convolution module, a dynamic graph neural module, a differentiable hierarchical graph pooling module, and a combined loss function module. The multi-scale collaborative attention convolution module is used to extract multi-scale spatiotemporal feature vectors from the key body location data. The dynamic graph neural module is used to construct and transform a dynamic graph based on the multi-scale spatiotemporal feature vectors to obtain a feature map containing relationships between nodes. The differentiable hierarchical graph pooling module is used to perform differentiable hierarchical graph pooling on the feature map to obtain output features. The combined loss function module is used to calculate the loss value using a combined loss function to optimize the performance of the action recognition model, and uses a softmax function to convert the output features into a classification probability distribution, outputting the probability of each action category to obtain the action classification result.

[0012] Optionally, the multi-scale collaborative attention convolution module includes: a multi-scale dilated convolution unit and a collaborative attention unit, wherein the multi-scale dilated convolution unit is used to extract parallel signal feature vectors from the key body location data and concatenate them to obtain a concatenated signal feature vector; the collaborative attention unit is used to process the concatenated signal feature vector using a collaborative attention mechanism to obtain the multi-scale spatiotemporal feature vector.

[0013] Optionally, the multi-scale dilated convolutional unit includes three parallel branches, each branch including a dilated convolutional layer, a batch norm layer and a first activation function layer connected in sequence.

[0014] Optionally, the dynamic graph neural module includes: a graph construction unit, a dynamic graph transformation unit, and a dynamic graph isomorphism unit, wherein the graph construction unit is used to construct a graph from the multi-scale spatiotemporal feature vector by encoding an adjacency matrix to obtain a graph adjacency matrix; the dynamic graph transformation unit is used to perform information transfer and feature aggregation between nodes to obtain an updated graph adjacency matrix; and the dynamic graph isomorphism unit is used to perform information aggregation on the updated graph adjacency matrix to obtain a feature graph containing relationships between nodes.

[0015] Optionally, the differentiable hierarchical graph pooling module employs a two-dimensional convolutional neural network layer, which is used to cluster the nodes of the feature map into clusters using a preset pooling ratio.

[0016] Optionally, the combined loss function includes cross-entropy loss and center loss, and the combined loss function is:

[0017] Total loss function = L e +L c ·λ

[0018]

[0019]

[0020] In the formula, L e For cross-entropy loss, L c Let y be the center loss, λ be the weight of the center loss, and y be the center loss. ij This represents the probability that sample i belongs to class j. This represents the probability value that i belongs to the j-th class in the probability distribution predicted by the model. Indicates category y i The center of all features, x i Let represent the i-th sample, k be the total number of categories, and m be the total number of samples.

[0021] The beneficial effects of this invention are as follows:

[0022] Compared with traditional graph neural network methods, the action recognition model used in this invention has significant advantages in capturing complex spatial relationships between nodes and can extract more robust feature representations.

[0023] The dynamic graph neural network designed in this invention includes dynamic graph transformation and dynamic graph isomorphism network. They work together to capture spatiotemporal correlations. Dynamic graph transformation reflects the changes in graph structure between different time steps by updating the adjacency matrix, which helps to capture the dynamic characteristics of actions evolving over time. Dynamic graph isomorphism network establishes initial relationships between different time steps, ensuring that the model can effectively process data at different time points and effectively capture complex spatial relationships between nodes.

[0024] The multi-degree collaborative attention convolutional network designed in this invention enhances the ability to capture input features, thereby extracting richer hidden information. Dilated convolution can expand the receptive field and capture spatial dependencies over longer distances, which is crucial for understanding the interactions between joints during movement. The collaborative attention mechanism helps the model focus on more relevant features and reduces interference from irrelevant information, thus extracting richer hidden information.

[0025] The combined loss function designed in this invention combines cross-entropy loss and center loss, improving the model's generalization ability and accuracy. Cross-entropy loss optimizes the classification task itself, while center loss increases the separation between different categories, making similar samples more compact in the feature space. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the improved time-dynamic graph neural network method for human action recognition in industrial scenarios according to an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram showing the IMU wearing position when collecting data on human motion datasets in an industrial context, according to an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the action recognition model structure based on a time-dynamic graph neural network with multi-scale collaborative attention according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the multi-scale collaborative attention convolution module structure according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the collaborative attention unit structure in the multi-scale collaborative attention convolution module of this invention.

[0032] Figure 6 This is a schematic diagram of the dynamic graph neural module structure according to an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the dynamic graph transformation unit structure in the dynamic graph neural module of this invention.

[0034] Figure 8This is a schematic diagram of the combined loss function module structure according to an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] This embodiment provides an improved time-dynamic graph neural network-based method for human action recognition in industrial scenarios, including:

[0038] This paper acquires key body position data of individuals to be identified in an industrial setting. This data is then input into a temporal dynamic graph neural network (TMN) action recognition model with multi-scale collaborative attention. The output is the action recognition result of the individuals to be identified. The action recognition model is trained using a training set that includes key body position data of several individuals in an industrial setting and their corresponding action category labels. The model is constructed using a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable hierarchical graph pooling network, and a combined loss function. Compared to traditional graph neural network methods, this action recognition model has significant advantages in capturing complex spatial relationships between nodes and can extract more robust feature representations.

[0039] Specifically, the method acquires key body position data of the person to be identified in an industrial setting, as well as key body position data of several people in an industrial setting included in the training set. Wearable sensors are fixed to key body positions such as the arms and legs of the person, and 3-axis acceleration and 3-axis gyroscope data are collected at each key body position. Since the inherent noise of the sensors and the unconscious shaking of the tester can cause noise in the collected data, it is necessary to filter and denoise the collected data before feature extraction. This embodiment selects a Kalman filter and uses window segmentation technology to divide the filtered data into several data segments.

[0040] like Figure 3As shown, the action recognition model includes: a multi-scale collaborative attention convolution module, a dynamic graph neural module, a differentiable hierarchical graph pooling module, and a combined loss function module. The multi-scale collaborative attention convolution module extracts multi-scale spatiotemporal feature vectors from the key body location data; the dynamic graph neural module constructs and transforms a dynamic graph based on the multi-scale spatiotemporal feature vectors to obtain a feature map containing relationships between nodes; the differentiable hierarchical graph pooling module performs differentiable hierarchical graph pooling on the feature map to obtain output features; and the combined loss function module calculates the loss value using a combined loss function to optimize the performance of the action recognition model, and uses the softmax function to convert the output features into a classification probability distribution, outputting the probability of each action category to obtain the action classification result.

[0041] 1. The multi-scale collaborative attention convolution module includes: a multi-scale dilated convolution unit and a collaborative attention unit. The multi-scale dilated convolution unit extracts parallel signal feature vectors from key body location data and concatenates them to obtain a concatenated signal feature vector. The collaborative attention unit processes the concatenated signal feature vector using a collaborative attention mechanism to obtain a multi-scale spatiotemporal feature vector. The structure of the multi-scale collaborative attention convolution module is as follows: Figure 4 As shown.

[0042] Specifically, the implementation steps of the multi-scale collaborative attention convolution module in this embodiment are as follows:

[0043] (1) In order to capture the multi-scale features in the input data X, dilated convolutional layers with different dilation coefficients are used for convolution processing to obtain three parallel signal feature vectors F1; F2; F3;

[0044]

[0045] Where ω(a,b) are elements of the convolution kernel, h and w are the height and width of the convolution kernel, respectively, m and n are the row and column indices of the convolution kernel, and q is the dilation coefficient.

[0046] (2) Further, the three parallel feature vectors are concatenated along the height direction to obtain the concatenated signal feature vector:

[0047] F concat =CONCAT[F1;F2;F3]∈R C×3H×W

[0048] (3) In order to extract features more effectively in the channel and spatial dimensions and enhance the attention to features at different scales, the spliced ​​feature vectors are processed by a collaborative attention mechanism to obtain multi-scale spatiotemporal feature vectors.

[0049] The multi-scale dilated convolutional unit that achieves the above functionality consists of three parallel branches, each composed of a stacked dilated convolutional layer, a batch norm layer, and a rectified linear unit (ReLU) activation function layer. Specifically, dilated convolution expands the receptive field by adding zero elements inside the convolutional kernel, thereby capturing more information hidden in the input features. However, since the number of parameters in the convolutional kernel does not increase, dilated convolution does not consume additional computational resources, effectively avoiding overfitting.

[0050] Considering that excessive expansion rates might lead to loss of local correlations, the expansion rates for the three branches are set to 1, 3, and 5, respectively. The feature vectors output from the three branches are stacked along the channel dimension to form a unified feature vector. This vector is then averaged in both the horizontal and vertical directions using a multi-scale collaborative attention mechanism to encode spatial information. Finally, the spatial information is fused through channel-weighted aggregation. The use of collaborative attention units not only captures channel information but also obtains information sensitive to both direction and position, enabling the model to more accurately locate and identify target regions. The structure of the collaborative attention unit is as follows: Figure 5 As shown, the specific steps are as follows:

[0051] ① Coordinate Information Embedding: To facilitate the capture of spatial long-range dependencies with precise location information by the collaborative attention unit, global pooling is decomposed into a one-to-one one-dimensional feature encoding operation. The input features are first encoded along the horizontal and vertical coordinate directions using pooling kernels of sizes (h, 1) and (1, w). The output formula for the c-th channel at height h and width w is:

[0052]

[0053]

[0054] in This represents the output of the c-th channel at height h; This represents the output of the c-th channel at width h. The two transformations above aggregate features along two spatial directions, producing a pair of direction-aware feature maps. They also preserve precise location information along the other spatial direction, which helps the network more accurately locate the target of interest.

[0055] ② Feature vector concatenation: In order to better utilize the feature vectors embedded with coordinate information, the feature vectors are concatenated... and The features are concatenated along the channel dimension and then reduced in dimensionality using a 1×1 convolution (K1) to obtain the intermediate feature vector f:

[0056]

[0057] ③Segmentation and Transformation: Divide f into two separate tensors f along the spatial dimension. h ∈R C / r×H and f w ∈R C / r×W Where r is the compression ratio. Then, two 1×1 convolutions (K...) are used... h and K w For the eigenvector f h and f w The transformation is performed to ensure that the number of channels matches the input features. Then, the attention weight matrix is ​​generated using the Sigmoid function, resulting in the following:

[0058] g h =σ(K) h (f h ))

[0059] g w =σ(K) w (f w ))

[0060] ④ Multiplication operation: In order to perform element-wise multiplication with the concatenated signal feature vector, the attention weights are expanded to increase the one-dimensional vector g. h and g w Extended to a two-dimensional matrix:

[0061] G h =expand(g h ,w)∈R C×H×W

[0062] G W =expand(g w ,h)∈R C×H×W

[0063] The final output of the collaborative attention unit can be expressed as follows:

[0064] y c (c,i,j)=x c (c,i,j)×G h (c,i,j)×G W (c,i,j)

[0065] Specifically, the multi-scale collaborative attention convolutional network designed in this embodiment enhances the ability to capture input features, thereby extracting richer hidden information. Dilated convolution can expand the receptive field and capture spatial dependencies over longer distances, which is crucial for understanding the interactions between joints in a movement. The collaborative attention mechanism helps the model focus on more relevant features and reduces interference from irrelevant information, thus extracting richer hidden information.

[0066] 2. The dynamic graph neural module includes: graph construction unit, dynamic graph transformation unit, and dynamic graph isomorphism unit. The structure is as follows: Figure 6 As shown in the diagram, the graph construction unit is used to construct a graph from multi-scale spatiotemporal feature vectors by encoding the adjacency matrix, thereby obtaining the graph adjacency matrix; the dynamic graph transformation unit is used to perform information transfer and feature aggregation between nodes, thereby obtaining the updated graph adjacency matrix; and the dynamic graph isomorphism unit is used to perform information aggregation on the updated graph adjacency matrix, thereby obtaining a feature graph containing the relationships between nodes.

[0067] Specifically, the implementation steps of the dynamic graph neural module in this embodiment are as follows:

[0068] (1) Considering that general time series data lacks a predefined graph structure, graph building units are used to generate an adjacency matrix to create initial associations in dimensions, thereby representing hidden dependencies between variables. The adjacency matrix is ​​optimized through training iterations to capture dynamic relationships between variables. For simplicity, this relationship is not established through time series data, but rather the adjacency matrix is ​​encoded through "shallow" embeddings. Two values ​​are assigned to each node, representing the source node and the target node, respectively. Specifically, two vectors Θ of length d are generated for each time slot t. T And Ψ, and all elements are randomly initialized learnable parameters. Calculate Θ. T The product of Ψ and Ψ is used as the initial adjacency matrix of the time slot. The calculation process is as follows:

[0069] A = Θ T ·Ψ

[0070] idx,idy=argtopk(A[:,:])idx≠idy

[0071] A[-idx,-idy]=0

[0072] Where, Θ=[θ t,1 ,θ t,2 ,…,θ t,d ], This represents a random initialization of the learnable node embeddings. argtopk(.) returns the indices of the first k largest values ​​in the adjacency matrix A. For each time slot's adjacency matrix, only the elements with the first k largest weights are retained, and the others are set to zero.

[0073] (2) To capture the dynamic characteristics of actions evolving over time, a dynamic graph transformation unit is used to process the graph adjacency matrix. The dynamic graph transformation network mainly studies the information transmission and feature aggregation between nodes. Although some GNN models can propagate neighbor messages, existing methods can only operate on static graphs. Therefore, a new dynamic graph message propagation model based on an improved graph isomorphic network is proposed.

[0074] To achieve discretization, a connection is constructed between each graph and the graph corresponding to the previous time slot. The structure of the dynamic graph transformation unit is as follows: Figure 7 As shown. For each time slot graph, except for the first graph, the same number of vertices are added, each describing the data of the corresponding vertex in the previous time slot graph. Therefore, the set of vertices will be modified to {v (t,1) ,v (t,2) ,...,v (t,N) ,v (t-1,1) ,v (t-1,1) ,…,v (t-1,N) Assign the orientation (i.e., directed edge) of the vertex from the previous time step to the corresponding vertex at the current time step. That is, in the case of n = 1, 2, ..., N, add the orientation from v to the vertex. (t-1,n) to v (t,n) The connection is then established. The newly generated atlas can then be passed to the next step.

[0075] (3) To capture the complex spatial relationships between nodes, the updated adjacency matrix is ​​input into the dynamic graph isomorphic unit. The dynamic graph isomorphic unit employs a novel information aggregation method, which consists of a dynamic paradigm implemented in parallel. Unlike static GNNs, dynamic GNNs completely separate data from different time slots of the same dimension after dynamic graph transformation. In reality, the same vertex typically aggregates information from different vertex sets in different time slots. The dynamic graph isomorphic network can be defined as:

[0076]

[0077]

[0078] To eliminate the influence of different edge weights, the edge weight ω ij Normalization To ensure the smoothness and stability of information transmission. Among these, It is a simple implementation of dynamic graph transformation, applicable to t>2 (the second and subsequent graphs). This represents the output of the layer graph isomorphic network of node v within the time slot at time t.

[0079] Specifically, the dynamic graph neural network designed in this embodiment includes dynamic graph transformation and dynamic graph isomorphism network. They work together to capture spatiotemporal correlations. Dynamic graph transformation reflects the changes in graph structure between different time steps by updating the adjacency matrix, which helps to capture the dynamic characteristics of actions evolving over time. Dynamic graph isomorphism network establishes initial relationships between different time steps, ensuring that the model can effectively process data at different time points and effectively capture complex spatial relationships between nodes.

[0080] 3. The differentiable hierarchical graph pooling module uses a two-dimensional convolutional neural network layer, which is used to cluster nodes of the feature map into clusters by a preset pooling ratio.

[0081] Specifically, in multivariate time series analysis, the vectors of each variable need to be pooled at the end. However, some common pooling methods, such as max pooling and average pooling, may lose a significant amount of information. To retain more effective information, a differentiable hierarchical graph pooling module is applied after each GNN layer to achieve more refined information transfer. The differentiable hierarchical graph pooling module designs a two-dimensional convolutional neural network (CNN) layer that clusters nodes into clusters according to a given pooling ratio. Nodes are considered as channels for feature extraction in the CNN. To avoid disrupting the receptive field window, the kernel size of the temporal convolution is allocated to this convolution kernel size. The specific steps are as follows:

[0082] (1) First, an efficient 2D cross-correlation operator is used to process the input node embedding tensor, as shown in the following equation. The input node embedding tensor can be viewed as a feature map, where the features of each node are encoded in a vector. The purpose of the cross-correlation operator is to extract useful information from this feature map and aggregate this information into a smaller number of output nodes.

[0083]

[0084] in, For an efficient 2D cross-correlation operator, N l Equal to the number of input nodes in the l-th layer, N l+1 Equal to the number of output nodes in layer l, X l This represents the embedding tensor of the input node in the l-th layer.

[0085] (2) Generate output tensor X l+1 Next, consider calculating the corresponding adjacency matrix. The learnable weights W... l The shape is [N l+1 N l ,1,kernel size]], and then generate a vector V consisting of the learnable parameters of the l-th layer. l ∈R 1×k A learnable assignment matrix can be obtained. The row corresponds to N l+1 There are N nodes or clusters, and the columns correspond to N. l There are several clusters. Given a matrix M of input data for this layer. (l) and adjacency matrix A (l) The following equation is used to generate the output adjacency matrix:

[0086] A (l+1) =M (l) A (l) M(l)T

[0087] Among them, A (l+1) This represents the connection relationships and corresponding weights of the new cluster.

[0088] 4. The combined loss function module: The output feature vector is input into the combined loss function module to train the model and achieve accurate classification of different actions. To enhance the separability of features and improve recognition performance, a combined loss function integrating cross-entropy loss and center loss is designed. This addresses the problem that single cross-entropy loss cannot fully explore the feature space structure, while overcoming the limitation of center loss in working independently in unsupervised situations. Cross-entropy loss is used to classify labels, measuring the difference between the predicted probability distribution and the actual probability distribution; while center loss optimizes the distribution of feature vectors, making feature vectors of different categories more clustered in the space. The cross-entropy loss function measures the difference between the probability distribution predicted by the model and the true label. By minimizing this difference, the model's learning process is more inclined to generate predictions closer to the true label. Its calculation formula is: The calculation formula for the cross-entropy loss function is:

[0089]

[0090] in, y represents the probability value of i belonging to class j in the probability distribution predicted by the model. ij Let y represent the probability that sample i belongs to class j. ij =1 indicates that it belongs to, y ij =0 indicates that it does not belong. Because for each sample, only one label is correct, the actual meaning of this loss function is to penalize the model's prediction result with the correct label.

[0091] To enhance the compactness of intra-class sample features and reduce intra-class variance, a center loss function is added after the cross-entropy loss function. The center loss optimizes the distribution of feature vectors, causing samples within the same class to cluster more tightly in the feature space, thereby improving the classifier's generalization ability and robustness. The formula for the center loss function is as follows:

[0092]

[0093] in, The representative category is y i The center of all features, x i Let represent the i-th sample. Specifically, during training, the center of the features for each class is statistically analyzed at a mini-batch scale, aiming to minimize the distance of all features from the center.

[0094] For the training process, the cross-entropy loss function and the center loss function are calculated separately, and then the two are weighted and combined to obtain the final total loss function. The structure diagram of the loss function is as follows. Figure 8 As shown, the specific formula is as follows:

[0095] Total loss function = L e +L c ·λ

[0096] Here, λ represents the weight of the center loss, which can be set manually based on the specific task and dataset. By adjusting the value of λ, the influence of cross-entropy loss and center loss during training can be balanced. This loss function design allows the model to maintain good classification performance in classification tasks while learning more robust feature representations, thus improving the model's generalization ability and accuracy.

[0097] Specifically, the combined loss function designed in this embodiment combines the cross-entropy loss function and the center loss function, which improves the generalization ability and accuracy of the model. The cross-entropy loss is used to optimize the classification task itself, while the center loss is used to increase the separation between different categories, making similar samples more compact in the feature space.

[0098] The following is combined Figure 1 The training and testing process of the action recognition model based on a time-dynamic graph neural network with multi-scale collaborative attention proposed in this embodiment is described in detail, including:

[0099] Step S1: Identify commonly used actions in an industrial context, which are 13 categories: walking, standing, turning valves, going up and down stairs, making a phone call, sitting, crawling, climbing ladders, welding, cleaning, lying down and calling for help, falling forward, and falling backward.

[0100] Step S2: Secure the four wearable sensors to key body locations such as the person's arms and legs, as shown in the image. Figure 2 As shown, 3-axis acceleration and 3-axis gyroscope data were collected at each key body position; to ensure data consistency over time, the acquisition frequency for both acceleration and gyroscope was 148Hz.

[0101] Step S3: Filter and segment the personnel movement data in the industrial scenario from Step S2;

[0102] Since the inherent noise of the sensor itself and the unconscious shaking of the tester can both cause noise in the collected data, it is necessary to filter and denoise the collected data before feature extraction. In this embodiment, a Kalman filter is selected.

[0103] The filtered data was divided into several data segments using a window segmentation technique. A sliding window with a length of 860ms and an overlap rate of 50% was used to segment the samples, resulting in training and testing samples. These segments were then combined to form a sample set, which was divided into training and testing sets in a 7:3 ratio.

[0104] Step S4: Input the action sample training set data into the action recognition model based on the time dynamic graph neural network with multi-scale collaborative attention, and train the model.

[0105] Step S5: Input the data from the action sample test set into the trained action recognition model based on a time-dynamic graph neural network with multi-scale collaborative attention, and perform action classification to obtain the classification results.

[0106] The key hyperparameter settings in this embodiment are shown in Table 1:

[0107] Table 1

[0108]

[0109] The performance of the model is evaluated using test data samples. Specifically, the model's accuracy, precision, recall, and F1 score for recognizing various actions on the test data are calculated.

[0110] First, the proposed algorithm model is compared and analyzed with the advanced algorithm model Todynet on a human action dataset in an industrial context to demonstrate its superiority.

[0111] Secondly, to demonstrate the effectiveness of this embodiment in improving each step of the process of constructing a time-dynamic graph neural network action recognition model with multi-scale collaborative attention, an ablation experiment was conducted for verification.

[0112] The specific experimental strategy is as follows:

[0113] Step (1): Validate a common graph neural network on a human action dataset in an industrial context;

[0114] Step (2): Based on the model in step (1), embed the dilated convolution module into the convolutional neural network and validate it on a human motion dataset in an industrial context;

[0115] Step (3): Based on the model in step (2), the collaborative attention module is embedded into the dilated convolution module and validated on a human action dataset in an industrial context;

[0116] Step (4): Based on the model in step (3), the loss function module is changed to a combination of cross-entropy loss function and center loss function and validated on a human action dataset in an industrial context.

[0117] Experiment execution strategy:

[0118] This embodiment first collects a dataset of human actions in an industrial context and then performs filtering on the dataset.

[0119] The preprocessed dataset is divided into a training set and a test set in a 7:3 ratio. The training set is used as the input training data for model feature extraction and parameter training. The test set can be used to verify the model's predicted values ​​and to fine-tune the model parameters.

[0120] First, the training dataset is fed into a multi-scale collaborative attention convolutional module for feature extraction. Then, the multi-scale spatiotemporal feature vectors are fed into a dynamic graph neural module to obtain graph feature vectors. Finally, the graph feature vectors are input into a fully connected layer for regression analysis and loss function calculation. Through these steps, local and periodic features of the data are extracted, and the model parameters are gradually optimized during training, completing the model training.

[0121] Finally, by combining the extracted features with the model regression analysis results, the worker's action type was identified.

[0122] The temporal dynamic graph neural network action recognition model with multi-scale collaborative attention employed in this invention provides a framework that comprehensively considers spatial, temporal, and dynamic changes, enabling more effective recognition of human actions. Through dynamic graph transformation, the action recognition model can better capture the dynamic correlations between different time points, which is crucial for understanding the evolution of actions over time. By employing dilated convolution and collaborative attention mechanisms, the action recognition model can extract more robust feature representations, which are more adaptable to different perspectives and environmental changes. Using a specifically designed combination of loss functions, the temporal dynamic graph neural network action recognition model with multi-scale collaborative attention not only improves the model's accuracy but also enhances its generalization ability, making it more stable across different datasets.

[0123] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An improved method for human action recognition in industrial scenarios using a time-dynamic graph neural network, characterized in that, include: Acquire key body location data of individuals to be identified in an industrial setting; The key body position data is input into a time-dynamic graph neural network action recognition model with multi-scale collaborative attention, and the action recognition result of the person to be identified is output. The action recognition model is obtained by training a training set, which includes key body position data of several people in an industrial background and corresponding action category labels. The action recognition model is constructed based on a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable hierarchical graph pooling network, and a combined loss function. The action recognition model includes: a multi-scale collaborative attention convolution module, a dynamic graph neural module, a differentiable hierarchical graph pooling module, and a combined loss function module. The multi-scale collaborative attention convolution module extracts multi-scale spatiotemporal feature vectors from the body's key position data. The dynamic graph neural module constructs and transforms a dynamic graph based on the multi-scale spatiotemporal feature vectors to obtain a feature map containing relationships between nodes. The differentiable hierarchical graph pooling module performs differentiable hierarchical graph pooling on the feature map to obtain output features. The combined loss function module calculates the loss value using a combined loss function to optimize the performance of the action recognition model and uses a softmax function to convert the output features into a classification probability distribution, outputting the probability of each action category to obtain the action classification result. The multi-scale collaborative attention convolution module includes a multi-scale dilated convolution unit and a collaborative attention unit. The multi-scale dilated convolution unit is used to extract parallel signal feature vectors from the key body location data and concatenate them to obtain a concatenated signal feature vector. The collaborative attention unit is used to process the concatenated signal feature vector using a collaborative attention mechanism to obtain the multi-scale spatiotemporal feature vector. The multi-scale dilated convolutional unit includes three parallel branches, each branch including a dilated convolutional layer, a batch norm layer and a first activation function layer connected in sequence; The dynamic graph neural module includes: a graph construction unit, a dynamic graph transformation unit, and a dynamic graph isomorphism unit. The graph construction unit constructs a graph from the multi-scale spatiotemporal feature vectors using an encoded adjacency matrix to obtain a graph adjacency matrix. The dynamic graph transformation unit performs information transfer and feature aggregation between nodes to obtain an updated graph adjacency matrix. The dynamic graph isomorphism unit aggregates information from the updated graph adjacency matrix to obtain a feature graph containing relationships between nodes. The differentiable hierarchical graph pooling module employs a two-dimensional convolutional neural network layer, which is used to cluster the nodes of the feature map into clusters using a preset pooling ratio.

2. The method for human action recognition in industrial scenarios using an improved time-dynamic graph neural network according to claim 1, characterized in that, Before training the action recognition model using the training set, the method further includes preprocessing the training set, wherein the preprocessing includes: The key body location data of several individuals in the industrial background of the training set are filtered and segmented using windowing.

3. The method for human action recognition in industrial scenarios using an improved time-dynamic graph neural network according to claim 1, characterized in that, The combined loss function includes cross-entropy loss and center loss, and the combined loss function is as follows: In the formula, For cross-entropy loss, Loss at the center The weights for the center loss, Indicates sample i Belongs to the j The probability of a class In the probability distribution predicted by the model i Belongs to the j The probability value of the class. Indicates category as The center of all features, Indicates the first i One sample, k The total number of categories, m The total number of samples.

Images