Personnel action recognition method in industrial scene based on improved time dynamic graph neural network

Abstract

The invention relates to the technical field of industrial production field safety management, in particular to an industrial scene personnel action recognition method based on an improved time dynamic graph neural network, and the method comprises the steps: obtaining the body key position data of a to-be-recognized personnel under an industrial background; and inputting the body key position data into a neural network action recognition model based on a time dynamic graph with multi-scale collaborative attention, and outputting an action recognition result of the to-be-recognized person, the action recognition model being obtained through training of a training set, and the action recognition result being obtained through training of the training set. The training set comprises body key position data of a plurality of persons under the industrial background and corresponding action category labels, and the action recognition model is constructed based on a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a micro-hierarchical graph pooling network and a combined loss function. According to the method, the learning ability of the complex space-time pattern in the time sequence data can be improved, and the accurate recognition of the actions of operators in the industrial production process is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial production site safety management, and in particular to a method for identifying human motion in industrial scenarios using an improved temporal dynamic graph neural network. Background Art

[0002] Industrial production plays a very important role in the national economic industrial chain. It is the key and key pillar to promote the high-quality development of my country's economy. It plays an important role in stabilizing economic growth and meeting the growing needs of residents for a better life. However, some raw materials in industrial production have dangerous characteristics such as flammable, explosive and highly toxic. If safety problems occur during processing or storage, it will cause immeasurable economic and property losses and even casualties.

[0003] Industrial production sites usually have complex working environments, dense personnel flows, and diverse safety risks. Human factors are one of the main causes of safety accidents at industrial production sites, because the process flow of industrial production is mostly complex, such as the control of inlet / outlet temperature and flow rate, the control of temperature, pressure, and flow rate in the reactor, etc., which puts high demands on employee operations. Traditional personnel safety management relies on manual inspections and fixed rule monitoring, which makes it difficult to capture and respond to various security threats in real time and comprehensively, especially in a densely populated and dynamically changing environment. Therefore, there is an urgent need for methods that can dynamically perceive personnel behavior, predict risks, and issue early warnings in a timely manner. Summary of the invention

[0004] The purpose of the present invention is to provide a method for identifying personnel actions in industrial scenarios based on an improved temporal dynamic graph neural network. By adopting a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable layered graph pooling and a combined loss function method, the learning ability of complex spatiotemporal patterns in time series data is improved, and accurate identification of the actions of operators in industrial production processes can be achieved.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] Improve the human action recognition method in industrial scenarios based on time-dynamic graph neural network, including:

[0007] Obtain key body position data of the person to be identified in an industrial context;

[0008] The key body position data is input into a temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention, and the action recognition result of the person to be identified is output, wherein the action recognition model is obtained by training with a training set, and the training set includes the key body position data of several people in an industrial background and corresponding action category labels, and the action recognition model is constructed based on a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable layered graph pooling network and a combined loss function.

[0009] Optionally, before training the action recognition model using the training set, the method further includes preprocessing the training set, wherein the preprocessing includes:

[0010] The key body position data of several persons in the industrial background in the training set are filtered and windowed.

[0011] Optionally, the action recognition model includes: a multi-scale collaborative attention convolution module, a dynamic graph neural module, a differentiable layered graph pooling module, and a combined loss function module, wherein the multi-scale collaborative attention convolution module is used to extract the multi-scale spatiotemporal feature vectors of the key position data of the body; the dynamic graph neural module is used to construct and transform the dynamic graph based on the multi-scale spatiotemporal feature vector to obtain a feature graph containing the relationship between nodes; the differentiable layered graph pooling module is used to perform differentiable layered graph pooling on the feature graph to obtain output features; the combined loss function module is used to calculate the loss value using a combined loss function to optimize the performance of the action recognition model, and use the softmax function to convert the output features into classification probability distributions, output the probability of each action category, and obtain action classification results.

[0012] Optionally, the multi-scale collaborative attention convolution module includes: a multi-scale void convolution unit and a collaborative attention unit, wherein the multi-scale void convolution unit is used to extract parallel signal feature vectors of the body key position data and splice them to obtain the spliced ​​signal feature vector; the collaborative attention unit is used to perform collaborative attention mechanism processing on the spliced ​​signal feature vector to obtain the multi-scale spatiotemporal feature vector.

[0013] Optionally, the multi-scale dilated convolution unit includes three parallel branches, each branch including a dilated convolution layer, a batch norm layer and a first activation function layer connected in sequence.

[0014] Optionally, the dynamic graph neural module includes: a graph construction unit, a dynamic graph transformation unit, and a dynamic graph isomorphism unit, wherein the graph construction unit is used to construct a graph of the multi-scale spatiotemporal feature vector by encoding the adjacency matrix to obtain a graph adjacency matrix; the dynamic graph transformation unit is used to perform information transmission and feature aggregation between nodes to obtain an updated graph adjacency matrix; the dynamic graph isomorphism unit is used to perform information aggregation on the updated graph adjacency matrix to obtain a feature graph containing relationships between nodes.

[0015] Optionally, the differentiable layered graph pooling module adopts a two-dimensional convolutional neural network layer, and the two-dimensional convolutional neural network layer is used to cluster the nodes of the feature graph through a preset pooling ratio.

[0016] Optionally, the combined loss function includes cross entropy loss and center loss, and the combined loss function is:

[0017] Total loss function = L e +L c ·λ

[0018]

[0019]

[0020] Where, L e is the cross entropy loss, L c is the center loss, λ is the weight of the center loss, y ij represents the probability that sample i belongs to the jth class, It represents the probability value that i belongs to the jth category in the probability distribution predicted by the model, Indicates that the category is y i The center of all features, x i represents the i-th sample, k is the total number of categories, and m is the total number of samples.

[0021] The beneficial effects of the present invention are:

[0022] Compared with the traditional graph neural network method, the action recognition model adopted by the present invention has significant advantages in capturing the complex spatial relationship between nodes and can extract more robust feature representation;

[0023] The dynamic graph neural network designed by the present invention includes dynamic graph transformation and dynamic graph isomorphic network, which work together to capture spatiotemporal correlation. Dynamic graph transformation reflects the changes in graph structure between different time steps by updating the adjacency matrix, which helps to capture the dynamic characteristics of action evolution over time. The dynamic graph isomorphic network establishes initial relationships between different time steps to ensure that the model can effectively process data at different time points and can effectively capture complex spatial relationships between nodes.

[0024] The multi-cooperative attention convolutional network designed by the present invention enhances the ability to capture input features, thereby extracting richer hidden information. The dilated convolution can expand the receptive field and capture longer-distance spatial dependencies, which is very important for understanding the interaction between joints in action. The cooperative attention mechanism helps the model focus on more relevant features and reduce the interference of irrelevant information, thereby extracting richer hidden information.

[0025] The combined loss function designed by the present invention combines the cross entropy loss function with the center loss function to improve the generalization ability and accuracy of the model. The cross entropy loss is used to optimize the classification task itself, while the center loss is used to increase the separation of different categories, making samples of the same type more compact in the feature space. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a flow chart of a method for identifying human actions in industrial scenarios using an improved time-dynamic graph neural network according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the wearing position of the IMU when collecting data of a human motion data set under an industrial background according to an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the structure of a temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention according to an embodiment of the present invention;

[0030] Figure 4 Schematic diagram of the structure of a multi-scale collaborative attention convolution module in an embodiment of the present invention;

[0031] Figure 5 Schematic diagram of the structure of a collaborative attention unit in a multi-scale collaborative attention convolution module according to an embodiment of the present invention;

[0032] Figure 6 A schematic diagram of the structure of a dynamic graph neural module according to an embodiment of the present invention;

[0033] Figure 7 A schematic diagram of the structure of a dynamic graph transformation unit in a dynamic graph neural module according to an embodiment of the present invention;

[0034] Figure 8Schematic diagram of the structure of the combined loss function module of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] This embodiment provides a method for identifying human actions in industrial scenarios using an improved time-dynamic graph neural network, including:

[0038] The key position data of the body of the person to be identified in the industrial background are obtained, and the key position data of the body are input into the action recognition model based on the time dynamic graph neural network with multi-scale collaborative attention, and the action recognition results of the person to be identified are output. The action recognition model is obtained by training with a training set, which includes the key position data of the body of several people in the industrial background and the corresponding action category labels. The action recognition model is constructed based on a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable hierarchical graph pooling network and a combined loss function. Compared with the traditional graph neural network method, this action recognition model has significant advantages in capturing the complex spatial relationship between nodes and can extract more robust feature representations.

[0039] Specifically, the key body position data of the person to be identified in the industrial background and the key body position data of several persons in the industrial background included in the training set are obtained, and the 3-axis acceleration and 3-axis gyroscope data of each key body position are collected by fixing the wearable sensor at the key body positions such as the arms and legs of the person. And because the inherent noise of the sensor itself and the unconscious shaking of the tester will cause noise in the collected data, the collected data needs to be filtered and denoised before feature extraction. This embodiment selects the Kalman filter and uses the window segmentation technology to divide the filtered data into several data segments.

[0040] like Figure 3As shown, the action recognition model includes: a multi-scale collaborative attention convolution module, a dynamic graph neural module, a differentiable layered graph pooling module, and a combined loss function module, wherein the multi-scale collaborative attention convolution module is used to extract the multi-scale spatiotemporal feature vectors of the key position data of the body; the dynamic graph neural module is used to construct and transform the dynamic graph based on the multi-scale spatiotemporal feature vector to obtain a feature graph containing the relationship between nodes; the differentiable layered graph pooling module is used to perform differentiable layered graph pooling on the feature graph to obtain output features; the combined loss function module is used to calculate the loss value using a combined loss function to optimize the performance of the action recognition model, and use the softmax function to convert the output features into classification probability distributions, output the probabilities of each action category, and obtain action classification results.

[0041] 1. The multi-scale collaborative attention convolution module includes: a multi-scale dilated convolution unit and a collaborative attention unit. The multi-scale dilated convolution unit is used to extract the parallel signal feature vectors of the key position data of the body and splice them to obtain the spliced ​​signal feature vector; the collaborative attention unit is used to perform collaborative attention mechanism processing on the spliced ​​signal feature vector to obtain a multi-scale spatiotemporal feature vector. The structure of the multi-scale collaborative attention convolution module is as follows: Figure 4 shown.

[0042] Specifically, the specific implementation steps of the multi-scale collaborative attention convolution module in this embodiment are:

[0043] (1) In order to capture the multi-scale features in the input data X, a dilated convolution layer with different dilation coefficients is used for convolution processing to obtain three parallel signal feature vectors F1, F2, and F3.

[0044]

[0045] Among them, ω(a,b) is the element of the convolution kernel, h and w are the height and width of the convolution kernel respectively, m and n are the row and column indices of the convolution kernel, and q is the dilation coefficient.

[0046] (2) Further concatenate the three parallel feature vectors in the height direction to obtain a concatenated signal feature vector:

[0047] F concat =CONCAT[F1;F2;F3]∈R C×3H×W

[0048] (3) In order to perform more effective feature extraction in the channel and spatial dimensions and enhance the attention to features of different scales, the concatenated feature vector is processed through a collaborative attention mechanism to obtain a multi-scale spatiotemporal feature vector.

[0049] The multi-scale dilated convolution unit that implements the above functions consists of three parallel branches, each of which is composed of a dilated convolution layer, a batch norm layer, and a rectified linear unit (ReLU) activation function layer. Specifically, dilated convolution can expand the receptive field by adding zero elements inside the convolution kernel, thereby capturing more information hidden in the input features. However, since the number of parameters in the convolution kernel has not increased, dilated convolution does not consume additional computing resources, effectively avoiding overfitting.

[0050] Considering that too large an expansion rate will lose the local correlation of information, the expansion rates of the three branches are set to 1, 3, and 5 respectively. The feature vectors output by the three branches will be stacked on the channel dimension to form an overall feature vector, and average pooled in the horizontal and vertical directions through a multi-scale collaborative attention mechanism, and then the spatial information is encoded, and finally the spatial information is fused in a weighted manner on the channel. The use of collaborative attention units not only captures channel information, but also obtains information that is sensitive to both direction and position, allowing the model to more accurately locate and identify the target area. The structure of the collaborative attention unit is shown in Figure 2. Figure 5 As shown, the specific steps are as follows:

[0051] ① Coordinate information embedding: In order to facilitate the collaborative attention unit to capture spatial long-range dependencies with precise location information, the global pooling is decomposed into a pair of one-dimensional feature encoding operations. The input features are first encoded along the horizontal and vertical coordinate directions using pooling kernels of size (h, 1) and (1, w). The output formula of the cth channel at height h and width w is:

[0052]

[0053]

[0054] in represents the output of the cth channel at height h; Represents the output of the cth channel at width h. The above two transformations aggregate features along two spatial directions respectively, producing a pair of direction-aware feature maps. And save the precise position information along another spatial direction, which helps the network locate the target of interest more accurately.

[0055] ② Feature vector cascade: In order to make better use of the feature vector embedded with coordinate information, and Cascade is performed on the channel dimension and dimensionality reduction is performed through a 1×1 convolution (K1) to obtain the intermediate feature vector f:

[0056]

[0057] ③ Split and transform: Split f into two separate tensors f along the spatial dimension h ∈R C / r×H and f w ∈R C / r×W , where r is the compression ratio. Then use two 1×1 convolutions (K h and K w ) for the eigenvector f h and f w Transform it so that the number of channels is consistent with the input features, and then generate the attention weight matrix through the Sigmoid function to obtain the following results:

[0058] g h =σ(K h (f h ))

[0059] g w =σ(K w (f w ))

[0060] ④Multiplication operation: In order to perform element-level multiplication operation with the concatenated signal feature vector, the one-dimensional vector g is expanded by expanding the attention weight. h and g w Expanded to a two-dimensional matrix:

[0061] G h = expand(g h ,w)∈R C×H×W

[0062] G W = expand(g w ,h)∈R C×H×W

[0063] The final output of the collaborative attention unit can be expressed as follows:

[0064] y c (c,i,j)=x c (c,i,j)×G h (c,i,j)×G W (c,i,j)

[0065] Specifically, the multi-scale collaborative attention convolutional network designed in this embodiment enhances the ability to capture input features, thereby extracting richer hidden information. The hollow convolution can expand the receptive field and capture longer-distance spatial dependencies, which is very important for understanding the interaction between joints in the action. The collaborative attention mechanism helps the model focus on more relevant features and reduce the interference of irrelevant information, thereby extracting richer hidden information.

[0066] 2. The dynamic graph neural module includes: graph construction unit, dynamic graph transformation unit, and dynamic graph isomorphism unit. The structure is as follows Figure 6 As shown. The graph construction unit is used to construct a graph of multi-scale spatiotemporal feature vectors by encoding the adjacency matrix to obtain a graph adjacency matrix; the dynamic graph transformation unit is used to transfer information and aggregate features between nodes to obtain an updated graph adjacency matrix; the dynamic graph isomorphism unit is used to aggregate information on the updated graph adjacency matrix to obtain a feature graph containing the relationship between nodes.

[0067] Specifically, the implementation steps of the dynamic graph neural module in this embodiment are as follows:

[0068] (1) Considering that general time series data does not have a predefined graph structure, a graph building unit is used to generate an adjacency matrix to create an initial association of dimensions, thereby representing the hidden dependencies between variables. The adjacency matrix is ​​optimized through training iterations to capture the dynamic relationship between variables. For simplicity, this relationship is not established through time series data, but the adjacency matrix is ​​encoded through "shallow" embedding. Two values ​​are assigned to each node, representing the source node and the target node respectively. Specifically, two vectors Θ of length d are generated for each time slot t T and Ψ, and all elements are randomly initialized learnable parameters. Compute Θ T The product of and Ψ will be regarded as the initial adjacency matrix of the time slot. The calculation process is as follows:

[0069] A=Θ T ·Ψ

[0070] idx,idy=argtopk(A[:,:])idx≠idy

[0071] A[-idx,-idy]=0

[0072] Where θ = [θ t,1 ,θ t,2 ,…,θ t,d ], represents a random initialization of the learnable node embeddings, and argtopk(.) returns the indices of the top k largest values ​​of the adjacency matrix A. For the adjacency matrix of each slot, only the elements with the top k largest weights are kept and the others are set to zero.

[0073] (2) In order to capture the dynamic characteristics of the action evolution over time, a dynamic graph transformation unit is used to process the above graph adjacency matrix. The dynamic graph transformation network mainly studies the information transmission and feature aggregation between nodes. Although some GNN models can propagate neighbor messages, existing methods can only operate on static graphs. Therefore, a new dynamic graph message propagation model based on an improved graph isomorphism network is proposed.

[0074] For discretization, a connection is constructed for each graph with the graph corresponding to the previous time slot. The structure of the dynamic graph transformation unit is as follows: Figure 7 For each time slot graph, except for the first one, the same number of vertices are added, which describe the data of the corresponding vertices of the previous time slot graph. Therefore, the set of vertices will be modified to {v (t,1) ,v (t,2) ,...,v (t,N) ,v (t-1,1) ,v (t-1,1) ,…,v (t-1,N)}. Assign the direction (i.e. directed edge) of the vertex at the previous moment to the corresponding vertex at the current moment, that is, when n = 1, 2, ..., N, add the direction from v (t-1,n) to v (t,n) The newly generated atlas can then be passed to the next step.

[0075] (3) In order to capture the complex spatial relationship between nodes, the updated adjacency matrix is ​​input into the dynamic graph isomorphism unit. The dynamic graph isomorphism unit designs a new information aggregation method, which consists of a dynamic paradigm through parallelism. Unlike static GNNs, dynamic GNNs completely separate the data of different time slots of the same dimension after dynamic graph transformation. In fact, the same vertex in different time slots generally aggregates information from different vertex sets. The dynamic graph isomorphism network can be defined as:

[0076]

[0077]

[0078] In order to eliminate the influence of different edge weights, the edge weight ω ij Normalized to To ensure the smoothness and stability of information transmission. It is a simple implementation of dynamic graph transformation, suitable for t>2 (the second and subsequent graphs). Represents the output of the inner graph isomorphic network of node v at time slot t.

[0079] Specifically, the dynamic graph neural network designed in this embodiment includes dynamic graph transformation and dynamic graph isomorphic network, which work together to capture spatiotemporal correlation. Dynamic graph transformation reflects the changes in graph structure between different time steps by updating the adjacency matrix, which helps to capture the dynamic characteristics of action evolution over time. The dynamic graph isomorphic network establishes initial relationships between different time steps to ensure that the model can effectively process data at different time points and can effectively capture complex spatial relationships between nodes.

[0080] 3. The differentiable layered graph pooling module uses a two-dimensional convolutional neural network layer, which is used to cluster the nodes of the feature map into clusters through a preset pooling ratio.

[0081] Specifically, in multivariate time series, the vectors of each variable need to be pooled at the end. However, some common pooling methods, such as maximum pooling and average pooling, may lose a lot of information. In order to retain more effective information, a differentiable hierarchical graph pooling module is applied after each GNN layer to achieve more refined information transfer. The differentiable hierarchical graph pooling module designs a two-dimensional convolutional neural network (CNN) layer to cluster nodes according to a given pooling ratio. The nodes are regarded as channels for extracting features in CNN. In order not to destroy the window of the receptive field, the kernel size of the temporal convolution is assigned to this convolution kernel size. The specific steps are:

[0082] (1) First, an effective 2D cross-correlation operator is used to process the input node embedding tensor as follows. The input node embedding tensor here can be regarded as a feature map, in which the features of each node are encoded in a vector. The purpose of the cross-correlation operator is to extract useful information from this feature map and aggregate this information into a smaller number of output nodes.

[0083]

[0084] in, is a valid 2D cross-correlation operator, N l Equal to the number of input nodes in layer l, N l+1 is equal to the number of output nodes in layer l, X l Represents the input node embedding tensor of layer l.

[0085] (2) Generate output tensor X l+1 After that, consider calculating the corresponding adjacency matrix. The learnable weight W l The shape is [N l+1 , N l , 1, kernel size]], and then generate a vector V consisting of the learnable parameters of the lth layer l ∈R 1×k . A learnable distribution matrix can be obtained where the row corresponds to N l+1 nodes or clusters, columns correspond to N l Clusters. Given the matrix M of the input data of this layer (l) and the adjacency matrix A (l) , the output adjacency matrix is ​​generated using the following equation:

[0086] A (l+1) =M (l) A (l) M(l)T

[0087] Among them, A (l+1) Represents the connection relationship and corresponding weight of the new cluster.

[0088] 4. Combined loss function module, the output feature vector is input into the combined loss function module to implement model training and complete accurate classification of different actions. In order to enhance the separability of features to improve recognition effect, a combined loss function that integrates cross entropy loss and center loss is designed. This is to solve the problem that a single cross entropy loss cannot fully explore the structure of feature space, while overcoming the limitation that center loss is difficult to work independently in an unsupervised situation. Among them, cross entropy loss is used for classification labels to measure the difference between the predicted probability distribution and the actual probability distribution; while center loss is used to optimize the distribution of feature vectors so that feature vectors of different categories are more clustered in space. The cross entropy loss function is used to measure the difference between the probability distribution predicted by the model and the true label. By minimizing this difference, the learning process of the model can be more inclined to generate predicted values ​​that are closer to the true label. Its calculation formula is: The calculation formula of the cross entropy loss function is:

[0089]

[0090] in, Indicates the probability value that i belongs to the jth class in the probability distribution predicted by the model, y ij represents the probability that sample i belongs to the jth class, y ij =1 means it belongs to, y ij = 0 means it does not belong. Because for each sample, there is only one correct label, so the actual meaning of this loss function is to use the correct label to penalize the model prediction result.

[0091] In order to enhance the compactness of sample features within a class and reduce intra-class differences, a center loss function is added after the cross entropy loss function. The center loss is used to optimize the distribution of feature vectors so that samples in the same category are clustered more tightly in the feature space, thereby improving the generalization ability and robustness of the classifier. The formula for the center loss function is as follows:

[0092]

[0093] in, The representative category is y i The center of all features, x i Represents the i-th sample. Specifically, during the training process, the center of the features of each category is counted on a small batch scale, and it is hoped that the distance between all features and the center is as small as possible.

[0094] For the training process, the cross entropy loss function and the center loss function are calculated separately, and then the two are weighted and combined to obtain the final total loss function. The structure diagram of the loss function is as follows Figure 8 As shown, the specific formula is as follows:

[0095] Total loss function = L e +L c ·λ

[0096] Among them, λ is the weight of the center loss, which can be set manually according to the specific task and data set; by adjusting the value of λ, the influence of cross entropy loss and center loss in the training process can be balanced. The loss function designed in this way can allow the model to maintain good classification performance in classification tasks while learning more robust feature representations, thereby improving the generalization ability and accuracy of the model.

[0097] Specifically, the combined loss function designed in this embodiment combines the cross entropy loss function with the center loss function to improve the generalization ability and accuracy of the model. The cross entropy loss is used to optimize the classification task itself, while the center loss is used to increase the separation of different categories, making samples of the same type more compact in the feature space.

[0098] Combine the following Figure 1 The training and testing process of the temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention proposed in this embodiment is described in detail, including:

[0099] Step S1: determine the commonly used actions in the industrial context, which are walking, standing, turning valves, going up and down stairs, making phone calls, sitting, crawling, climbing ladders, welding, cleaning, lying down to call for help, falling forward, and falling backward, a total of 13 categories;

[0100] Step S2: Fix the four wearable sensors on key positions of the person's body such as arms and legs. Figure 2 As shown, the 3-axis acceleration and 3-axis gyroscope data of each key position of the body are collected; to ensure the consistency of the data in time, the acquisition frequency of the acceleration and gyroscope is 148Hz;

[0101] Step S3: filtering and window segmenting the personnel action data in the industrial scene in step S2;

[0102] Since the inherent noise of the sensor itself and the unconscious jitter of the tester will cause noise in the collected data, the collected data needs to be filtered and denoised before feature extraction. In this embodiment, a Kalman filter is selected.

[0103] The window segmentation technology is used to divide the filtered data into several data segments, and the sliding window with a length of 860ms and an overlap rate of 50% is used to segment the samples to obtain training and test samples. Several data segments are obtained by segmentation, and all data segments jointly construct a sample set, which is divided into a training set and a test set in a ratio of 7:3.

[0104] Step S4: Input the action sample training set data into the temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention to perform model training.

[0105] Step S5: Input the data of the action sample test set into the trained temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention to perform action classification to obtain classification results.

[0106] The key hyperparameter settings in this embodiment are shown in Table 1:

[0107] Table 1

[0108]

[0109] Use the test data sample to evaluate the performance of the model. The specific method is to calculate the recognition accuracy, precision, recall rate, and F1 value of the model for various actions in the test data.

[0110] Firstly, the temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention of this embodiment is compared with the advanced algorithm model Todynet on the personnel action dataset in an industrial context to prove the superiority of the proposed algorithm model.

[0111] Secondly, in order to prove the effectiveness of each step of the improvement in this embodiment in the process of building a temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention, ablation experiments were performed to verify it.

[0112] The specific experimental strategies are as follows:

[0113] Step (1): A common graph neural network is validated on a human action dataset in an industrial context;

[0114] Step (2): Based on the model in step (1), the hole convolution module is embedded into the convolutional neural network and verified on the human action dataset in an industrial context;

[0115] Step (3): Based on the model in step (2), the collaborative attention module is embedded after the hole convolution module and verified on the human action dataset in an industrial context;

[0116] Step (4): Based on the model in step (3), the loss function module is changed to a combination of the cross entropy loss function and the center loss function and verified on the human motion dataset in an industrial context.

[0117] Experiment execution strategy:

[0118] This embodiment first collects a data set of personnel actions in an industrial background and performs filtering processing on the data set.

[0119] The preprocessed data set is divided into a training set and a test set in a ratio of 7:3. The training set is used as the model input training data for model feature extraction and parameter training. The test set can be used to verify the model prediction value, and the model parameters can be fine-tuned through test set verification.

[0120] First, the training data set is passed into the multi-scale collaborative attention convolution module for feature extraction, and then the multi-scale spatiotemporal feature vector is passed into the dynamic graph neural module to obtain the graph feature vector. Finally, the graph feature vector is input into the fully connected layer, where regression analysis is performed and the loss function is calculated. Through the above steps, the local features and periodic features of the data are extracted, and the model parameters are gradually optimized during the training process, and the model training is completed.

[0121] Finally, the extracted features are combined with the model regression analysis results to identify the worker's action type.

[0122] The temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention adopted by the present invention provides a framework that comprehensively considers spatial, temporal and dynamic changes, and can more effectively recognize human actions. Through dynamic graph transformation, the action recognition model can better capture the dynamic correlation between different time points, which is crucial for understanding the evolution of actions over time. Through the void convolution and collaborative attention mechanism, the action recognition model can extract more robust feature representations, which are more adaptable to different perspectives and environmental changes. Using a specifically designed loss function combination, the action recognition model based on the temporal dynamic graph neural network action recognition model with multi-scale collaborative attention not only improves the accuracy of the model, but also enhances the generalization ability of the model, making it more stable on different data sets.

[0123] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. Improve the method of personnel action recognition in industrial scenes based on time dynamic graph neural network, characterized in that: include: Obtain key body position data of the person to be identified in an industrial context; The key body position data is input into a temporal dynamic graph neural network action recognition model based on multi-scale collaborative attention, and the action recognition result of the person to be identified is output, wherein the action recognition model is obtained by training with a training set, and the training set includes the key body position data of several people in an industrial background and corresponding action category labels, and the action recognition model is constructed based on a multi-scale collaborative attention convolutional network, a dynamic graph neural network, a differentiable layered graph pooling network and a combined loss function.

2. According to claim 1, the method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network is characterized in that: Before training the action recognition model through the training set, the method further includes preprocessing the training set, wherein the preprocessing includes: The key body position data of several persons in the industrial background in the training set are filtered and windowed.

3. According to claim 1, the method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network is characterized in that: The action recognition model includes: a multi-scale collaborative attention convolution module, a dynamic graph neural module, a differentiable layered graph pooling module, and a combined loss function module, wherein the multi-scale collaborative attention convolution module is used to extract the multi-scale spatiotemporal feature vectors of the key position data of the body; the dynamic graph neural module is used to construct and transform the dynamic graph based on the multi-scale spatiotemporal feature vector to obtain a feature graph containing the relationship between nodes; the differentiable layered graph pooling module is used to perform differentiable layered graph pooling on the feature graph to obtain output features; the combined loss function module is used to calculate the loss value using a combined loss function to optimize the performance of the action recognition model, and use the softmax function to convert the output features into classification probability distributions, output the probability of each action category, and obtain action classification results.

4. The method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network according to claim 3 is characterized in that: The multi-scale collaborative attention convolution module includes: a multi-scale void convolution unit and a collaborative attention unit, wherein the multi-scale void convolution unit is used to extract the parallel signal feature vectors of the key position data of the body and splice them to obtain the spliced ​​signal feature vector; the collaborative attention unit is used to perform collaborative attention mechanism processing on the spliced ​​signal feature vector to obtain the multi-scale spatiotemporal feature vector.

5. The method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network according to claim 4 is characterized in that: The multi-scale dilated convolution unit includes three parallel branches, each of which includes a dilated convolution layer, a batch norm layer and a first activation function layer connected in sequence.

6. The method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network according to claim 3 is characterized in that: The dynamic graph neural module includes: a graph construction unit, a dynamic graph transformation unit, and a dynamic graph isomorphism unit, wherein the graph construction unit is used to construct a graph of the multi-scale spatiotemporal feature vector by encoding an adjacency matrix to obtain a graph adjacency matrix; the dynamic graph transformation unit is used to perform information transmission and feature aggregation between nodes to obtain an updated graph adjacency matrix; the dynamic graph isomorphism unit is used to perform information aggregation on the updated graph adjacency matrix to obtain a feature graph containing relationships between nodes.

7. The method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network according to claim 3 is characterized in that: The differentiable layered graph pooling module adopts a two-dimensional convolutional neural network layer, which is used to cluster the nodes of the feature graph into clusters through a preset pooling ratio.

8. The method for identifying human actions in industrial scenarios using an improved temporal dynamic graph neural network according to claim 3 is characterized in that: The combined loss function includes cross entropy loss and center loss, and the combined loss function is: Total loss function = L e +L c ·λ Where, L e is the cross entropy loss, L c is the center loss, λ is the weight of the center loss, y ij represents the probability that sample i belongs to the jth class, It represents the probability value that i belongs to the jth class in the probability distribution predicted by the model, Indicates that the category is y i The center of all features, x i represents the i-th sample, k is the total number of categories, and m is the total number of samples.

Images