A skeleton data action recognition method based on a multi-relation graph convolutional neural network

Abstract

The application provides a skeleton data action recognition method based on a multi-relation graph convolutional neural network, which extracts high-level features of skeleton data from different scales by considering human natural connection relations, symmetric relations and global cooperation relations, and effectively fuses the high-level features by a relation attention mechanism. In this way, the network pays more attention to key part information in different actions, and does not lose effective information of other parts. In order to solve the overfitting and oversmoothing problems commonly existing in the graph convolutional neural network, the application proposes a new regularization method: Drop-Relation. The traditional method often discards single graph nodes or block graph nodes, which cannot prevent the node information from continuing to spread in the graph. Drop-Relation makes the whole relation matrix inactivate, effectively prevents the information in the relation graph from spreading in the network, and can inhibit the dependency between relations, effectively alleviates the overfitting and oversmoothing problems of the graph convolution.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and action recognition, specifically relating to a method for action recognition of skeleton data based on a multi-relation graph convolutional neural network. Background Technology

[0002] Human motion recognition is an emerging research field that integrates computer vision, machine learning and pattern recognition, artificial intelligence, and image processing. It has broad development prospects in areas such as human-computer interaction, elderly care, intelligent video surveillance systems, sign language recognition, and smart homes, and can play an important role in economic and social development. Initially, motion recognition tasks were mainly based on image sequences, which achieved rapid development. However, image-based motion recognition methods have certain limitations; lighting conditions, complex backgrounds, and different viewing angles can all easily affect the results. Due to its strong adaptability to complex environments and dynamic backgrounds, skeleton data-based motion recognition methods are increasingly attracting researchers' attention.

[0003] However, most existing skeleton-based graph convolutional algorithms use heuristic natural human connectivity as the graph architecture, failing to model multiple relationships within actions and ignoring collaborative relationships between non-adjacent body parts. While some methods employ learnable adjacency matrices, they simply stack these matrices, resulting in indistinguishable relationships. Currently, some graph convolutional methods construct global relationships, but these merely use fully connected graphs as the global framework. Fully connected graphs cannot accurately describe complex interactions within actions and easily overlook key interactions. Another problem with graph convolutional networks is overfitting and oversmoothing; ordinary regularization methods are difficult to apply effectively in graph convolutional networks. Therefore, there is an urgent need to propose a multi-relationship action recognition method that adaptively acquires key interactions within actions and effectively mitigates the oversmoothing problem. Summary of the Invention

[0004] The purpose of this invention is to provide a method for action recognition of skeleton data based on multi-relation graph convolutional neural networks.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for action recognition based on skeleton data using a multi-relation graph convolutional neural network, the specific steps of which are as follows:

[0007] Step 1: Preprocess each skeleton data segment in the training set, aligning the bones connecting the hip and spine with the z-axis and the joints connecting the left and right shoulders with the x-axis to effectively train the neural network.

[0008] Step 2: Input the data from Step 1 into three graph convolutional neural networks simultaneously: natural connection relation network, symmetric relation network, and global key relation network with global coordination attention module; during this process, the Drop-Relation algorithm is performed simultaneously.

[0009] Step 3: Feed the output vectors of the three networks from Step 2 into the relational attention module to further improve network performance;

[0010] Step 4: Construct 9 modules consisting of the networks from Step 2 and Step 3, add a global pooling layer and a fully connected layer to form an action recognition network, and output the action category.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1.1: Using the original human body sequence as input, first calculate the cross product of the bone connecting the hip and spine with the z-axis as the rotation axis, then calculate the angle between the bone and the z-axis to obtain the rotation matrix, and rotate the entire human body bone sequence on the axis.

[0013] Step 1.2: Based on Step 1.1, calculate the cross product of the bones connecting the left and right shoulders with the x-axis as the rotation axis, and then calculate the angle between the bones and the x-axis to obtain the rotation matrix. Rotate the entire human skeleton sequence on the axis.

[0014] Step 1.3: Due to the certain error of the measuring equipment, it is necessary to fill in the empty frames in the action by taking the skeleton sequence of the previous frame as the empty frame sequence.

[0015] Furthermore, step 2 specifically includes:

[0016] Step 2.1: Before passing through the multi-relation graph convolutional block, the Drop-Relation algorithm needs to be executed first to reduce the redundancy of the network and reduce the dependencies between relations. The main parameter of Drop-Relation is γ, which is used to control the sampling probability. On an input feature map, relations r are first abstractly sampled using a Bernoulli distribution with probability γ. Then, the activation of relations r is stopped. The elements in the graph of relation r are either all 0 or all activated. keep_prob represents the probability that relation r is retained.

[0017] Step 2.2: Construct a natural connectivity network, a symmetric connectivity network, and a global key connectivity network with a global coordination attention module; the implementation of the multi-connection graph convolutional network module is expressed by the following formula:

[0018]

[0019] In the formula, where, K represents the label partitioning in relation r. v The number of kernel functions in the spatial dimension should ideally be defined as 3, but for multi-relation networks, the number of kernel functions in the spatial dimension becomes 9, K. v It should be defined as 9; α r The importance of relation r is represented; to balance the contributions of different relations to the network, the calculation results of different relations will be effectively fused together through an attention mechanism; the weight vector α r The importance of relation r is determined by the subsequent relation attention network;

[0020] Step 2.3: Construct a global key relationship network.

[0021] Furthermore, step 2.3 specifically includes:

[0022] Step 2.3.1: First, perform spatiotemporal graph information embedding, and perform one-dimensional encoding on the input information, given input f. in The dimension is C×T×V; two pooling kernels of size (1,T) and (V,1) are used to encode each channel along the spatial and temporal directions respectively: the output of the c-th channel of node v is represented as:

[0023]

[0024] The output of the c-th channel of time frame t is represented as:

[0025]

[0026] The two transformations described above aggregate features in the time and spatial directions, respectively, to obtain a spatiotemporal perception feature map;

[0027] Step 2.3.2: Next, generate a spatiotemporally coordinated attention map, making full use of the spatiotemporal awareness map to highlight the areas of attention concentration and capture the relationships between channels; first, connect the spatiotemporal awareness map through a 1×1 convolutional layer, and then through a BatchNorm layer and a ReLU layer. This process is represented as follows:

[0028]

[0029] Here, `concat()` represents the feature concatenation operation, σ is the non-linear activation function, and `f` is the intermediate feature map with dimension iC×1×(V+T), used to encode spatiotemporal information across channels; where iC = C / r, and r is the reduction ratio of block size. Then, `f` is decomposed into two separate tensors, `f...` v and f t The dimensions are iC×1×V and iC×T×1, respectively;

[0030] Then, two 1×1 convolutional layers are used to convert fv and f t The number of channels is restored to the number of channels of the input tensor, and finally activated by two sigmoid functions. This process can be represented as:

[0031] g v =σ(conv) v (f v ))

[0032] g t =σ(conv) t (f t ))

[0033] Where, conv v and conv t There are two 1×1 convolutional layers, and σ is the sigmoid function;

[0034] Output g v and g t These are used as spatiotemporal attention weights, and finally, the attention weights are multiplied into the input features to obtain the spatiotemporal attention-based feature map. The output of this attention module is represented as:

[0035]

[0036] The spatiotemporal coordinated attention mechanism focuses on both the attention weights of different nodes in the skeleton and the attention weights of nodes between different frames, and can obtain a global feature map based on the entire time series.

[0037] Step 2.3.3: Construct a top-K based global key relationship interaction matrix, and obtain a spatiotemporally coordinated skeleton attention graph through a spatiotemporally coordinated attention mechanism; given the skeleton attention graph z in The dimension is C×T×V. First, two embedding functions are used to embed the input features into a C×T×V matrix. e In the ×T×V space, the embedding function here uses a 1×1 convolutional layer, and the two embedded features are rearranged and reshaped into V×C. e T and C e We have two matrices, T×V; then we multiply them to obtain the V×V node interaction matrix A. g Its elements Representing vertex v i and vertex v j The interaction relationships are then determined. Finally, the interaction relationship matrix is ​​sorted, and the top K neighbor nodes with the strongest interaction relationships for each node are selected to obtain the global interaction relationship matrix for the top-K nodes. The values ​​of the interaction relationship matrix are then normalized to between 0 and 1. This process is represented as follows:

[0038]

[0039] Where, conv T `conv` and `sort` represent two 1×1 convolutional layers, respectively. `sort` is the sorting function, and `f` is the sorting function. K The selection function is where K is a hyperparameter, determined experimentally.

[0040] Furthermore, step 3 specifically includes:

[0041] Step 3.1: After calculating the feature vectors of various relationships through a multi-relation graph convolutional network, in order to balance the contribution of different relationships to the network, the calculation results of different relationships will be effectively fused together through an attention mechanism. First, the calculated features of different relationships are connected and globally pooled in the time dimension; then, a 1×1 convolutional layer, a BatchNorm layer, and a ReLU layer are used; finally, an attention matrix is ​​calculated through a 1×1 convolutional layer, and a softmax classifier is used to obtain the weight vector α. r This attention mechanism is represented as:

[0042] f out =f in (r)×θ(σ(pool(f in )W)W r )

[0043] Where pool(·) is the global average pooling function, σ(·) is the ReLU activation function, and θ(·) is the softmax function; and These are all parameters learned in the convolutional layer, where iC represents the number of channels in the intermediate layer; × indicates that the corresponding elements are multiplied one by one.

[0044] Furthermore, step 4 specifically includes:

[0045] The spatial dimension of the spatiotemporal multi-relation graph convolutional block adopts a multi-relation graph convolutional block, while the temporal dimension adopts a K-axis on a feature map of dimension C×N×T. t ×1 temporal convolution; after spatial GCN and temporal GCN, there are BatchNorm layers and ReLU layers, and the Drop-Relation algorithm is executed before spatial GCN; residual connections are added to each module to stabilize training;

[0046] The multi-relation graph convolutional network is a stack of spatiotemporal multi-relation graph convolutional layers, with a total of 10 blocks. The number of output channels for each block is 64, 64, 64, 64, 128, 128, 128, 256, 256, and 256. At the beginning, a BatchNorm layer is added to standardize the input data. Then, a global average pooling layer is used to pool the features of different samples to the same size. Finally, a fully connected layer is used to obtain the prediction.

[0047] The beneficial effects of this invention are as follows:

[0048] This invention investigates a method for modeling the human skeleton using multiple relationships to extract various collaborative relationships in motion, including natural human connectivity, symmetry, and global collaboration. High-level features are extracted from skeleton data at different scales and effectively fused using a relational attention mechanism. This approach allows the network to focus more on key parts of different movements without losing valuable information from other parts. To address the overfitting and oversmoothing problems commonly found in graph convolutional neural networks, this invention proposes a novel regularization method: Drop-Relation. Traditional methods often discard individual or block-like graph nodes, which fails to prevent node information from propagating within the graph. Drop-Relation deactivates the entire relation matrix, effectively preventing information from spreading within the network and suppressing dependencies between relationships, thus effectively mitigating the overfitting and oversmoothing problems of graph convolution. Attached Figure Description

[0049] Figure 1 A diagram illustrating the natural connections within the human body;

[0050] Figure 2 This is a schematic diagram of the symmetrical connections in the human body.

[0051] Figure 3 This is a schematic diagram of a multi-relation graph convolutional block;

[0052] Figure 4 A schematic diagram for constructing the global key interaction matrix;

[0053] Figure 5 A schematic diagram of a spatiotemporal coordinated attention mechanism;

[0054] Figure 6 This is a schematic diagram of the relational attention mechanism. Detailed Implementation

[0055] The present invention will now be further described with reference to the accompanying drawings.

[0056] This invention proposes a skeleton data action recognition method based on multi-relation graph convolutional networks. Before introducing the method, the data used in this embodiment will be introduced. Human skeleton data can be collected by depth sensors. Currently, there are a large number of open-source skeleton datasets, such as NTU-RGB+D, NTU-RGB+D120, SYSU-3D, HDM05, etc. Among them, the NTU-RGB+D dataset is the largest action recognition dataset at present, and NTU-RGB+D120 is its extended version. This embodiment will select these two datasets.

[0057] The method proposed in this invention includes the following steps:

[0058] Step 1: Process the NTU-RGB+D dataset. Using the original human skeleton sequence as input, first calculate the cross product of the bones connecting the hip and spine with the z-axis as the rotation axis. Then calculate the angle between the bones and the z-axis to obtain the rotation matrix. Rotate the entire human skeleton sequence on the axis. Similarly, calculate the cross product of the bones connecting the left and right shoulders with the x-axis as the rotation axis. Again, calculate the angle between the bones and the x-axis to obtain the rotation matrix. Rotate the entire human skeleton sequence on the axis. Due to measurement equipment errors, it is necessary to fill in empty frames in the motion. Specifically, the skeleton sequence of the frame preceding the empty frame is used as the empty frame sequence.

[0059] Step 2: Input the data from Step 1 into three graph convolutional neural networks simultaneously: a natural connectivity network, a symmetric relation network, and a global key relation network with a global coordination attention module. During this process, the Drop-Relation algorithm is also performed.

[0060] Step 2.1: First, the Drop-Relation algorithm is executed. Specifically, on an input feature map, relations r are first abstracted and collected using a Bernoulli distribution with probability γ. Then, activation of relation r is stopped; all elements in the graph of relation r are either all 0 or all are activated. `keep_prob` represents the probability that relation r is retained. For the traditional dropout algorithm, `keep_prob` = 1 - γ. In experiments, `keep_prob` is set between 0.7 and 0.9. In actual experiments, `keep_prob` is not a fixed parameter but varies with the number of experimental batches; the longer the model is trained, the smaller the value of `keep_prob` becomes. Experiments show that a `keep_prob` value of 0.85 yields the best results.

[0061] Step 2.2: Next, construct the natural connection network, the symmetric relationship network, and the global key relationship network with a global coordinating attention module.

[0062] Figure 1 and Figure 2 This is a schematic diagram of human skeleton modeling according to an embodiment of the present invention. Figure 1 This is a diagram illustrating the natural connections within the human body. Figure 2 This is a schematic diagram of the symmetrical connections in the human body.

[0063] The adjacency matrix of the human skeleton graph is naturally divided into three parts: Ap, As, and Ag, based on different relationships. Ap represents the natural connectivity of the human skeleton and is an N×N two-dimensional matrix, where N = 25, representing 25 skeletal key points. The position of Ap(n1,n2) indicates the connection state between positions n1 and n2, with a value of 1 indicating a connection and a value of 0 indicating no connection. As represents symmetric connectivity, used to obtain long-distance information in the skeleton graph and extract key features of symmetrical motion. It is also an N×N two-dimensional matrix, but unlike Ap, As adopts symmetric connectivity within the human body. Based on the coordinated attention mechanism and top-K module proposed in this invention, the most important interaction relationships can be calculated, and the skeleton data most helpful for the recognition task can be flexibly obtained. Compared with Ap and As, it can uncover new connections that do not exist in the predefined skeleton structure.

[0064] In this implementation, the feature vector fed into the network has dimensions (T, N, C), where T = 300 is the time dimension, N = 25 is the spatial dimension, and C = 3 represents the three-dimensional coordinates of the skeleton joints.

[0065] Figure 3 This is a schematic diagram of a multi-relation graph convolutional block.

[0066] Next, it is fed into a multi-relationship graph convolutional block. The implementation of the multi-relationship graph convolutional network module is expressed by the following formula:

[0067]

[0068] In the formula, where, K represents the label partitioning in relation r. v The number of kernel functions in the spatial dimension should ideally be defined as 3, but for multi-relation networks, the number of kernel functions in the spatial dimension becomes 9, K. v It should be defined as 9. α r This represents the importance of relation r. To balance the contributions of different relations to the network, the computational results of different relations are effectively fused together through an attention mechanism. The weight vector α r The importance of relation r is represented by the subsequent relation attention network calculation.

[0069] The natural connections and symmetries in the human body are heuristic fixed graph structures, while the globally critical collaborative relationships are derived through computation. Figure 4 This is a schematic diagram illustrating the construction of the global collaboration relationship matrix in this invention. Figure 5 This is a schematic diagram of a spatiotemporal coordinated attention mechanism.

[0070] Step 2.2.1: To construct a spatiotemporally coordinated attention graph, firstly, spatiotemporal graph information embedding is performed, and the input information is encoded in one dimension. Given the input f... inThe dimension is C×T×V. Two pooling kernels of size (1,T) and (V,1) are used to encode each channel along the spatial and temporal directions, respectively. Therefore, the output of the c-th channel of node v can be represented as...

[0071]

[0072] The output of the c-th channel of time frame t can be expressed as

[0073]

[0074] The two transformations described above aggregate features in the temporal and spatial directions, respectively, to obtain a spatiotemporal-aware feature map. This captures long-range dependencies while preserving feature information in another dimension.

[0075] Step 2.2.2: Next, a spatiotemporally coordinated attention map is generated. This fully utilizes the spatiotemporal awareness map to highlight the areas of attention concentration and effectively capture the relationships between channels. Specifically, the spatiotemporal awareness map is first connected through a 1×1 convolutional layer, followed by a BatchNorm layer and a ReLU layer. This process can be represented as follows:

[0076]

[0077] Here, `concat()` represents the feature concatenation operation, `σ` is the non-linear activation function (in this example, the sigmoid function), and `f` is the intermediate feature map with dimension `iC×1×(V+T)`, used to encode spatiotemporal information across channels. Here, `iC = C / r`, where `r` is the reduction ratio of the block size (in this example, set to 4). Then, `f` is decomposed into two separate tensors, `f...` v and f t The dimensions are iC×1×V and iC×T×1, respectively.

[0078] Then, two 1×1 convolutional layers are used to convert f v and f t The number of channels is restored to the number of channels of the input tensor, and finally activated by two sigmoid functions. This process can be represented as follows:

[0079] g v =σ(conv) v (f v ))

[0080] g t =σ(conv) t (f t ))

[0081] Where, conv v and conv tThere are two 1×1 convolutional layers, and σ is the sigmoid function.

[0082] Output g v and g t These are used as spatiotemporal attention weights, and finally, the attention weights are multiplied into the input features to obtain the feature map based on spatiotemporal attention. The output of this attention module can be represented as...

[0083]

[0084] The spatiotemporal coordinated attention mechanism focuses on the attention weights of different nodes in the skeleton, as well as the attention weights of nodes between different frames, and can obtain a global feature map based on the entire time series.

[0085] Step 2.2.3: Construct a top-K based global key relationship interaction matrix. Through a spatiotemporal coordinated attention mechanism, a spatiotemporal skeleton attention graph can be obtained. This invention uses dot products to measure the interaction relationships between different nodes in the embedding space. Specifically, given a skeleton attention graph z... in The dimension is C×T×V. First, two embedding functions are used to embed the input features into a C×T×V matrix. e In the ×T×V space, the embedding function here uses a 1×1 convolutional layer, and the two embedded features are rearranged and reshaped into V×C. e T and C e We have two matrices, T×V. Multiplying them together gives us the V×V node interaction matrix A. g Its elements Representing vertex v i and vertex v j The interaction relationships are then determined. Finally, the interaction relationship matrix is ​​sorted, and the top K neighbor nodes with the strongest interaction relationships for each node are selected to obtain the global interaction relationship matrix for the top-K nodes. The values ​​of the interaction relationship matrix are then normalized to the range of 0-1, a process that can be represented as follows:

[0086]

[0087] Where, conv T `conv` and `sort` represent two 1×1 convolutional layers, respectively. `sort` is the sorting function, and `f` is the sorting function. K The selection function is K, which is a hyperparameter. Through experiments, the best results are achieved when K is set to 10.

[0088] Figure 6 This is a schematic diagram of the relational attention module of the present invention.

[0089] Step 3: After calculating feature vectors for multiple relationships using a multi-relation graph convolutional network, the calculation results for different relationships are effectively fused together through an attention mechanism to balance their contributions to the network. First, the calculated features of different relationships are concatenated and globally pooled along the time dimension. Then, a 1×1 convolutional layer, a BatchNorm layer, and a ReLU layer are used. Finally, an attention matrix is ​​calculated using a 1×1 convolutional layer, and a softmax classifier is used to obtain the weight vector α. r This attention mechanism can be represented as

[0090] f out =f in (r)×θ(σ(pool(f in )W)W r )

[0091] Where pool(·) is the global average pooling function, σ(·) is the ReLU activation function, and θ(·) is the softmax function. and These are all learnable parameters in the convolutional layer, where iC represents the number of channels in the intermediate layer. × indicates that corresponding elements are multiplied one by one.

[0092] Step 4: Construct 9 modules consisting of the networks from Steps 2 and 3, plus a global pooling layer and a fully connected layer to form an action recognition network, outputting the action category; the spatial dimension of the spatiotemporal multi-relation graph convolutional blocks uses multi-relation graph convolutional blocks, and the temporal dimension uses K on a feature map of dimension C×N×T. t A time convolution of ×1, where K t =9. Both spatial and temporal GCNs are followed by BatchNorm and ReLU layers, with the Drop-Relation algorithm executed before the spatial GCN. Residual connections are added to each module for stable training.

[0093] The multi-relation graph convolutional network is a stack of spatiotemporal multi-relation graph convolutional layers, with a total of 10 blocks. Each block has 64, 64, 64, 64, 128, 128, 128, 256, 256, and 256 output channels. A BatchNorm layer is added at the beginning to standardize the input data. Then, a global average pooling layer pools the features of different samples to the same size, and finally, a fully connected layer is used to obtain the prediction.

[0094] The NTU-RGB+D dataset was divided into training and test sets. The network was trained using the training set and the model performance was validated using the test set. The final accuracy reached 95.9%, while the accuracy reached 85.8% on the more challenging NTU-RGB+D dataset, demonstrating significant results.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A skeleton data action recognition method based on a multi-relation graph convolutional neural network, characterized by comprising the steps of: The specific steps are as follows: Step 1: Preprocess each piece of skeleton data in the training set, align the skeleton connecting the hips and spine with the z-axis, and align the joints connecting the left and right shoulders with the x-axis to effectively train the neural network; Step 2: Input the data in step 1 into three types of relational graph convolutional neural networks simultaneously, namely natural connection relationship network, symmetric relationship network, and global key relationship network with global coordination attention module; in this process, the Drop-Relation algorithm is simultaneously performed; Step 2.1: Before passing through the multi-relation graph convolution block, the Drop-Relation algorithm needs to be executed first to reduce the redundancy of the network and reduce the dependence between relations; the main parameters of Drop-Relation are , which is used to control the sampling probability; on an input feature map, first abstractly collect relations with a Bernoulli distribution with probability ; then stop activating relations , and the elements in the graph of relation are either all 0 or all activated; represents the probability that relation is retained; Step 2.2: Construct natural connection relationship network, symmetric relationship network, and global key relationship network with global coordination attention module; the multi-relation graph convolutional network module is implemented as follows: ； In the formula, wherein, indicates the relationship label division in, According to the number of spatial dimension kernel function should be defined as 3, but for multi-relationship network, the number of spatial dimension kernel function will become 9, Should be defined as 9; indicates the relationship The importance of relationship; in order to balance the contribution of different relationships to the network, the calculation results of different relationships will be effectively fused together through attention mechanism; weight vector indicates the importance of relationship , calculated by the subsequent relationship attention network; Step 2.3: Construct the global key relationship network; Step 2.3.1: First, the spatiotemporal graph information embedding is performed, which is a one-dimensional encoding operation on the input information, given the input with dimension ; two pooling kernels with size and are used to encode each channel along the spatial direction and the temporal direction, respectively: the output of the th channel of the th node is represented as: ； Time frame The output of the first channel is represented as: ； The above two transformations respectively aggregate the features in the time direction and the spatial direction to obtain spatiotemporal perception feature maps; Step 2.3.2: Then generate a spatiotemporal coordination attention map to fully utilize the spatiotemporal perception map, highlight the areas where attention is concentrated, and capture the relationships between channels; first, connect the spatiotemporal perception map through a 1x1 convolution layer, and then pass it through a BatchNorm layer and a ReLU layer, which is represented as: ； wherein, a concatenation operation representing a feature, is a nonlinear activation function, is an intermediate feature mapping with dimension , used to encode the spatio-temporal information on the channel; wherein , is a reduction ratio of the block size, then is decomposed into two separate tensors, and with dimensions and ; Then the two Convolutional layers are used to restore the channel number of and the input tensor, and finally activated by two sigmoid functions, which is represented as: ； ； wherein, and are two convolutional layers, is a sigmoid function; Output and are used as spatio-temporal attention weights, respectively. Finally, the attention weights are multiplied into the input features, and the spatio-temporal attention-based feature map is obtained. The output of the attention module is represented as: ； The spatiotemporal coordination attention mechanism focuses on both the attention weights of different nodes in the skeleton and the attention weights of nodes between different frames, and can obtain a global feature map based on the entire time sequence; Step 2.3.3: Constructing the top-K based global key relation interaction matrix, obtaining the skeleton attention graph based on space-time through the space-time coordination attention mechanism; given the skeleton attention graph , the dimension is , first, the input features are embedded into a space with a dimension of using two embedding functions, here the embedding function uses a convolutional layer with , the two embedded features are rearranged and reshaped into two matrices and ; then multiply them to get the node interaction relationship matrix , the element represents the interaction relationship between vertex and vertex ; finally, sort the interaction relationship matrix, take the top neighbor nodes of each node interaction relationship, and get the global interaction relationship matrix of top- nodes; normalize the value of the interaction relationship matrix to 0-1, the process is represented as: ； wherein, and respectively represent two convolutional layers, is an ordering function, is a selection function, wherein is a hyperparameter, determined by experiment; Step 3: Input the output vectors of the three networks in step 2 into the relationship attention module to further improve the network performance; Step 4: Construct 9 modules composed of the networks in steps 2 and 3, add a global pooling layer and a fully connected layer to form an action recognition network, and output the action category.

2. The skeleton data action recognition method based on the multi-relation graph convolutional neural network according to claim 1, characterized in that: The step 3 specifically includes: Step 3.1: After calculating the feature vectors of multiple relationships by the multi-relation graph convolution network, in order to balance the contribution of different relationships to the network, the calculation results of different relationships will be effectively fused together through an attention mechanism; first, the calculation features of different relationships are connected and globally pooled in the time dimension; then, a 1x1 convolution layer, a BatchNorm layer and a ReLU layer are used, and finally a 1x1 convolution layer is used to calculate the attention matrix, and a softmax classifier is used to obtain the weight vector The attention mechanism is represented as: ； wherein, is a global average pooling function, is a ReLU activation function, is a softmax function; and are parameters learned in the convolutional layers, wherein denotes the number of channels in the intermediate layer; denotes element-wise multiplication.

3. The skeleton data action recognition method based on the multi-relation graph convolutional neural network according to claim 1, characterized in that: The step 4 specifically includes: The spatial dimension of the spatio-temporal multi-relation graph convolution block adopts a multi-relation graph convolution block, and the time dimension adopts a time convolution on a feature map with a dimension of A BatchNorm layer and a ReLU layer are arranged after the spatial GCN and the time GCN, and a Drop-Relation algorithm is performed before the spatial GCN; a residual connection is added to each module to stabilize training.​ The multi-relation graph convolutional network is a stack of spatiotemporal multi-relation graph convolutional layers, with a total of 10 blocks, each block having an output channel number of 64, 64, 64, 64, 128, 128, 128, 256, 256, and 256; a BatchNorm layer is added at the beginning to standardize the input data; then, a global average pooling layer is used to pool the features of different samples into the same size, and finally, a fully connected layer is used to obtain the prediction.

Images