Skeleton behavior recognition method based on dynamic timing multi-dimensional adaptive graph convolution network

Abstract

The application provides a skeleton behavior recognition method based on a dynamic timing multi-dimensional adaptive graph convolution network. Through a non-local network and a dimension transformation module, a multi-dimensional adaptive graph convolution module is constructed. Meanwhile, dynamic convolution and timing convolution are combined. In the timing convolution, an SE attention network is introduced to calculate the weights of each convolution kernel weight and bias, a dynamic timing convolution module is constructed, the two are combined, and end-to-end training is performed under a multi-flow network to construct a dynamic timing multi-dimensional adaptive graph convolution network model. The dynamic timing multi-dimensional adaptive graph convolution network model in the application can fully extract multi-dimensional information in a spatial domain from a skeleton space-time graph, and meanwhile, the representation ability of time characteristics is enhanced. Based on the skeleton behavior recognition method of the application, excellent recognition accuracy can be achieved, and good generalization performance is also possessed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a skeleton behavior recognition method based on a dynamic time sequence multi-dimensional adaptive graph convolution network. BACKGROUND

[0002] Behavior recognition, as a hot spot in the field of computer vision, has a wide range of applications in intelligent security, intelligent monitoring, intelligent medical treatment and other fields in today's society. With the development and application of deep learning technology in the field of computer vision, it has achieved more remarkable results in behavior recognition research. Skeleton information is a high-level feature of the human body, which has the advantages of being less affected by appearance, effectively avoiding the noise influence caused by background occlusion, illumination change and view change, and convenient storage and calculation compared with RGB data and depth data. Therefore, the skeleton-based behavior recognition method has become one of the popular research fields. Since the human skeleton joint itself is a topological graph, the graph convolutional neural network has superior performance in processing such topological graphs. Some scholars combine graph convolution with skeleton behavior recognition and propose a skeleton behavior recognition technology based on graph convolution. However, the existing graph convolution network for skeleton behavior recognition has weak representation ability when modeling the time feature while considering the time feature when modeling the spatial feature, resulting in low recognition accuracy and poor generalization performance. SUMMARY

[0003] In order to solve the problem of low recognition accuracy and poor generalization performance of the existing skeleton behavior recognition method, the present application provides a skeleton behavior recognition method based on a dynamic time sequence multi-dimensional adaptive graph convolution network, which can fully extract multi-dimensional, dynamic and effective spatio-temporal feature information in the skeleton spatio-temporal graph, has high recognition accuracy and good generalization performance.

[0004] The technical scheme of the present application is as follows: a skeleton behavior recognition method based on a dynamic time sequence multi-dimensional adaptive graph convolution network, comprising the following steps:

[0005] S1: obtaining an original video sample, preprocessing the original video sample, and obtaining skeleton information data in the original video sample;

[0006] S2: modeling the skeleton information data of the original video sample into a skeleton spatio-temporal graph;

[0007] S3: based on the obtained skeleton information data, after data processing, extracting skeleton behavior feature data, obtaining multi-branch training data based on the skeleton behavior feature data, denoted as: multi-branch training feature data;

[0008] characterized in that it further comprises the following steps:

[0009] S4: constructing a dynamic time-series multi-dimensional adaptive graph convolution network model based on an adaptive graph convolution method and a dynamic convolution, as a skeleton behavior recognition model; the skeleton behavior recognition model comprises: a skeleton behavior recognition branch model;

[0010] S5: setting and adjusting hyperparameters of the skeleton behavior recognition model, training each of the skeleton behavior recognition branch models in the skeleton behavior recognition model using the multi-branch training feature data, and obtaining a trained skeleton behavior recognition branch model;

[0011] S6: obtaining to-be-recognized video data, extracting skeleton information data in a to-be-recognized video data group, extracting a to-be-recognized skeleton space-time graph based on the skeleton information data, inputting the to-be-recognized skeleton space-time graph into each of the trained human skeleton behavior recognition branch models respectively, fusing outputs of the branch models, and obtaining a final skeleton behavior recognition result;

[0012] In step S4, the dynamic time-series multi-dimensional adaptive graph convolution network model is constructed, including the following steps:

[0013] S4-1: performing subset division on a neighborhood of each node in the skeleton space-time graph to obtain a skeleton node subset corresponding to the skeleton space-time graph; the skeleton node subset comprises: a node self-subset, a centripetal neighbor node subset, and a centrifugal neighbor node subset;

[0014] The node self-subset comprises the node itself;

[0015] The centripetal neighbor node subset comprises a centripetal neighbor node close to a skeleton barycenter;

[0016] The centrifugal neighbor node subset comprises a centrifugal neighbor node away from the skeleton barycenter;

[0017] S4-2: constructing a multi-dimensional adaptive graph based on the skeleton space-time graph, wherein the multi-dimensional adaptive graph comprises: a spatial adaptive graph, a temporal adaptive graph, and a channel adaptive graph;

[0018] The spatial adaptive graph is a spatial graph extracted based on the skeleton space-time graph;

[0019] The temporal adaptive graph is generated after dimension transformation of a temporal graph extracted based on the skeleton space-time graph;

[0020] The channel adaptive graph is generated after dimension transformation of a channel graph extracted based on the skeleton space-time graph;

[0021] combine the multi-dimensional adaptive graph and the initial adjacency matrix and the mask matrix to obtain a multi-dimensional adaptive adjacency matrix, perform graph convolution on the multi-dimensional adaptive adjacency matrix, and construct a spatial graph convolution layer together with the skeleton node subset, denoted as: a multi-dimensional adaptive graph convolution module;

[0022] S4-3: based on the skeleton spatio-temporal graph, introducing an attention mechanism for two-dimensional convolution weights and biases, constructing a 9x1 dynamic temporal convolution to extract the time features of the skeleton spatio-temporal graph, and constructing a temporal graph convolution layer, denoted as: a dynamic temporal convolution module;

[0023] S4-4: combining the multi-dimensional adaptive graph convolution module and the dynamic temporal convolution module to construct a dynamic temporal multi-dimensional adaptive graph convolution block;

[0024] The dynamic temporal multi-dimensional adaptive graph convolution block comprises: a multi-dimensional adaptive graph convolution layer, a BN layer, a RELU, and a dynamic temporal convolution layer connected in sequence;

[0025] S4-5: constructing the skeleton behavior recognition model;

[0026] The skeleton behavior recognition model comprises: a BN layer, 10 dynamic temporal multi-dimensional adaptive graph convolution blocks, a GAP layer, and a softmax layer connected in sequence.

[0027] It is further characterized in that:

[0028] In step S3, the multi-branch training feature data comprises: node information Joint, skeleton information Bone, node motion information Joint_motion, and skeleton motion information Bone_motion.

[0029] In step S3, based on the skeleton information data, the method for extracting skeleton behavior feature data and modeling the skeleton behavior feature data comprises the following steps:

[0030] S3-1: taking the node itself modeling as joint information, assuming that the node information Joint of t frame is represented as Jt;

[0031] S3-2: taking the node close to the center of gravity of the skeleton as a source node taking the node far away from the center of gravity of the skeleton as a target node skeleton information Bone represents a vector from the source node to the target node Since the number of nodes is one more than the number of skeletons, the center node is set as an empty skeleton, i.e. 0;

[0032] S3-3: the node motion information and the skeleton motion information are represented by the coordinate difference between each two adjacent frames, and therefore the calculation formula of the node motion information Joint_motion is The formula for calculating the bone motion information Bone_motion is Since the number of time frames is one more than the motion information, the value of the last frame is set to 0;

[0033] In step S4-2, the step of constructing a multi-dimensional adaptive graph convolution module includes:

[0034] S4201: Define nodes, time frames, and channels as dimensional nodes in respective dimensions, and use a normalized embedding Gaussian function to calculate the similarity f(v i , v j ) of two dimensional nodes v i and v j :

[0035]

[0036] where N is the number of dimensional nodes, and θ(v i ) and φ(v j ) are selected as embedding functions of dimensional node i and dimensional node j using two parallel 1x1 convolution operations;

[0037] S4202: Calculate the spatial graph S k , the time graph T k , and the channel graph C k using a softmax function, and the calculation formulas are respectively:

[0038]

[0039]

[0040]

[0041] where f in represents the input skeleton behavior feature data, W θ and W φ represent the embedding Gaussian functions θ and φ, respectively;

[0042] where the spatial graph S k ∈R V×V is a spatial adaptive graph, and the time graph T k ∈R T×T and the channel graph C are dimensionally transformed to generate a time adaptive graph and a channel adaptive graph

[0043] S4203: Generate a multi-dimensional adaptive adjacency matrix by combining the dimensional adaptive graphs and the initial adjacency matrix, and perform graph convolution operation:

[0044]

[0045] in, C represents the input features and the output of the multidimensional adaptive graph convolution, respectively. in Indicates the input channel, C out Indicates the output channel; W k K is the convolution kernel function. v Represents a predefined maximum distance; A k B is the initial adjacency matrix. k It is an enhanced data mask matrix, S k , These are spatial adaptive graphs, temporal adaptive graphs, and channel adaptive graphs generated through nonlocal networks, respectively.

[0046] Step S4-3, the steps of constructing the dynamic temporal convolution module include:

[0047] S4301: Extrusion stage;

[0048] Global average pooling is used to compress the dimension of features to the dimension represented by the channels; the compression process is represented as:

[0049]

[0050] In the formula, T represents the number of frames, V represents the number of nodes, i and j represent the i-th frame in the time domain and the j-th node in the spatial domain, X is the input feature, and z is the feature map after channel compression.

[0051] S4302: Incentive Phase;

[0052] After two fully connected computations, the weights are obtained through softmax activation and temperature coefficient calculation. The calculation formula is as follows:

[0053] z = W2δ(W1Z)

[0054]

[0055] Where δ represents the ReLU activation function, σ represents the softmax activation function, W1 and W2 represent the weight parameters of the two fully connected layers, z represents the feature map after channel compression, z represents the output of the second fully connected layer, τ represents the temperature coefficient, K represents the number of convolutional kernels participating in the aggregation, and π k Let z be the attention weight corresponding to the k-th convolutional kernel. k Let z be the value corresponding to the k-th convolutional kernel in z. j Let z be the value corresponding to the j-th convolutional kernel;

[0056] S4303: Based on the attention values ​​and biases obtained from the SE module, calculate the weights and biases in the dynamic convolution kernel:

[0057]

[0058]

[0059] In the formula, K is the number of convolution kernels, W k is the weight of the kth convolution kernel, π k is the attention weight of the kth convolution kernel, W is the dynamic convolution kernel weight, b k is the bias of the kth convolution kernel, and b is the bias of the dynamic convolution kernel.

[0060] S4304: A 9x1 dynamic convolution layer is constructed to perform convolution operation on the output of the multi-dimensional adaptive graph convolution layer, and the calculation formula is:

[0061] y = δ(Wf out +b)

[0062] In the formula, y is the output after dynamic convolution, f out is the output of the multi-dimensional graph convolution layer, and δ is the ReLU activation function.

[0063] In step S5, based on the training method of the multi-branch network, when the dynamic time-series multi-dimensional adaptive graph convolution network model is trained using the branch training data respectively, the stochastic gradient descent method is used as the optimization strategy, and the cross-entropy function is selected as the loss function of gradient back propagation.

[0064] The application provides a skeleton behavior recognition method based on a dynamic timing multi-dimensional adaptive graph convolution network. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 A skeleton behavior recognition method based on a dynamic timing multi-dimensional adaptive graph convolution network in the application is shown in the flowchart;

[0066] Figure 2 A node division method in the application is shown in the schematic diagram;

[0067] Figure 3 A working principle of time dimension transformation in the application is shown in the schematic diagram;

[0068] Figure 4 A working principle of channel dimension transformation in the application is shown in the schematic diagram;

[0069] Figure 5 A structure of the multi-dimensional adaptive adjacency matrix proposed in the application is shown in the schematic diagram;

[0070] Figure 6 A working principle of dynamic timing convolution in the application is shown in the schematic diagram;

[0071] Figure 7A structural schematic diagram of a spatio-temporal graph convolution block in the application;

[0072] Figure 8 A structural schematic diagram of a dynamic timing multi-dimensional adaptive graph convolution network model in the application. DETAILED DESCRIPTION

[0073] As shown in Figure 1 The application includes a skeleton behavior recognition method based on a dynamic timing multi-dimensional adaptive graph convolution network, which includes the following steps.

[0074] S1: Obtain original video samples in historical data, pre-process the original video samples, and obtain skeleton information data in the original video samples.

[0075] S2: Model the skeleton information data of the original video samples into a skeleton spatio-temporal graph.

[0076] S3: Based on the obtained skeleton information data, after data processing, extract skeleton behavior feature data, and obtain multi-branch training data based on the skeleton behavior feature data, denoted as: multi-branch training feature data. In specific applications, when extracting training data from skeleton behavior feature data, test skeleton behavior feature data also needs to be extracted.

[0077] In this embodiment, the multi-branch training feature data includes: node information Joint, skeleton information Bone, node motion information Joint_motion, and skeleton motion information Bone_motion.

[0078] Specifically, based on the skeleton information data, the skeleton behavior feature data is extracted, and the method for modeling the skeleton behavior feature data includes the following steps:

[0079] S3-1: Model the node itself as joint information, and assume that the node information Joint of frame t is represented as Jt;

[0080] S3-2: Take the node close to the center of the skeleton as the source node Take the node away from the center of the skeleton as the target node The skeleton information Bone represents a vector from the source node to the target node Since the number of nodes is one more than the number of skeletons, the central node is set to an empty skeleton, i.e., 0, to facilitate the simplification of network design and maintain the feature dimension unchanged;

[0081] S3-3: The node motion information and the skeleton motion information are represented by the coordinate difference between each two adjacent frames, so the calculation formula of the node motion information Joint_motion is The calculation formula of the skeleton motion information Bone_motion is Since the number of time frames is one more than the number of motion information frames, the value of the last frame is set to 0 to simplify network design and keep the feature dimensions unchanged.

[0082] S4: Based on adaptive graph convolution and dynamic convolution, a dynamic temporal multidimensional adaptive graph convolution network model is constructed as a skeleton behavior recognition model; the skeleton behavior recognition model includes: skeleton behavior recognition branch model.

[0083] The specific steps for constructing a dynamic temporal multidimensional adaptive graph convolutional network model include the following steps.

[0084] S4-1: Divide the neighborhood of each node in the skeleton spatiotemporal graph into subsets to obtain the skeleton node subsets corresponding to the skeleton spatiotemporal graph; the skeleton node subsets include: the node itself subset, the centripetal neighbor node subset, and the centrifugal neighbor node subset.

[0085] The subset of a node itself includes the node itself;

[0086] The centripetal neighbor node subset includes: centripetal neighbor nodes that are closer to the centroid of the skeleton;

[0087] The centrifugal neighbor node subset includes: centrifugal neighbor nodes that are far from the centroid of the skeleton;

[0088] like Figure 2 As shown, Figure 2 The node partitioning method in the skeleton spatiotemporal graph is defined, where each small circle represents a skeleton node.

[0089] Let G be the centroid node. Taking node B as an example, its neighborhood includes three nodes A, B, and C. Here, B represents the node itself, A represents the near-centroid neighbor node close to the centroid of the skeleton, and C represents the far-centroid neighbor node.

[0090] S4-2: Based on the skeleton spatiotemporal graph, construct a multi-dimensional adaptive graph, which includes: spatial adaptive graph, temporal adaptive graph and channel adaptive graph;

[0091] The spatial adaptive graph is a spatial graph extracted based on the skeleton spatiotemporal graph;

[0092] The time-adaptive graph is generated by transforming the time graph extracted from the skeleton spatiotemporal graph.

[0093] The channel adaptive graph is generated by transforming the channel graph extracted from the skeleton spatiotemporal graph.

[0094] The multidimensional adaptive graph is combined with the initial adjacency matrix and the mask matrix to obtain the multidimensional adaptive adjacency matrix. The multidimensional adaptive adjacency matrix is ​​then subjected to graph convolution and together with the skeleton node subset to construct a spatial graph convolution layer, denoted as: multidimensional adaptive graph convolution module.

[0095] In the technical solution of this invention, while obtaining the spatial adaptive graph, a temporal adaptive graph and a channel adaptive graph are obtained through nonlocal networks and dimensional transformations. These are then combined to obtain a multidimensional adaptive graph convolutional module, which enhances the extraction of spatial, temporal, and channel-related human skeleton information features in the graph convolutional network and further improves the accuracy of skeleton behavior recognition in the graph convolutional network.

[0096] In practical applications, the steps for constructing a multidimensional adaptive graph convolution module include:

[0097] S4201: Define nodes, time frames, and channels as dimensional nodes of their respective dimensions, and use a normalized embedding Gaussian function to compute the values ​​of two dimensional nodes v. i and v i similarity f(v) i v i ):

[0098]

[0099] Where: N is the number of dimension nodes, θ(v i ), φ(v j Two parallel 1×1 convolution operations are selected as the embedding functions for dimension node i and dimension node j.

[0100] S4202: Calculate the spatial graph S using the softmax function. k Time graph T k Channel diagram C k The calculation formulas are as follows:

[0101]

[0102]

[0103]

[0104] In the formula f in W represents the skeleton behavior feature data of the input. θ W φ Let θ and φ represent the embedded Gaussian functions, respectively.

[0105] Among them, spatial diagram S k ∈R V×V That is, a spatially adaptive graph, relative to a time graph T k ∈R T×T and channel diagram Perform dimensional transformation to generate a time-adaptive graph. and channel adaptive graph

[0106] like Figure 3The working principle of the time dimension transformation module is shown. The time graph T k ∈R T×T As input, the dimension is increased by the unsqueeze function, the dimension is converted by the permute function ((0, 1, 2, 3)→(0, 2, 1, 3)), then a convolution operation (Convt1) is performed, the channel number T is changed to 1, the dimension is converted again by the permute function ((0, 1, 2, 3)→(0, 3, 1, 3)), then a batch normalization processing (BN_T) is performed, a second convolution operation (Convt2) is performed, the channel number T is changed to V×V, the reshape is performed by the view function (Reshape), and finally the time adaptive graph is obtained by the softmax function calculation.

[0107] As shown in Figure 4 The working principle of the channel dimension transformation module is shown. The channel graph C As input, the dimension is increased by the unsqueeze function, the dimension is converted by the permute function ((0, 1, 2, 3)→(0, 2, 1, 3)), then a convolution operation (Convc1) is performed, the channel number C e is changed to 1, the dimension is converted again by the permute function ((0, 1, 2, 3)→(0, 3, 1, 2)), then a batch normalization processing (BN_T) is performed, a second convolution operation (Convc2) is performed, the channel number C e is changed to V×V, the reshape is performed by the view function (Reshape), and finally the channel adaptive graph is obtained by the softmax function calculation.

[0108] S4203: Generate a multi-dimensional adaptive adjacency matrix by combining the adaptive graphs of each dimension and the initial adjacency matrix, and perform a graph convolution operation:

[0109]

[0110] Wherein, are the input features and the output of the multi-dimensional adaptive graph convolution respectively, C in represents the input channel, C out represents the output channel, W k is a convolution kernel function, K v represents a predefined maximum distance; the value of K v is set according to the node division strategy, i.e., if there are 3 node subsets in the skeleton node subset, K v is set to 3;

[0111] A k is the initial adjacency matrix, B k is an enhanced data mask matrix, Sk 、 respectively are a spatial adaptive graph, a temporal adaptive graph and a channel adaptive graph generated by a non-local network.

[0112] As shown in Figure 5 , it is a multi-dimensional adaptive adjacency matrix in the technical scheme of the application, that is Figure 5 The input f in represents the skeleton behavior feature data, and the size is NxCXTxV, which respectively represents batch size, channel number, frame number and joint number;θ k (x) and φ k (x) number represent the Gaussian embedding function of the non-local network, which is used to calculate the similarity between each node, each time frame and each channel, obtain the spatial graph, time graph and channel graph, and further convert the time graph and channel graph into time adaptive graph and channel adaptive graph The multi-dimensional adaptive adjacency matrix in the application enhances the extraction ability of human skeleton information in the space, time and channel dimensions, can better express the connection relationship between nodes, frames and channels and the correlation between dimensions, so as to better mine the multi-dimensional structure features of human skeleton in the graph convolution process, and improve the recognition accuracy of the model.

[0113] S4-3: based on the skeleton space-time graph, the attention mechanism for two-dimensional convolution weight and bias is introduced, a 9x1 dynamic time series convolution is constructed to extract the time feature of the skeleton space-time graph, and a time series graph convolution layer is constructed, which is recorded as: dynamic time series convolution module.

[0114] The specific steps of constructing the dynamic time series convolution module include:

[0115] S4301: extrusion stage;

[0116] The dimension of the feature is compressed to the dimension represented by the channel through global average pooling;The extrusion process is represented as:

[0117]

[0118] In the formula, T represents the frame number, V represents the node number, i and j represent the i-th frame in the time domain and the j-th node in the space domain, X is the input feature, and z is the feature map after channel extrusion;

[0119] S4302: excitation stage;

[0120] After two full connection calculations, the weight value is obtained through the softmax function activation and the temperature coefficient, and the calculation formula is:

[0121] z=W2delta(W1z)

[0122]

[0123] Wherein, delta represents the ReLU activation function, sigma represents the softmax activation function, W1 and W2 represent the weight parameters of two full connections respectively, z represents the feature map after channel squeezing, z represents the output of the second layer full connection, tau represents the temperature coefficient; K represents the number of convolution kernels participating in aggregation, pi k is the attention weight value corresponding to the kth convolution kernel, z k is the value corresponding to the kth convolution kernel in z j is the value corresponding to the jth convolution kernel in z.

[0124] S4303: According to the weight and bias of the attention obtained by the SE module, the weight and bias in the dynamic convolution kernel are calculated:

[0125]

[0126]

[0127] In the formula, K is the number of convolution kernels, W k is the weight of the kth convolution kernel, pi k is the attention weight value of the kth convolution kernel, W is the dynamic convolution kernel weight, b k is the bias of the kth convolution kernel, and b is the bias of the dynamic convolution kernel.

[0128] S4304: A 9x1 dynamic convolution layer is constructed to perform convolution operation on the output of the multi-dimensional adaptive graph convolution layer, and the calculation formula is:

[0129] y=delta(Wf out +b)

[0130] In the formula, y is the output after dynamic convolution, f out is the output of the multi-dimensional graph convolution layer, and delta is the ReLU activation function.

[0131] As Figure 6 shown is an embodiment of the working process of dynamic time series convolution, the input f in is the input feature of the dynamic time series convolution layer, and the dashed box in the figure is the attention mechanism module, which includes: avg Pool, FC, Relu, FC and softmax connected in turn, f inAfter processing by the attention mechanism module, attention weights π1-π4 are obtained, which are combined with convolution kernels Covn1-Covn4 to obtain a dynamic convolution kernel Conv for convolution operation, f in After being sequentially sent into Conv, BN, and ReLU, the output f is obtained out .

[0132] S4-4: Combine the multi-dimensional adaptive graph convolution module and the dynamic temporal convolution module to construct a dynamic temporal multi-dimensional adaptive graph convolution block.

[0133] The dynamic temporal multi-dimensional adaptive graph convolution block comprises a multi-dimensional adaptive graph convolution layer, a BN layer, a ReLU, and a dynamic temporal convolution layer connected in sequence.

[0134] As shown in Figure 7 , the dynamic temporal multi-dimensional adaptive graph convolution block comprises a multi-dimensional adaptive graph convolution module (labeled MD-AGCN in the figure), a BN layer, a ReLU layer, and a dynamic temporal convolution module (labeled DYTCN in the figure) connected in sequence.

[0135] The skeleton feature is input into the multi-dimensional adaptive graph convolution module to extract spatial multi-dimensional features, and then is activated by a ReLU activation function and batch normalized by a BatchNorm2d function, and then is input into the dynamic temporal convolution module to obtain time sequence features with stronger time sequence representation ability through a dynamic convolution layer that aggregates four convolution kernels.

[0136] Finally, a Res selective convolution is constructed using a dynamic convolution, and is added to the output of the above steps to construct a residual model. The residual connection can enhance the flexibility of the model without reducing the original performance.

[0137] S4-5: Construct a skeleton behavior recognition model.

[0138] The skeleton behavior recognition model comprises a BN layer, 10 dynamic temporal multi-dimensional adaptive graph convolution blocks, a GAP layer, and a softmax layer connected in sequence. As shown in Figure 8 , the three numbers on each graph convolution block represent the input channel, the output channel, and the step length, respectively.

[0139] S5: Set and adjust the hyperparameters of the skeleton behavior recognition model, use the multi-branch training feature data to train each skeleton behavior recognition branch model in the skeleton behavior recognition model, and obtain the trained skeleton behavior recognition branch model.

[0140] In a specific implementation, based on the training method of the multi-branch network, when the dynamic temporal multi-dimensional adaptive graph convolution network model is trained using branch training data, a stochastic gradient descent method is used as the optimization strategy, and a cross-entropy function is selected as the loss function for gradient backpropagation.

[0141] S6: Obtain the to-be-identified video data, extract the skeleton information data in the to-be-identified video data group, extract the to-be-identified skeleton space-time graph based on the skeleton information data, input the to-be-identified skeleton space-time graph into each trained human skeleton behavior recognition branch model respectively, fuse the outputs of each branch model, and obtain the final skeleton behavior recognition result.

[0142] When the model is implemented, it is implemented based on a pytorch deep learning framework. The optimization strategy adopts stochastic gradient descent (SGD), the Nesterov momentum is set to 0.9, the iteration period is set to 50, and the initial learning rate is set to 0.1. In order to overcome the problem of overfitting during training, cross entropy is selected as the loss function, the weight decay is set to 0.0001, the decay period is set to the 30th and 40th periods, and the batch size is set to 32.

[0143] Cross entropy is selected as the loss function of gradient back propagation:

[0144]

[0145] Where i represents the ith sample, c represents the class, p ic represents the predicted probability that the ith sample belongs to class c, y ic represents an indicator variable (0 or 1), which is 1 if the behavior class is the same as the class of the sample i (i.e. each frame skeleton sequence) in the training set, otherwise 0; M represents the number of behavior classes.

[0146] Table 1 is an embodiment of the structure of the dynamic timing multi-dimensional adaptive graph convolution network model. Based on the network structure in Table 1 below, after the input data is processed by 10 sub-network structure blocks, it enters the GAP layer, the 3-dimensional data output by the sub-network structure block is converted to 1-dimensional data in the GAP layer, then the data is reduced from 480000 to 60 (120) dimensions through the FC layer, and finally mapped to 60 (120) dimensions for prediction.

[0147] Table 1: Structure of dynamic timing multi-dimensional adaptive graph convolution network model

[0148] Model layer Input and output channel number Step length (stride) Output data size Input [3，300，25] 1 st ]] in_channels = 3, out_channels = 64 1 [64，300，25] 2 nd ]] in_channels = 64, out_channels = 64 1 [64，300，25] 3 rd ]] in_channels = 64, out_channels = 64 1 [64，300，25] 4 th ]] in_channels = 64, out_channels = 64 l [64，300，25] 5 th ]] in_channels = 64, out_channels = 128 2 [128，150，25] 6 th ]] in_channels = 128, out_channels = 128 1 [128，150，25] 7 th ]] in_channels = 128, out_channels = 128 1 [128，150，25] 8 th ]] in_channels = 128, out_channels = 256 2 [256，75，25] 9 th ]] in_channels = 256, out_channels = 256 1 [256，75，25] 10 th ]] in_channels = 256, out_channels = 256 1 [256，75，25] GAP [480000] FC [480000, 60] or [480000, 120]

[60] or

[120]

[0149] In order to verify the effectiveness and practicability of the human behavior recognition method in the technical scheme of the application, the NTU-RGB+D and NTU-RGB+D120 data sets are selected as experimental data sets for experiments.

[0150] NTU-RGB+D is collected by 3 Microsoft Kinect v2 sensors, and the angles of the three sensors are -45°, 0° and 45° respectively, and a total of 60 types of actions are included. Among them, the first 49 actions are single-person actions, and the 50th to 60th actions are two-person interactive actions. The NTU-RGB+D dataset has two evaluation methods: cross-subject (X-Sub) and cross-view (X-View). The test results are shown in Table 2.

[0151] Table 2: Recognition accuracy (%) in X-View and X-Sub in NTU-RGB+D dataset

[0152] Scenario ]]> ​ Joint ]]> ​ Bone ]]> ​ Joint-motion ]]> ​ Bone-motion ]]> ​ Joint+Bone ]]> ​ DTMDA-GCN ]]> ​ X-View ]]> ​ 94.8 ]]> ​ 94.8 ]]> ​ 93.1 ]]> ​ 92.9 ]]> ​ 96.1 ]]> ​ 96.5 ]]> ​ X-Sub ]]> ​ 87.8 ]]> ​ 88.5 ]]> ​ 85.9 ]]> ​ 85.9 ]]> ​ 89.9 ]]> ​ 90.4 ]]> ​

[0153] In Table 2, Joint, Bone, Joint-motion, Bone-motion, Joint+Bone respectively represent joint, bone, joint motion, bone motion, joint and bone features; DTMDA-GCN represents the skeleton behavior recognition model based on the dynamic time series multi-dimensional adaptive graph convolution network model in the technical solution of the present application, and the corresponding accuracy is 96.5% for X-View and 90.4% for X-Sub. The outputs of the four branches are fused to obtain the final skeleton behavior recognition result. Because this experiment is based on the dataset to verify the accuracy of the model, the output results of the four branches are defined as the accuracy. In actual application of the skeleton behavior recognition of the technical solution of the present application, the output results of the four branches are defined according to specific needs, such as defining the output results as the probability of skeleton behavior.

[0154] At the same time, based on the NTU-RGB+D dataset, the dynamic time series multi-dimensional adaptive graph convolution network model in the technical solution of the present application is compared with other skeleton behavior recognition models, and the comparison results are shown in Table 3 below:

[0155] Table 3: Comparison of recognition accuracy (%) of different models based on NTU-RGB+D dataset

[0156] Model X-sub X-view STA-LSTM 73.4 81.2 VA-LSTM 79.2 87.7 TCN 74.3 83.1 ClipCNN+MTLN 79.6 84.8 AS-GCN 86.8 94.2 ST-GCN 81.5 88.3 2s-AGCN 88.5 95.1 MS-AAGCN 90.0 96.2 DTMDA-GCN 90.4 96.5

[0157] From the data in Table 3, it can be seen that in the NTU dataset divided in two ways of X-View and X-Sub, the technical solution of the present application achieves the highest recognition accuracy, which is 96.5% and 90.4% respectively, and the technical solution of the present application can improve the recognition accuracy of skeleton behavior.

[0158] The NTU-RGB+D120 dataset has more samples performed by more subjects in more different camera views compared to the NTU-RGB+D dataset. The dataset contains 120 actions performed by 106 different subjects. The NTU-RGB+D120 dataset provides two benchmarks for evaluation, namely cross-subject (X-Sub) and cross-set (X-Set). Based on the NTU-RGB+D120 dataset, the dynamic timing multi-dimensional adaptive graph convolution network model in the technical solution of the application is compared with other skeleton behavior recognition models, and the comparison results are shown in Table 4.

[0159] Table 4: Comparison of recognition accuracy of different models based on NTU-RGB+D120 dataset (%)

[0160] Model X-sub X-setup Soft RNN 36.3 44.9 Dynamic Skeleton 50.8 54.7 Spatio-Temporal LSTM 55.7 57.9 GCA-LSTM 58.3 59.2 SGN 79.2 81.5 SkeleMotion 67.7 66.9 2s-AGCN 82.9 84.9 4s Shift-GCN 85.9 87.6 DTMDA-GCN(Joint) 81.2 83.9 DTMDA-GCN(Bone) 83.4 85.6 DTMDA-GCN(Joint+Bone) 85.4 87.4 DTMDA-GCN 86.5 88.2

[0161] From the data in Table 4, it can be seen that on the NTU-RGB+D120 dataset divided in two ways of X-sub and X-setup, the technical solution of the application achieves the highest recognition accuracy of 86.5% and 88.2% respectively, further verifying the good recognition accuracy of the model, and the NTU-RGB+D dataset and the NTU-RGB+D120 dataset based on the application can achieve good recognition accuracy, indicating that the skeleton behavior recognition model in the application has good generalization performance.

[0162] The application provides a skeleton behavior recognition method based on a dynamic timing multi-dimensional adaptive graph convolution network. When modeling the skeleton spatiotemporal graph, the correlation between the three dimensions of space, time and channel is considered. It can not only realize feature extraction and fusion of skeleton information in the space, time and channel dimensions, but also enhance the representation ability of time features. At the same time, it achieves excellent recognition accuracy on large skeleton datasets and has good generalization performance.

Claims

1. A skeleton behavior recognition method based on a dynamic timing multi-dimensional adaptive graph convolutional network, comprising the following steps: S1: obtaining an original video sample, preprocessing the original video sample, and obtaining skeleton information data in the original video sample; S2: modeling the skeleton information data of the original video sample into a skeleton space-time graph; S3: based on the obtained skeleton information data, after data processing, extracting skeleton behavior feature data, obtaining multi-branch training data based on the skeleton behavior feature data, denoted as: multi-branch training feature data; characterized in that it further comprises the following steps: S4: based on the adaptive graph convolution method and the dynamic convolution, a dynamic timing multi-dimensional adaptive graph convolutional network model is constructed as a skeleton behavior recognition model; the skeleton behavior recognition model comprises: a skeleton behavior recognition branch model; S5: setting and adjusting the hyperparameters of the skeleton behavior recognition model, using the multi-branch training feature data to train each skeleton behavior recognition branch model in the skeleton behavior recognition model, and obtaining a trained skeleton behavior recognition branch model; S6: obtaining to-be-recognized video data, extracting skeleton information data in the to-be-recognized video data group, extracting a to-be-recognized skeleton space-time graph based on the skeleton information data, inputting the to-be-recognized skeleton space-time graph into each trained skeleton behavior recognition branch model, respectively, fusing the outputs of each branch model, and obtaining a final skeleton behavior recognition result; In step S4, the dynamic timing multi-dimensional adaptive graph convolutional network model is constructed, comprising the following steps: S4-1: performing subset division on the neighborhood of each node in the skeleton space-time graph to obtain a skeleton node subset corresponding to the skeleton space-time graph; the skeleton node subset comprises: a node self-subset, a centripetal neighbor node subset, and a centrifugal neighbor node subset; the node self-subset comprises the node itself; the centripetal neighbor node subset comprises centripetal neighbor nodes close to the skeleton center of gravity; the centrifugal neighbor node subset comprises centrifugal neighbor nodes away from the skeleton center of gravity; S4-2: based on the skeleton space-time graph, a multi-dimensional adaptive graph is constructed, which comprises: a spatial adaptive graph, a temporal adaptive graph, and a channel adaptive graph; the spatial adaptive graph is a spatial graph extracted based on the skeleton space-time graph; the temporal adaptive graph is generated after dimension transformation of a temporal graph extracted based on the skeleton space-time graph; the channel adaptive graph is generated after dimension transformation of a channel graph extracted based on the skeleton space-time graph; the multi-dimensional adaptive graph, an initial adjacency matrix, and a mask matrix are combined to obtain a multi-dimensional adaptive adjacency matrix, the multi-dimensional adaptive adjacency matrix is subjected to graph convolution, and a spatial graph convolution layer is constructed together with the skeleton node subset, denoted as: a multi-dimensional adaptive graph convolution module; S4-3: based on the skeleton space-time graph, a two-dimensional convolution weight and bias attention mechanism is introduced to construct a 9x1 dynamic timing convolution to extract the temporal features of the skeleton space-time graph, and a timing graph convolution layer is constructed, denoted as: a dynamic timing convolution module; S4-4: combine the multi-dimensional adaptive graph convolution module and the dynamic temporal convolution module to construct a dynamic temporal multi-dimensional adaptive graph convolution block; The dynamic temporal multi-dimensional adaptive graph convolution block comprises a multi-dimensional adaptive graph convolution layer, a BN layer, a RELU, and a dynamic temporal convolution layer connected in sequence. S4-5: construct the skeleton behavior recognition model. The skeleton behavior recognition model comprises a BN layer, 10 dynamic temporal multi-dimensional adaptive graph convolution blocks, a GAP layer, and a softmax layer connected in sequence.

2. The method of claim 1, wherein the method comprises: In step S3, the multi-branch training feature data comprises node information Joint, skeleton information Bone, node motion information Joint_motion, and skeleton motion information Bone_motion.

3. The method of claim 1, wherein the method further comprises: In step S3, based on the skeleton information data, the skeleton behavior feature data is extracted, and a method for modeling the skeleton behavior feature data comprises the following steps: S3-1: Model the node itself as joint information, assume the node information Joint at t frame is represented as J t ; S3-2: take the node close to the center of the skeleton as the source node take the node far from the center of the skeleton as the target node Bone information Bone represents the vector from the source node to the target node Since the number of nodes is one more than the number of skeletons, the center node is set to an empty skeleton, i.e. 0; S3-3: The node motion information and the bone motion information are represented by coordinate differences between each two adjacent frames, so the calculation formula of the node motion information Joint_motion is The calculation formula of the bone motion information Bone_motion is Since the number of time frames is one more than the motion information, the value of the last frame is set to 0.

4. The method of claim 1, wherein the method further comprises: In step S4-2, the steps of constructing the multi-dimensional adaptive graph convolution module comprise: S4201 : Define nodes, time frames, and channels as dimension nodes for respective dimensions, compute similarity f(v i ,v j ) of two dimension nodes v i and v j using normalized embedding Gaussian function: where: N is the number of dimension nodes, θ(v i ), φ(v j ) selects two parallel 1x1 convolution operations as the embedding function of dimension node i and dimension node j; S4202: Calculate the spatial graph S in combination with the softmax function k , the temporal graph T k , the channel graph C k , and the calculation formula is respectively where f in represent the input skeleton behavior feature data, W θ , W φ respectively represent the embedding Gaussian functions θ and φ; where the spatial map S k ∈ R V×V i.e. the spatial adaptive map, to the temporal map T k ∈ R T×T and the channel map dimensional transformation, generating the temporal adaptive map and the channel adaptive map S4203: combine each dimension adaptive graph and the initial adjacency matrix to generate a multi-dimensional adaptive adjacency matrix and perform graph convolution operation: wherein, are input features and output of multi-dimensional adaptive graph convolution, respectively, C in denotes input channels, C out denotes output channels; W k is a convolution kernel function, K v represents a predefined maximum distance; A k is an initial adjacency matrix, B k is an enhanced data mask matrix, S k , are spatial, temporal and channel adaptive graphs generated by non-local network, respectively.

5. The method of claim 1, wherein the method further comprises: In step S4-3, the steps of constructing the dynamic temporal convolution module comprise: S4301: squeeze stage; The dimension of the feature is compressed to the dimension represented by the channel through global average pooling; the squeezing process is represented as: In the formula, T represents the number of frames, V represents the number of nodes, i and j represent the i-th frame in the time domain and the j-th node in the spatial domain, X is the input feature, and Z is the feature map after channel squeezing; S4302: excitation stage; After two fully connected calculations, the weights are obtained through softmax function activation and temperature coefficient, and the calculation formula is: z=W2δ(W1Z) Wherein, δ represents the ReLU activation function, σ represents the softmax activation function, W1 and W2 represent the weight parameters of the two full connections respectively, Z represents the feature map after channel squeezing, z represents the output of the second layer full connection, τ represents the temperature coefficient; K represents the number of convolution kernels participating in aggregation, π k is the attention weight corresponding to the kth convolution kernel, z k is the value corresponding to the kth convolution kernel in z, z j is the value corresponding to the jth convolution kernel in z. S4303: calculate the weights and biases in the dynamic convolution kernel according to the attention of the weights and biases obtained by the SE module: In the formula, K is the number of convolution kernels, W k is the weight of the kth convolution kernel, π k is the attention weight of the kth convolution kernel, W is the weight of the dynamic convolution kernel, b k is the bias of the kth convolution kernel, and b is the bias of the dynamic convolution kernel. S4304: construct a 9x1 dynamic convolution layer to perform convolution operation on the output of the multi-dimensional adaptive graph convolution layer, and the calculation formula is: y = δ(Wf out + b) where y is the output after dynamic convolution, f out is the output of the multi-dimensional graph convolution layer, and δ is the ReLU activation function.

6. The method of claim 1, wherein the method further comprises: In step S5, based on the training method of the multi-branch network, when the dynamic temporal multi-dimensional adaptive graph convolution network model is trained using branch training data respectively, the stochastic gradient descent method is used as the optimization strategy, and the cross-entropy function is selected as the loss function of gradient backpropagation.

Images