A skeleton behavior recognition method based on hyperconnection graph convolution network

Abstract

The application provides a skeleton behavior recognition method based on a hyperconnection graph convolution network, a hyperconnection adjacency matrix is constructed according to the connection relationship of a human skeleton, a hyperconnection adaptive graph convolution module is further constructed, two residual connections are introduced in a full-dimensional dynamic convolution, a residual full-dimensional dynamic time convolution module is constructed, the two are combined, end-to-end training is carried out based on fused feature data, and a hyperconnection graph convolution network model is obtained; the hyperconnection graph convolution network model in the application can fully extract long-distance dependency relationships of nodes from skeleton feature data, meanwhile, feature loss is reduced, and the representation ability of time features is enhanced; based on the skeleton behavior recognition method of the application, excellent recognition accuracy can be obtained, and good generalization performance is also possessed.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a skeleton behavior recognition method based on hyperconnected graph convolutional networks. Background Technology

[0002] As a high-level feature that can directly represent human behavior information, the skeleton is less affected by noise caused by changes in environment, lighting, and viewing angle, and is easy for computers to process and store. Therefore, skeleton-based behavior recognition technology is widely used in human-computer interaction, intelligent monitoring, and smart healthcare. In existing research, some researchers have used graph convolution technology to extract behavioral feature data to recognize human behavior and have achieved certain results in the field of skeleton behavior recognition. However, most existing graph convolution-based network models only construct the corresponding adjacency matrix based on the physical connection relationship of the human skeleton to participate in graph convolution. However, the adjacency matrix constructed based on physical connection lacks modeling of long-distance dependencies between joints, and cannot accurately identify some behaviors based on long-distance joint motion relationships. At the same time, most existing graph convolution models rely on TCN (Temporal Convolutional Network) to extract temporal features, which has a weak ability to represent temporal features. As a result, the recognition accuracy and generalization performance in skeleton behavior recognition are low. Summary of the Invention

[0003] To address the issues of low accuracy and poor generalization performance in existing graph convolution-based skeleton behavior recognition technologies, this invention provides a skeleton behavior recognition method based on hyperconnected graph convolutional networks. This method can fully extract long-distance joint motion relationships from skeleton behavior features, possesses strong temporal feature representation capabilities, and exhibits both high recognition accuracy and good generalization performance.

[0004] The technical solution of this invention is as follows: a skeleton behavior recognition method based on hyperconnected graph convolutional networks, comprising the following steps:

[0005] S1: Obtain the original video sample, preprocess the original video sample, and obtain the skeleton information data in the original video sample;

[0006] Its characteristic is that it further includes the following steps:

[0007] S2: Based on the skeleton information data, perform data processing, extract the skeleton sequence, obtain skeleton motion features based on the skeleton sequence, and perform feature fusion on the skeleton motion features, denoted as: training fusion feature data;

[0008] The skeletal motion features include: first-order motion features, second-order motion features, and third-order motion features;

[0009] S3: Based on the skeleton physical connection relationship, construct the hyperconnection adjacency matrix by adding the connection relationship beyond the adjacent node;

[0010] S4: Construct a hyperconnected graph convolutional network model as a skeleton behavior recognition model, and train the skeleton behavior recognition model based on the training fusion feature data to obtain a trained behavior recognition model;

[0011] Constructing the hyperconnected graph convolutional network model includes the following steps:

[0012] S4-1: Construct a multi-dimensional adaptive graph based on the skeleton sequence;

[0013] The multi-dimensional adaptive graph includes: spatial adaptive graph, temporal adaptive graph, and channel adaptive graph;

[0014] The spatial adaptive graph is a spatial graph extracted based on the skeleton sequence;

[0015] The time-adaptive graph is generated by transforming the dimensionality of the time graph extracted from the skeleton sequence.

[0016] The channel adaptive graph is generated by transforming the channel graph extracted from the skeleton sequence.

[0017] S4-2: Expand the dimensions of the hyperconnection adjacency matrix and combine it with its mask matrix and the multi-dimensional adaptive graph to obtain the hyperconnection adaptive adjacency matrix;

[0018] Based on the hyperconnected adaptive adjacency matrix, a graph convolution module is constructed, denoted as: hyperconnected adaptive graph convolution module;

[0019] S4-3: Add residual connections to full-dimensional dynamic convolution to construct residual full-dimensional dynamic convolution;

[0020] Based on the residual full-dimensional dynamic convolution, the temporal features of the skeleton sequence are extracted to construct a temporal convolution module, denoted as: residual full-dimensional dynamic temporal convolution module;

[0021] S4-4: Combine the hyperconnected adaptive graph convolution module and the residual full-dimensional dynamic temporal convolution module to construct a hyperconnected graph convolution layer;

[0022] The hyperconnected graph convolutional layer includes: a hyperconnected adaptive graph convolutional module, a BN layer, a ReLU layer, a residual full-dimensional dynamic temporal convolutional module, a BN layer, and a ReLU layer connected in sequence;

[0023] S4-5: Construct the skeleton behavior recognition model;

[0024] The skeleton behavior recognition model includes: a BN layer connected in sequence, 10 consecutive hyperconnected graph convolutional layers, a GAP layer, and a softmax layer;

[0025] S5: Obtain the video data to be identified, extract the skeleton sequence from the video data group to be identified, denoted as: skeleton sequence to be identified; input the skeleton sequence to be identified into the trained skeleton behavior recognition model to obtain the final skeleton behavior recognition result.

[0026] Its further features are:

[0027] Step S1 involves obtaining the skeleton information data from the original video sample, which includes the following steps:

[0028] S1-1: Perform frame segmentation processing on the acquired original video samples to decompose the continuous video segments into an image sequence including static frames;

[0029] S1-2: Calculated based on the Openpose pose estimation algorithm;

[0030] Set the calculation parameters of the Openpose algorithm, input the images of the static frames obtained from the decomposed video into Openpose, and obtain the human skeleton data of the corresponding number of nodes in the static frames;

[0031] S1-3: Based on the human body nodes and the corresponding joint numbers in the Openpose algorithm, construct the connection relationship of the human skeleton data to represent the morphological features of the human body, that is, obtain the skeleton information data.

[0032] In step S2, skeleton motion features are obtained based on the skeleton sequence, and feature fusion is performed on the skeleton motion features to obtain the fused feature data for training. Specifically, this includes the following steps:

[0033] S2-1: Suppose that the skeleton information data contains N nodes and T frames, the complete skeleton sequence can be constructed as L=(S,E);

[0034] Where S = {s ti |t=1,2,...,T,i=1,2,...,N} represents the set of nodes, and E={E S E T} represents the set of skeletal edges, which is constructed by connecting adjacent nodes within the same frame to form the set of spatial edges E. S The connection of the same node between adjacent frames constructs the set of temporal edges E. T ;

[0035] S2-2: Using the node itself as the first-order motion feature of the node, assuming that the first-order motion feature Joint of the node in frame t is represented as J t ;

[0036] S2-3: Use the node closest to the bone's center of gravity as the source node. Target nodes are those furthest from the center of gravity of the skeleton. The first-order motion feature of the skeleton, represented by a bone, is the vector pointing from the source node to the target node. Since the number of nodes is one more than the number of bones, the center node is set to an empty bone, i.e., a value of 0.

[0037] S2-4: The second-order motion features of nodes and skeletons are represented by the coordinate difference between their respective first-order motion features in two adjacent frames. Therefore, the formula for calculating the second-order motion feature Joint_motion of nodes is: The formula for calculating the second-order motion feature of the skeleton, bone_motion, is as follows: Since the number of time frames is one more than the dimension of the second-order motion feature, the value of the last frame is set to 0.

[0038] S2-5: The third-order motion features of nodes and skeletons are represented by the coordinate difference between their respective second-order motion features in two adjacent frames. Therefore, the formula for calculating the third-order motion feature of nodes, Joint_accelerate, is as follows: The formula for calculating the third-order motion feature of the skeleton, Bone_accelerate, is as follows: Since the number of time frames is 2 more than the dimension of the third-order motion feature, the values ​​of the last two frames are set to 0.

[0039] S2-6: Based on the first-order motion features, second-order motion features, and third-order motion features of the above nodes and bones, the channel dimensions are concatenated using the Concatenation function to obtain the training fusion feature data.

[0040] The training fusion feature data includes: node fusion feature Joint_merge and bone fusion feature Bone_merge;

[0041] In step S3, the hyperconnection adjacency matrix is ​​constructed by adding connections beyond neighboring nodes, including the following steps:

[0042] S3-1: Constructing an adjacency matrix G1∈R based on the physical connections of the human skeleton 25×25 ;

[0043] S3-2: Using the chest node as the center node, connect nodes with a gap of one node to construct a second-order hyperconnection adjacency matrix G2∈R. 13×13 ;

[0044] S3-3: Using the chest node as the central node, connect nodes with a gap of two to construct a third-order hyperconnection adjacency matrix G3∈R. 9×9 ;

[0045] In step S4-1, the step of constructing the hyperconnected adaptive graph convolution module includes:

[0046] S4101: Calculate the spatial graph S using the softmax function. k Time graph T k Channel diagram C k The calculation formulas are as follows:

[0047]

[0048]

[0049]

[0050] In the formula, f in W represents the input features. θ W φ Let θ and φ represent the embedded Gaussian functions, respectively.

[0051] Among them, spatial diagram S k ∈R N×N That is, a spatially adaptive graph, relative to a time graph T k ∈R T×T and channel diagram Perform dimensional transformation to generate a time-adaptive graph. and channel adaptive graph

[0052] R N×N In this context, N represents the number of skeleton nodes. C in e The number of transition channels is 0.25 × C. in ), C in Indicates the input channel; k represents the layer index number of each graph;

[0053] R T×T In this context, T indicates that the skeleton information data contains the total number of frames;

[0054] S4102: Combine the adaptive graphs of each dimension and the extended hyperconnected adjacency matrix to generate a hyperconnected adaptive adjacency matrix, and then perform graph convolution operations:

[0055]

[0056] in, C represents the input features and the output of the graph convolution, respectively. in Indicates the input channel, C outIndicates the output channel; W k K is the convolution kernel function. v Set to 3, B k To enhance the data mask matrix, S k , and These represent spatial adaptive graphs, time adaptive graphs, and channel adaptive graphs, respectively.

[0057] This indicates an expanded hyperconnection adjacency matrix;

[0058] Among them G i ∈{G1,G2,G3}, where G1 represents a first-order hyperlink adjacency matrix, G2 represents a second-order hyperlink adjacency matrix, G3 represents a third-order hyperlink adjacency matrix, α represents the expansion function, v is the number of nodes used to construct the hyperlink adjacency matrix, and Q∈R. v×25 To initialize the transformation matrix;

[0059] In step S4101, the time graph T k Channel diagram C k Convert to time-adaptive graph and channel adaptive graph The method is as follows:

[0060] Construct a conversion module; the conversion module uses a time map T k Channel diagram C k As input, with and For output; the working process of the conversion module is as follows:

[0061] Let X be the input of the conversion module. k ∈R y×y In the conversion module, X k ∈R y×y The transformation module increases the dimension to 1×y×y using the unsqueeze function, converts the dimension to y×1×y using the permute function, performs a convolution operation to transform it to 1×1×y, uses the permute function again to convert the dimension to y×1×1, performs batch normalization, performs a second convolution operation to transform it to (N×N)×1×1, reconstructs it to N×N using the view function, and finally calculates the output of the transformation module using the softmax function.

[0062] In step S4-3, constructing the residual full-dimensional dynamic temporal convolution module specifically includes the following steps:

[0063] S4301: Acquiring full-dimensional attention;

[0064] The input channel attention of the convolution kernel is obtained through the SE module. Output channel attention Convolutional Spatial Attention Atten K Number of convolutions and attention. M The calculation formula is:

[0065]

[0066]

[0067] Atten K =σ(W4(δ(W1(Avgpool(f)) in )))))

[0068] Atten M =σ(W5(δ(W1(Avgpool(f)) in )))))

[0069] Where δ represents the ReLU activation function, σ represents the softmax activation function, Avgpool represents average pooling, W1, W2, W3, W4, and W5 represent the weight parameters of the fully connected layer, and C... in Indicates the input channel, C out K represents the output channel, K represents the kernel size, and M represents the number of kernels.

[0070] S4302: Calculate the weights of the full-dimensional dynamic convolution kernel based on the attention based on the convolution space and the number of convolutions:

[0071] W od =(Atten K Atten M W M

[0072] In the formula, W M W is the weight matrix for M convolutional kernels. od These are the weights for the full-dimensional dynamic convolution;

[0073] S4303: Combining two residual connections, a 9×1 residual full-dimensional dynamic temporal convolution module is constructed to perform a convolution operation on the output of the hyper-connected adaptive graph convolution module. The calculation formula is as follows:

[0074]

[0075] In the formula, Y is the output of the residual full-dimensional dynamic temporal convolution, and X is the output of the hyperconnected adaptive graph convolution module.

[0076] This application provides a skeleton behavior recognition method based on hyperconnected graph convolutional networks. Based on the physical connection relationship of the skeleton, it constructs a hyperconnected adjacency matrix by adding connections beyond adjacent nodes, which fully extracts the long-distance joint motion relationship in the skeleton behavior features. At the same time, it combines a multi-dimensional adaptive graph to construct a hyperconnected adaptive graph convolutional module. Based on the hyperconnected adaptive graph convolutional module, the modeling ability of the long-distance dependency relationship of the joint is enhanced, thereby improving the accuracy of the skeleton behavior recognition results. This method constructs residual full-dimensional dynamic convolution by adding residual connections to the full-dimensional dynamic convolution. In the time domain of the hyperconnected graph convolutional network model, it extracts the temporal features of the skeleton sequence and calculates full-dimensional attention. Simultaneously, it adds two residual connections to construct a residual full-dimensional dynamic temporal convolution module, enhancing the extraction capability of temporal features and reducing the loss of temporal feature information during computation. The hyperconnected graph convolutional network model is constructed by combining the hyperconnected adaptive graph convolution module and the residual full-dimensional dynamic temporal convolution module, and trained based on fused feature data. This ensures that the skeleton behavior recognition model in this method can fully extract the long-distance dependencies and temporal information features of the skeleton, while exhibiting excellent recognition accuracy and good generalization performance. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of the skeleton behavior recognition method based on hyperconnected graph convolutional networks in this invention.

[0078] Figure 2 This is a schematic diagram of the fused feature data used for training in this invention;

[0079] Figure 3 This is a schematic diagram of the hyperconnected adjacency matrix construction method in this invention;

[0080] Figure 4 This is a schematic diagram of the hyperconnected adaptive graph convolution module proposed in this invention;

[0081] Figure 5 This is a schematic diagram of the residual full-dimensional dynamic temporal convolution module in this invention;

[0082] Figure 6 This is a schematic diagram of the structure of the hyperconnected graph convolutional layer in this invention;

[0083] Figure 7 This is a schematic diagram of the hyperconnected graph convolutional network model structure in this invention. Detailed Implementation

[0084] like Figure 1 As shown, this application includes a skeleton behavior recognition method for hyperconnected graph convolutional networks, which includes the following steps.

[0085] S1: Obtain the original video samples, preprocess the original video samples, and obtain the skeleton information data from the original video samples. Specifically, this includes the following steps:

[0086] S1-1: Perform frame segmentation on the acquired raw video samples to decompose continuous video segments into an image sequence that includes static frames;

[0087] S1-2: Calculated based on the Openpose pose estimation algorithm;

[0088] Set the calculation parameters of the Openpose algorithm, input the images of static frames obtained from the decomposed video into Openpose, and obtain the human skeleton data of the corresponding number of nodes in the static frame;

[0089] S1-3: Based on the human body nodes and the corresponding joint numbers in the Openpose algorithm, construct the connection relationship of the human skeleton data to represent the morphological features of the human body, that is, obtain the skeleton information data.

[0090] S2: Based on skeleton information data, perform data processing, extract skeleton sequences, obtain skeleton motion features based on skeleton sequences, and perform feature fusion on skeleton motion features, denoted as: training fused feature data;

[0091] Skeletal motion features include: first-order motion features, second-order motion features, and third-order motion features.

[0092] The process of obtaining skeleton motion features based on skeleton sequences to obtain fused feature data for training specifically includes the following steps:

[0093] S2-1: Suppose the skeleton information data contains N nodes and T frames, the complete skeleton sequence can be constructed as L=(S,E);

[0094] Where S = {s ti |t=1,2,...,T,i=1,2,...,N} represents the set of nodes, and E={E S E T} represents the set of skeletal edges, which is constructed by connecting adjacent nodes within the same frame to form the set of spatial edges E. S The connection of the same node between adjacent frames constructs the set of temporal edges E. T ;

[0095] S2-2: Using the node itself as the first-order motion feature of the node, assuming that the first-order motion feature Joint of the node in frame t is represented as J t ;

[0096] S2-3: Use the node closest to the bone's center of gravity as the source node. Target nodes are those furthest from the center of gravity of the skeleton. The first-order motion feature of the skeleton, represented by a bone, is the vector pointing from the source node to the target node. Since the number of nodes is one more than the number of bones, setting the center node to an empty bone (i.e., a value of 0) keeps the feature dimension unchanged and simplifies network design.

[0097] S2-4: The second-order motion features of nodes and skeletons are represented by the coordinate difference between their respective first-order motion features in two adjacent frames. Therefore, the formula for calculating the second-order motion feature Joint_motion of nodes is: The formula for calculating the second-order motion feature of the skeleton, bone_motion, is as follows: Since the number of time frames is one more than the second-order motion feature dimension, the value of the last frame is set to 0, so that the feature dimension remains unchanged, and at the same time, it is convenient to simplify the network design.

[0098] S2-5: The third-order motion features of nodes and skeletons are represented by the coordinate difference between their respective second-order motion features in two adjacent frames. Therefore, the formula for calculating the third-order motion feature of nodes, Joint_accelerate, is as follows: The formula for calculating the third-order motion feature of the skeleton, Bone_accelerate, is as follows: Since the number of time frames is 2 more than the dimension of the third-order motion feature, the values ​​of the last two frames are set to 0, so that the feature dimension remains unchanged and the network design is simplified.

[0099] S2-6: Based on the first-order motion features, second-order motion features, and third-order motion features of the above nodes and bones, the channel dimensions are concatenated using the Concatenation function to obtain fused feature data for training.

[0100] The training fusion feature data includes: node fusion feature Joint_merge and bone fusion feature Bone_merge, specifically as follows: Figure 2 As shown. Figure 2 In this method, the node fusion feature `Joint_merge` simultaneously integrates the features of the node's third-order motion feature `Joint Accelerate`, the node's second-order motion feature `Joint Motion`, and the node's first-order motion feature `Joint`. Similarly, the skeleton fusion feature `Bone_merge` simultaneously integrates the features of the skeleton's third-order motion feature `Bone Accelerate`, the skeleton's second-order motion feature `Bone Motion`, and the skeleton's first-order motion feature `Bone`. This method fully extracts long-distance joint motion relationships from the skeleton's behavioral features based on the fusion feature data used in training, improving the accuracy of subsequent calculation results.

[0101] S3: Based on the skeleton physical connection relationship, construct the hyperconnection adjacency matrix by adding connections that transcend adjacent nodes. This includes the following steps:

[0102] S3-1: Constructing an adjacency matrix G1∈R based on the physical connections of the human skeleton 25×25 ;

[0103] S3-2: Using the chest node as the center node, connect nodes with a gap of one node to construct a second-order hyperconnection adjacency matrix G2∈R. 13×13 ;

[0104] S3-3: Using the chest node as the central node, connect nodes with a gap of two to construct a third-order hyperconnection adjacency matrix G3∈R. 9×9 ;

[0105] like Figure 3 The diagram illustrates the construction method of a hyperconnected adjacency matrix. A person's skeleton sequence data contains 25 joints. Each dot represents a skeleton node, and each line segment represents the constructed node connection relationship. Therefore, the adjacency matrix constructed based on the physical connections of the human skeleton is G1∈R. 25×25 The second-order hyperconnected adjacency matrix contains 13 nodes, and the third-order hyperconnected adjacency matrix contains 9 nodes. Based on the second-order and third-order hyperconnected adjacency matrices, node connection relationships that transcend physical connections are constructed on the basis of the physical connection relationships of the human skeleton, providing a data foundation for subsequent extraction of long-distance dependencies of the skeleton.

[0106] S4: Construct a hyperconnected graph convolutional network model as a skeleton behavior recognition model, and train the skeleton behavior recognition model based on the fused feature data used for training to obtain a trained behavior recognition model.

[0107] Constructing a hyperconnected graph convolutional network model includes the following steps:

[0108] S4-1: Construct a multi-dimensional adaptive graph based on the skeleton sequence;

[0109] Multidimensional adaptive graphs include: spatial adaptive graphs, temporal adaptive graphs, and channel adaptive graphs;

[0110] The spatial adaptive graph is a spatial graph extracted based on the skeleton sequence;

[0111] The time-adaptive graph is generated by transforming the time graph extracted from the skeleton sequence.

[0112] The channel adaptive graph is generated by transforming the channel graph extracted from the skeleton sequence.

[0113] The steps for constructing a hyperconnected adaptive graph convolutional module include the following:

[0114] S4101: Calculate the spatial graph S using the softmax function. k Time graph Tk Channel diagram C k The calculation formulas are as follows:

[0115]

[0116]

[0117]

[0118] In the formula, f in W represents the input features. θ W φ Let θ and φ represent the embedded Gaussian functions, respectively.

[0119] Among them, spatial diagram S k ∈R N×N That is, a spatially adaptive graph; for a time graph T k ∈R T×T and channel diagram Perform dimensional transformation to generate a time-adaptive graph. and channel adaptive graph

[0120] R N×N In this context, N represents the number of skeleton nodes. C in e The number of transition channels is 0.25 × C. in ), C in Indicates the input channel; k represents the layer index number of each graph;

[0121] R T×T In this context, T indicates that the skeleton information data contains the total number of frames;

[0122] S4102: Combine the adaptive graphs of each dimension and the extended hyperconnected adjacency matrix to generate a hyperconnected adaptive adjacency matrix, and then perform graph convolution operations:

[0123]

[0124] in, C represents the input features and the output of the graph convolution, respectively. in Indicates the input channel, C out Indicates the output channel; W k K is the convolution kernel function. v Set to 3, B k To enhance the data mask matrix, S k , and These represent spatial adaptive graphs, time adaptive graphs, and channel adaptive graphs, respectively.

[0125] This indicates an expanded hyperconnection adjacency matrix;

[0126] Among them G i ∈{G1,G2,G3}, where G1 represents a first-order hyperlink adjacency matrix, G2 represents a second-order hyperlink adjacency matrix, G3 represents a third-order hyperlink adjacency matrix, α represents the expansion function, v is the number of nodes used to construct the hyperlink adjacency matrix, and Q∈R. v×25 To initialize the transformation matrix.

[0127] S4-2: Expand the dimensions of the hyperconnection adjacency matrix and combine it with its mask matrix and multi-dimensional adaptive graph to obtain the hyperconnection adaptive adjacency matrix;

[0128] Based on the hyperconnected adaptive adjacency matrix, a graph convolution module is constructed, denoted as: hyperconnected adaptive graph convolution module.

[0129] like Figure 4 As shown, this is the hyperconnected adaptive graph convolution module in the technical solution of the present invention, where the input f in The skeleton sequence data is represented by a size of B×C×T×N, where B represents the batch size, number of channels, number of frames, and number of nodes, respectively; θ k (x) and φ k (x) represents the Gaussian embedding function of the nonlocal network, used to obtain the spatial, temporal, and channel plots. After the obtained spatial, temporal, and channel plots are processed by the softmax function, they are further transformed into time-adaptive plots through the temporal dimension transformation module (labeled as TemporalTM in the figure) and the channel dimension transformation module (labeled as ChannelTM in the figure). and channel adaptive graph and Combined with the extended hyperconnection adjacency matrix A Hyper B k and S k The final hyperconnection adaptive adjacency matrix is ​​obtained, i.e. Graph convolution operations are performed; finally, a 1×1 convolution is introduced to construct Res residual connections to ensure model training stability. The hyperconnection adaptive adjacency matrix in this invention artificially constructs long-distance node connections, enhancing the feature extraction capability for long-distance motion information, thereby improving the model's recognition accuracy.

[0130] Among them, for time graph T k ∈R T×T Perform dimensional transformation to generate a time-adaptive graph During the process, the TemporalTM time-dimensional transformation module works by transforming the time-dimensional map T... k ∈R T×TAs input, the dimensions are increased to 1×T×T using the `unsqueeze` function, then transformed to T×1×T using the `permute` function, followed by convolution to 1×1×T. The `permute` function is then used again to transform the dimensions to T×1×1. Batch normalization is then performed, followed by a second convolution to (N×N)×1×1. The `view` function reconstructs the data to N×N, and finally, the `softmax` function is used to calculate the time-adaptive graph.

[0131] Time plot Perform dimensional transformation to generate channel adaptive graph During the process, the ChannelTM module works by transforming the channel graph. As input, the dimension is increased to 1×C using the unsqueeze function. e ×C e Use the permute function to convert the dimension to C. e ×1×C e Then, a convolution operation is performed to transform it to 1×1×C. e Then, the permute function is used again to convert the dimension to C. e The process involves performing a ×1×1 operation, followed by batch normalization, a second convolution operation to transform it to (N×N)×1×1, reconstruction using the view function to N×N, and finally, calculation using the softmax function to obtain the channel adaptive map.

[0132] In this application, the temporal graph and channel graph are transformed using the temporal dimension transformation module TemporalTM and the channel dimension transformation module ChannelTM, and the transformed dimensions are adaptive to the spatial dimensions. Figure 1 This is to facilitate subsequent participation in graph convolution calculations.

[0133] S4-3: Add residual connections to the full-dimensional dynamic convolution ODConv to construct a residual full-dimensional dynamic convolution. In summary, the size of the residual full-dimensional dynamic convolution is 9×1 to ensure the best recognition performance.

[0134] Based on residual full-dimensional dynamic convolution, the temporal features of the skeleton sequence are extracted, and a temporal convolution module is constructed, denoted as: residual full-dimensional dynamic temporal convolution module.

[0135] The construction of a residual full-dimensional dynamic temporal convolution module includes the following steps.

[0136] S4301: Acquiring full-dimensional attention;

[0137] Attention to the input channels of the convolutional kernel is obtained through the SE (Squeeze and Excitation) module. Output channel attention Convolutional Spatial Attention Atten K Number of convolutions and attention. M The calculation formula is:

[0138]

[0139]

[0140] Atten K =σ(W4(δ(W1(Avgpool(f)) in )))))

[0141] Atten M =σ(W5(δ(W1(Avgpool(f)) in )))))

[0142] Where δ represents the ReLU activation function, σ represents the softmax activation function, Avgpool represents average pooling, W1, W2, W3, W4, and W5 represent the weight parameters of the fully connected layer, and C... in Indicates the input channel, C out K represents the output channel, K represents the kernel size, and M represents the number of kernels.

[0143] S4302: Calculate the weights of the full-dimensional dynamic convolution kernel based on the attention based on the convolution space and the number of convolutions:

[0144] W od =(Atten K Atten M W M

[0145] In the formula, W M W is the weight matrix for M convolutional kernels. od The weights are for the full-dimensional dynamic convolution.

[0146] S4303: Combining two residual connections, a 9×1 residual full-dimensional dynamic temporal convolution module is constructed to perform a convolution operation on the output of the hyper-connected adaptive graph convolution module. The calculation formula is as follows:

[0147]

[0148] In the formula, Y is the output of the residual full-dimensional dynamic temporal convolution, and X is the output of the hyperconnected adaptive graph convolution module.

[0149] like Figure 5The diagram shows an example of the working process of residual full-dimensional dynamic temporal convolution, f in The input features of the residual full-dimensional dynamic temporal convolutional layer are processed by the SE module to obtain the input channel attention. Output channel attention Convolutional Spatial Attention Atten K Number of convolutions and attention. M Further, through convolutional spatial attention Attenuation... K Attention and the number of convolutions M The weights W of the full-dimensional dynamic convolution are obtained through aggregation calculation. od Then, focus on the input channel. Acting directly on f in The process begins by combining residual connections to obtain the input for convolution calculations, followed by the extraction of temporal features using full-dimensional dynamic convolution, and finally, the input is combined with a second residual connection and output channel attention. Obtain the output feature f out In this method, based on residual full-dimensional dynamic temporal convolution, temporal features of the skeleton sequence are extracted in the temporal domain of the hyperconnected graph convolutional network model, and full-dimensional attention is calculated to fully obtain the contextual features in the temporal domain, thereby enhancing the extraction capability of temporal features. At the same time, two residual connections are added to construct a residual full-dimensional dynamic temporal convolution module, which makes the temporal features more smoothly transmitted in the information process on the basis of stronger temporal feature extraction, and reduces the loss of temporal feature information during the calculation process.

[0150] S4-4: Combine the hyperconnected adaptive graph convolution module and the residual full-dimensional dynamic temporal convolution module to construct a hyperconnected graph convolution layer;

[0151] like Figure 6 As shown, the hyperconnected graph convolutional layer includes: a hyperconnected adaptive graph convolutional module (labeled HC-AGCN in the figure), a batch normalization (BN) layer, a ReLU layer, a residual full-dimensional dynamic temporal convolutional module (labeled ROD-TCN in the figure), a BN layer, and a ReLU layer connected sequentially. The skeleton features are processed by the hyperconnected adaptive graph convolutional module HC-AGCN to extract long-distance dependencies, then activated by the ReLU activation function and batch normalized by the BatchNorm2d function. These are then input into the residual full-dimensional dynamic temporal convolutional module ROD-TCN to obtain temporal features with stronger representational capabilities. Finally, by constructing selective residual connections (Res), the model's flexibility is enhanced, feature information loss during training is reduced, and model performance is ensured.

[0152] S4-5: Construct a skeleton behavior recognition model;

[0153] like Figure 7As shown, the skeleton behavior recognition model includes: sequentially connected BN layers and 10 consecutive hyperconnected graph convolutional layers (labeled as L1 to L2 in the figure). 10 The graph consists of a GAP layer and a softmax layer. The three numbers on each graph convolutional block represent the input channel, output channel, and stride, respectively.

[0154] S5: Obtain the video data to be recognized, extract the skeleton sequence from the video data group to be recognized, denoted as: skeleton sequence to be recognized; input the skeleton sequence to be recognized into the trained skeleton behavior recognition model to obtain the final skeleton behavior recognition result.

[0155] In this embodiment, the skeleton behavior recognition model is trained using the PyTorch deep learning framework. The optimization strategy employed is stochastic gradient descent (SGD), with Nesterov momentum set to 0.9, an iteration period of 50, and an initial learning rate of 0.1. To overcome overfitting during training, cross-entropy is chosen as the loss function, and the weight decay is set to 0.0001, with decay periods at the 30th and 40th periods. The batch size is set to 32.

[0156] Table 1 shows an example of the hyperconnected graph convolutional network model structure. Based on the network structure in Table 1, the input data is processed through 10 HC-GCN (hyperconnected graph convolutional network) layers and then enters the GAP layer. In the GAP layer, the 3D data output by the sub-network structure block is converted into 1D data. Then, the data is reduced from 480,000 dimensions to 60 (120) dimensions through the FC layer. Finally, the data is mapped to 60 (120) dimensions for prediction.

[0157] Table 1: Structure of Hyperconnected Graph Convolutional Network Model

[0158]

[0159] To verify the effectiveness and practicality of the human behavior recognition method in the technical solution of this invention, the NTU-RGB+D and NTU-RGB+D120 datasets were selected as experimental datasets for the experiment.

[0160] The NTU-RGB+D dataset was acquired using three Microsoft Kinect v2 sensors positioned at -45°, 0°, and 45° angles, encompassing a total of 60 action categories. Actions 1-49 are single-person actions, while actions 50-60 are two-person interactive actions. The NTU-RGB+D dataset employs two evaluation methods: cross-view (X-View) and cross-object (X-Sub). The test results are shown in Tables 2 and 3, respectively.

[0161] Table 2: Recognition accuracy (%) under X-View in the NTU-RGB+D dataset

[0162]

[0163] Table 3: Recognition accuracy (%) under X-Sub in the NTU-RGB+D dataset

[0164]

[0165] In Tables 2 and 3, Joint_merge and Bone_merge represent node and bone fusion features, respectively; G i The corresponding accuracy represents the recognition result when the i-th order hyperconnected adjacency matrix participates in graph convolution. G1+G2+G3 corresponds to the accuracy obtained by fusing the recognition results of the three hyperconnected adjacency matrices participating in graph convolution. In the table, HC-GCN represents the skeleton behavior recognition model constructed based on the hyperconnected graph convolution network model in the technical aspect of this invention. The corresponding accuracy is obtained by fusing the outputs of the six branches to obtain the final skeleton behavior recognition result.

[0166] Meanwhile, based on the NTU-RGB+D dataset, the hyperconnected graph convolutional network model in the technical solution of this invention is compared with other skeleton behavior recognition models. The comparison results are shown in Table 4 below:

[0167] Table 4: Comparison of recognition accuracy (%) of different models based on the NTU-RGB+D dataset

[0168] Model X-sub X-view STA-LSTM 73.4 81.2 VA-LSTM 79.2 87.7 TCN 74.3 83.1 HCN 86.5 91.1 AS-GCN 86.8 94.2 ST-GCN 81.5 88.3 2s-AGCN 88.5 95.1 FGCN 90.2 96.3 HC-GCN 91.2 96.7

[0169] As can be seen from the data in Table 4, on the NTU dataset divided into X-view and X-sub methods, the HC-GCN model of this method achieves the highest recognition accuracy compared with other models, at 96.7% and 91.2% respectively. The technical solution of this invention can improve the recognition accuracy of skeleton behavior.

[0170] Compared to the NTU-RGB+D dataset, the NTU-RGB+D120 dataset contains more samples performed by more subjects in more different camera views. This dataset contains 120 actions performed by 106 different subjects. The NTU-RGB+D120 dataset provides two benchmarks for evaluation: cross-object (X-Sub) and cross-set (X-Set). Based on the NTU-RGB+D120 dataset, the hyperconnected graph convolutional network model in this invention is compared with other skeleton action recognition models; the comparison results are shown in Table 5.

[0171] Table 5: Comparison of recognition accuracy (%) of different models based on the NTU-RGB+D120 dataset

[0172] Model X-sub X-set Soft RNN 36.3 44.9 Dynamic Skeleton 50.8 54.7 Spatio-Temporal LSTM 55.7 57.9 GCA-LSTM 58.3 59.2 SGN 79.2 81.5 ST-GCN 72.4 71.3 2s-AGCN 82.9 84.9 4s Shift-GCN 85.9 87.6 <![CDATA[HC-GCN(G1+Joint_merge)]]> 82.0 83.4 <![CDATA[HC-GCN(G1+Bone_merge)]]> 84.4 85.9 <![CDATA[HC-GCN(G2+Joint_merge)]]> 82.0 83.9 <![CDATA[HC-GCN(G2+Bone_merge)]]> 84.8 86.0 <![CDATA[HC-GCN(G3+Joint_merge)]]> 82.4 83.4 <![CDATA[HC-GCN(G3+Bone_merge)]]> 84.2 85.4 HC-GCN 87.6 89.0

[0173] As can be seen from the data in Table 5, on the NTU-RGB+D120 dataset divided by X-sub and X-set, the HC-GCN model of this method achieves the highest recognition accuracy, at 87.6% and 89.0% respectively, further verifying the good recognition accuracy of the HC-GCN model. Furthermore, the present invention can achieve good recognition accuracy on both the NTU-RGB+D dataset and the NTU-RGB+D120 dataset, indicating that the skeleton behavior recognition model in this application has good generalization performance.

[0174] This invention provides a skeleton behavior recognition method based on a hyperconnected graph convolutional network. It constructs a hyperconnected adjacency matrix based on the connection relationships of the human skeleton, further builds a hyperconnected adaptive graph convolutional module, and introduces two residual connections into the full-dimensional dynamic convolution to construct a residual full-dimensional dynamic temporal convolution module. These two modules are combined and trained end-to-end based on fused feature data to obtain a hyperconnected graph convolutional network model. The hyperconnected graph convolutional network model in this method can fully extract long-distance dependencies of nodes from skeleton feature data, while reducing feature loss and enhancing the representation ability of temporal features. When modeling skeleton sequences, this method fully considers long-distance dependencies of nodes, strengthening the recognition effect for actions with long-distance motion information. Simultaneously, the construction of residual full-dimensional dynamic temporal convolution reduces feature loss and enhances the extraction ability of temporal features. Finally, it achieves excellent recognition accuracy on large skeleton datasets and has good generalization performance.

Claims

1. A skeleton behavior recognition method based on hyperconnected graph convolutional networks, comprising the following steps: S1: Obtain the original video sample, preprocess the original video sample, and obtain the skeleton information data in the original video sample; Its characteristic is that it further includes the following steps: S2: Based on the skeleton information data, perform data processing, extract the skeleton sequence, obtain skeleton motion features based on the skeleton sequence, and perform feature fusion on the skeleton motion features, denoted as: training fusion feature data; The skeletal motion features include: first-order motion features, second-order motion features, and third-order motion features; S3: Based on the skeleton physical connection relationship, construct the hyperconnection adjacency matrix by adding the connection relationship beyond the adjacent node; S4: Construct a hyperconnected graph convolutional network model as a skeleton behavior recognition model, and train the skeleton behavior recognition model based on the training fusion feature data to obtain a trained behavior recognition model; Constructing the hyperconnected graph convolutional network model includes the following steps: S4-1: Construct a multi-dimensional adaptive graph based on the skeleton sequence; The multi-dimensional adaptive graph includes: spatial adaptive graph, temporal adaptive graph, and channel adaptive graph; The spatial adaptive graph is a spatial graph extracted based on the skeleton sequence; The time-adaptive graph is generated by transforming the dimensionality of the time graph extracted from the skeleton sequence. The channel adaptive graph is generated by transforming the channel graph extracted from the skeleton sequence. S4-2: Expand the dimensions of the hyperconnection adjacency matrix and combine it with its mask matrix and the multi-dimensional adaptive graph to obtain the hyperconnection adaptive adjacency matrix; Based on the hyperconnected adaptive adjacency matrix, a graph convolution module is constructed, denoted as: hyperconnected adaptive graph convolution module; S4-3: Add residual connections to full-dimensional dynamic convolution to construct residual full-dimensional dynamic convolution; Based on the residual full-dimensional dynamic convolution, the temporal features of the skeleton sequence are extracted to construct a temporal convolution module, denoted as: residual full-dimensional dynamic temporal convolution module; S4-4: Combine the hyperconnected adaptive graph convolution module and the residual full-dimensional dynamic temporal convolution module to construct a hyperconnected graph convolution layer; The hyperconnected graph convolutional layer includes: a hyperconnected adaptive graph convolutional module, a BN layer, a ReLU layer, a residual full-dimensional dynamic temporal convolutional module, a BN layer, and a ReLU layer connected in sequence; S4-5: Construct the skeleton behavior recognition model; The skeleton behavior recognition model includes: a BN layer connected in sequence, 10 consecutive hyperconnected graph convolutional layers, a GAP layer, and a softmax layer; S5: Obtain the video data to be identified, extract the skeleton sequence from the video data group to be identified, denoted as: skeleton sequence to be identified; input the skeleton sequence to be identified into the trained skeleton behavior recognition model to obtain the final skeleton behavior recognition result.

2. The skeleton behavior recognition method based on hyperconnected graph convolutional networks according to claim 1, characterized in that: Step S1 involves obtaining the skeleton information data from the original video sample, which includes the following steps: S1-1: Perform frame segmentation processing on the acquired original video samples to decompose the continuous video segments into an image sequence including static frames; S1-2: Calculated based on the Openpose pose estimation algorithm; Set the calculation parameters of the Openpose algorithm, input the images of the static frames obtained from the decomposed video into Openpose, and obtain the human skeleton data of the corresponding number of nodes in the static frames; S1-3: Based on the human body nodes and the corresponding joint numbers in the Openpose algorithm, construct the connection relationship of the human skeleton data to represent the morphological characteristics of the human body, that is, obtain the skeleton information data.

3. The skeleton behavior recognition method based on hyperconnected graph convolutional networks according to claim 1, characterized in that: In step S2, skeleton motion features are obtained based on the skeleton sequence, and feature fusion is performed on the skeleton motion features to obtain the fused feature data for training. Specifically, this includes the following steps: S2-1: Suppose that the skeleton information data contains N nodes and T frames, the complete skeleton sequence can be constructed as L=(S,E); Where S = {s ti |t=1,2,...,T,i=1,2,...,N} represents the set of nodes, and E={E S E T } represents the set of skeletal edges, which is constructed by connecting adjacent nodes within the same frame to form the set of spatial edges E. S The connection of the same node between adjacent frames constructs the set of temporal edges E. T ; S2-2: Using the node itself as the first-order motion feature of the node, assuming that the first-order motion feature Joint of the node in frame t is represented as J t ; S2-3: Use the node closest to the bone's center of gravity as the source node. Target nodes are those furthest from the center of gravity of the skeleton. The first-order motion feature of the skeleton, represented by a bone, is the vector pointing from the source node to the target node. Since the number of nodes is one more than the number of bones, the center node is set to an empty bone, i.e., a value of 0. S2-4: The second-order motion features of nodes and skeletons are represented by the coordinate difference between their respective first-order motion features in two adjacent frames. Therefore, the formula for calculating the second-order motion feature Joint_motion of nodes is: The formula for calculating the second-order motion feature of the skeleton, bone_motion, is as follows: Since the number of time frames is one more than the dimension of the second-order motion feature, the value of the last frame is set to 0. S2-5: The third-order motion features of nodes and skeletons are represented by the coordinate difference between their respective second-order motion features in two adjacent frames. Therefore, the formula for calculating the third-order motion feature of nodes, Joint_accelerate, is as follows: The formula for calculating the third-order motion feature of the skeleton, Bone_accelerate, is as follows: Since the number of time frames is 2 more than the dimension of the third-order motion feature, the values ​​of the last two frames are set to 0. S2-6: Based on the first-order motion features, second-order motion features, and third-order motion features of the above nodes and bones, the channel dimensions are concatenated using the Concatenation function to obtain the training fusion feature data. The training fusion feature data includes: node fusion feature Joint_merge and bone fusion feature Bone_merge.

4. The skeleton behavior recognition method based on hyperconnected graph convolutional networks according to claim 1, characterized in that: In step S3, the hyperconnection adjacency matrix is ​​constructed by adding connections beyond neighboring nodes, including the following steps: S3-1: Constructing an adjacency matrix G1∈R based on the physical connections of the human skeleton 25×25 ; S3-2: Using the chest node as the center node, connect nodes with a gap of one node to construct a second-order hyperconnection adjacency matrix G2∈R. 13×13 ; S3-3: Using the chest node as the central node, connect nodes with a gap of two to construct a third-order hyperconnection adjacency matrix G3∈R. 9×9 .

5. The skeleton behavior recognition method based on hyperconnected graph convolutional networks according to claim 1, characterized in that: In step S4-1, the step of constructing the hyperconnected adaptive graph convolution module includes: S4101: Calculate the spatial graph S using the softmax function. k Time graph T k Channel diagram C k The calculation formulas are as follows: In the formula, f in W represents the input features. θ W φ Let θ and φ represent the embedded Gaussian functions, respectively. Among them, spatial diagram S k ∈R N×N That is, a spatially adaptive graph, relative to a time graph T k ∈R T×T and channel diagram Perform dimensional transformation to generate a time-adaptive graph. and channel adaptive graph R N×N In this context, N represents the number of skeleton nodes. C in e The number of transition channels is 0.25 × C. in ), C in Indicates the input channel; k represents the layer index number of each graph; R T×T In this context, T indicates that the skeleton information data contains the total number of frames; S4102: Combine the adaptive graphs of each dimension and the extended hyperconnected adjacency matrix to generate a hyperconnected adaptive adjacency matrix, and then perform graph convolution operations: in, C represents the input features and the output of the graph convolution, respectively. in Indicates the input channel, C out Indicates the output channel; W k K is the convolution kernel function. v Set to 3, B k To enhance the data mask matrix, S k , and These represent spatial adaptive graphs, time adaptive graphs, and channel adaptive graphs, respectively. This indicates an expanded hyperconnection adjacency matrix; Among them G i ∈{G1,G2,G3}, where G1 represents a first-order hyperlink adjacency matrix, G2 represents a second-order hyperlink adjacency matrix, G3 represents a third-order hyperlink adjacency matrix, α represents the expansion function, v is the number of nodes used to construct the hyperlink adjacency matrix, and Q∈R. v×25 To initialize the transformation matrix.

6. The skeleton behavior recognition method based on hyperconnected graph convolutional networks according to claim 5, characterized in that: In step S4101, the time graph T k and channel diagram C k Convert to time-adaptive graph and channel adaptive graph The method is as follows: Construct a conversion module; the conversion module uses a time map T k and channel diagram C k As input, with and For output; the working process of the conversion module is as follows: Let X be the input of the conversion module. k ∈R y×y In the conversion module, X k ∈R y×y The transformation module increases the dimension to 1×y×y using the unsqueeze function, converts the dimension to y×1×y using the permute function, performs a convolution operation to transform it to 1×1×y, uses the permute function again to convert the dimension to y×1×1, performs batch normalization, performs a second convolution operation to transform it to (N×N)×1×1, reconstructs it to N×N using the view function, and finally calculates the output of the transformation module using the softmax function.

7. The skeleton behavior recognition method based on hyperconnected graph convolutional networks according to claim 1, characterized in that: In step S4-3, constructing the residual full-dimensional dynamic temporal convolution module specifically includes the following steps: S4301: Acquiring full-dimensional attention; The input channel attention of the convolution kernel is obtained through the SE module. Output channel attention Convolutional Spatial Attention Atten K Attention and the number of convolutions M The calculation formula is: Atten K =σ(W4(δ(W1(Avgpool(f in ))))) Atten M =σ(W5(δ(W1(Avgpool(f in ))))) Where δ represents the ReLU activation function, σ represents the softmax activation function, Avgpool represents average pooling, W1, W2, W3, W4, and W5 represent the weight parameters of the fully connected layer, and C... in Indicates the input channel, C out K represents the output channel, K represents the kernel size, and M represents the number of kernels. S4302: Calculate the weights of the full-dimensional dynamic convolution kernel based on the attention based on the convolution space and the number of convolutions: W od =(Atten K Atten M )W M In the formula, W M W is the weight matrix for M convolutional kernels. od These are the weights for the full-dimensional dynamic convolution; S4303: Combining two residual connections, a 9×1 residual full-dimensional dynamic temporal convolution module is constructed to perform a convolution operation on the output of the hyper-connected adaptive graph convolution module. The calculation formula is as follows: In the formula, Y is the output of the residual full-dimensional dynamic temporal convolution, and X is the output of the hyperconnected adaptive graph convolution module.

Images