A skeleton action recognition method based on space-time dependency enhanced network

Abstract

The application relates to the technical field of action recognition, in particular to a skeleton action recognition method based on a space-time dependence enhanced network, which comprises the following steps: in the SDE-Net, a FEM module is used to extract local and global features, local information is fused with output information of CNNs in a global average pooling layer through a short connection mode; in the OPM, the positions of joints in grid data are adjusted, so that joints with structural relationships are located at adjacent positions in the grid data; grid data with a height of T and a width of V are constructed as the input of the CNNs, the grid data are processed through the CNNs, correlations between different joints of adjacent frames are established, and the space-time dependence of the SDE-Net is enhanced. The application solves the problems of the existing GCN method, the interframe edge only connects the same joints between adjacent frames, the connection between different joints is cut off, direct information exchange across space-time is hindered, and complex regional space-time dependence is difficult to capture.

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, and in particular to a skeleton action recognition method based on a spatiotemporal dependency enhancement network. Background Technology

[0002] Skeletal motion recognition extracts information about the joints of the human skeleton. Skeletal behavior recognition focuses only on changes in the joints and movement trajectories, without considering the effects of occlusion, viewpoint changes, or complex backgrounds. Therefore, it has higher recognition accuracy and lower computational complexity. Recent studies have shown that skeleton motion recognition is a promising method that can analyze athletes' movement skills and postures and provide suggestions for improvement. In addition, skeleton motion recognition has wide applications in intelligent monitoring, motion-sensing games, healthcare, and marketing advertising, providing intelligent, personalized, and precise technical support for various scenarios.

[0003] Skeleton data has received widespread attention in recent years due to its robustness in complex backgrounds and its efficiency in storage and computation. In skeleton-based action recognition tasks, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have been extensively studied due to their inherent advantages. However, these methods struggle to leverage the graph structure of the human skeleton. Yan et al. first proposed Graph Convolutional Networks (GCNs) to generalize convolution from images to graphs. They constructed a spatial graph based on the natural connections of human joints and created temporal margins between the same joints in adjacent frames. In recent years, GCN-based algorithms have significantly advanced the development of skeleton-based action recognition and have attracted increasing attention.

[0004] Currently, most GCN-based methods connect only the same joints between adjacent frames, severing the connection between different joints. This hinders direct information exchange across time and space, making it difficult to capture complex regional spatiotemporal dependencies. However, modeling complex spatiotemporal dependencies is key to powerful feature extractors. Summary of the Invention

[0005] To address the shortcomings of existing methods, this invention proposes the SDE-Net network, which leverages the powerful spatiotemporal modeling capabilities of CNNs to compensate for the deficiencies of GCNs. To obtain stable action descriptions, a FEM is designed, utilizing graph convolutional layers with kernels of different sizes to extract local and global features. Furthermore, short connections are used to directly transmit local information to the model's end, achieving effective utilization of local features. Considering the different input data processing methods of GCNs and CNNs, an OPM is designed, which adjusts the joint coordinate tensor by predicting the adjacency matrix, solving the problem of easily losing important joint topological information during data transmission between different networks. Finally, by combining with CNNs, the spatiotemporal dependencies between joints are enhanced.

[0006] The technical solution adopted in this invention is: a skeleton action recognition method based on a spatiotemporal dependency enhancement network, comprising the following steps:

[0007] Step 1: In SDE-Net, local and global features are extracted using FEM. Local information is fused with the output information of CNNs in the global average pooling layer through short connections.

[0008] In a preferred embodiment of the present invention, FEM consists of a two-layer network composed of four AGCN convolutional layers. The upper layer network uses two AGCN convolutional layers with 1x1 and 3x1 temporal convolutional kernels to obtain local features; the lower layer network uses two AGCN convolutional layers with 9x1 temporal convolutional kernels to obtain global features.

[0009] Step 2: Global features are used as input to OPM. In OPM, the positions of joints in the grid data are adjusted so that joints with structural relationships are in adjacent positions in the grid data.

[0010] In a preferred embodiment of the present invention, a fully connected matrix (Batch, N, N) is predicted from the FEM output feature map; the fully connected matrix is ​​multiplied by the joint coordinate tensor (Batch, V, 2); after passing through OPM, the matrix is ​​arranged in adjacent regions in the feature map of CNNs; the OPM outputs a feature map of size (N, C, T, V), where T and V represent the height and width of the feature map, respectively, and C represents the number of channels.

[0011] Step 3: Construct grid data with height T and width V as input to CNNs. Process the grid data through CNNs to establish correlations between different joints in adjacent frames, thereby enhancing the spatiotemporal dependence of SDE-Net.

[0012] In a preferred embodiment of the present invention, CNNs include four stages, each stage having two bottleneck layers; the input channels of the four stages are 128, 256, 512, and 1024, respectively; after the second and third stages, the CNNs use a normal k×k 2D convolution operator through average pooling.

[0013] As a preferred embodiment of the present invention, the vertex V of SDE-Net ti The sampling area is:

[0014]

[0015] Among them, B(v) t,i Let v be the sampling area of ​​keypoint i in frame t. q,j Let j represent a key point in frame q, where i and j represent different key points, and k is the convolution operator.

[0016] As a preferred embodiment of the present invention, the output of SDE-Net is:

[0017]

[0018] Where W(m,c,q,j) are the weights of the 2D convolution, representing the weight at position (q,j) of the c-th convolution kernel in filter m; each filter corresponds to one channel, Y and X represent the output and input feature maps respectively, b is the convolution bias, and V... t+q This represents a time frame that is q frames away from the current frame t.

[0019] As a preferred embodiment of the present invention, a skeleton action recognition system based on a spatiotemporal dependency enhancement network includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the skeleton action recognition method based on a spatiotemporal dependency enhancement network.

[0020] As a preferred embodiment of the present invention, a computer-readable medium storing computer program code implements a skeleton action recognition method based on a spatiotemporal dependency enhancement network when executed by a processor.

[0021] The beneficial effects of this invention are:

[0022] 1. Design an action feature extraction module (FEM) to extract local and global features using convolutional kernels of different sizes, and achieve effective utilization of local information through short connections;

[0023] 2. The Design Optimization Module (OPM) adjusts the joint coordinate tensor by predicting the joint adjacency matrix, which solves the problem that the exponential arrangement of human joints in grid data can easily lead to the loss of important topological information.

[0024] 3. By combining convolutional neural networks (CNNs), the correlation between different joints in adjacent frames is constructed, enhancing the spatiotemporal dependence of the network and enabling direct communication of information between different joints across time and space. Attached Figure Description

[0025] Figure 1 This is a diagram of the SDE-Net architecture of the present invention;

[0026] Figure 2 This is a schematic diagram illustrating the local and global feature extraction using FEM according to the present invention;

[0027] Figure 3 This is a schematic diagram of the OPM of the present invention;

[0028] Figure 4 This is a schematic diagram of the second and fourth stages of convolution in the CNNs of this invention;

[0029] Figure 5 This is a comparison chart of the FLOP and verification accuracy of the present invention with different numbers of convolutional layers;

[0030] Figure 6 This is a comparison diagram showing the impact of losing local features on recognizing various actions according to the present invention; Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0032] In practical applications, mainstream methods for action recognition include graph convolutional networks (GCNs) and convolutional neural networks (CNNs). Currently, most GCN-based methods interweave spatial-only and temporal-only modules, similar to 3D convolutions in factorization. A common approach is to first use spatial modules to extract spatial relationships within each time step, and then use 1D convolutional layers to model temporal relationships. This results in inter-frame edges only connecting the same joints between adjacent frames, severing connections between different joints. This hinders direct cross-temporal and spatial information exchange and makes it difficult to capture complex regional spatiotemporal dependencies. However, complex spatiotemporal dependency modeling is crucial for powerful feature extractors. For example, the action of "standing up" is generally performed by both the upper and lower body, with a close relationship between the forward leaning motion of the upper body and the future standing motion of the lower body. GCNs can utilize the structural information of the human skeleton, but lack flexibility in constructing cross-temporal connections between joints. CNNs have the ability to explore the spatiotemporal correlations of the skeleton, but they cannot effectively utilize the structural information between skeletons.

[0033] like Figure 1 As shown, a skeleton action recognition method based on a spatiotemporal dependency enhancement network includes the following steps:

[0034] SDE-Net establishes dependencies between different joints in adjacent frames within a region while preserving the structural information of the skeleton. This network inherits the advantages of graph convolutional networks and convolutional neural networks well when applied to skeleton-based action recognition tasks.

[0035] The FEM module is used to extract local and global features. Local information is fused with the output information of the convolutional neural network in the global average pooling layer through short connections. Global information is used as input to the OPM. In the OPM, the position of joints in the grid data is adjusted so that joints with structural relationships can be adjacent in the grid data. A grid data with height T and width V is constructed as input to CNNs. Finally, the CNNs process the data to establish correlations between different joints in adjacent frames (as shown in the dashed box), which enhances the spatiotemporal dependence of the network.

[0036] In a graph convolutional network based on skeleton action recognition, the GCN is divided into two parts: graph convolutional blocks (GC blocks) and temporal convolutional blocks (TC blocks). Given a skeleton sequence CxTxV, the graph convolutional blocks and temporal convolutional blocks aggregate features along the keypoint V and time frame T, respectively. The FEM uses the adaptive graph convolutional layer in 2S-AGCN, and the spatial part of the graph convolution can be represented as follows:

[0037] Y = ∑ k W k X(A k +B k +C k (1)

[0038] Where k represents different time frames, X and Y are the input and output feature maps respectively, and W represents the weights; A k It is an adjacency matrix defined by the human skeletal structure; B k It is a trainable adaptive matrix; C k It is a self-attention matrix.

[0039] Regarding the time aspect of graph convolution, SDE-Net vertex V ti The time sampling region can be formulated as:

[0040]

[0041] Where Γ is a predefined parameter used to specify the length of the time window, and the temporal convolutional block is composed of convolutional kernels of size Γx1, used to extract skeleton features from adjacent frames; B(v t,i ) represents the time sampling region of keypoint i in frame t, v q,i This represents the i-th joint point on the q-th frame.

[0042] Both spatial and temporal convolutional layers are followed by normalization and activation function layers, which complete the construction of basic blocks.

[0043] like Figure 2 The FEM shown consists of four AGCN convolutional layers. The top two AGCN convolutional layers use 1x1 and 3x1 temporal convolutional kernels to obtain local features; the bottom two convolutional layers use 9x1 temporal convolutional kernels to obtain global features; short connections are set to effectively utilize local temporal information.

[0044] To obtain accurate action descriptions, both local and global features are crucial. Therefore, a parallel structure is used to extract local and global information separately. In the local feature extraction part, adaptive graph convolutional layers are used, but instead of the fixed 9x1 temporal convolutional kernel used in AGCN, which is problematic because a large temporal kernel might cause the model to focus only on global features and ignore local details, small 1x1 and 3x1 convolutional kernels are used. These small kernels can effectively extract local temporal information. Local details extracted in shallow layers are ignored as convolution deepens and cannot be effectively utilized. Therefore, short connections are set in the local feature extraction part to skip other convolutional parts, thus ensuring the integrity of local information. In addition, considering the problem that the computational cost of SDE-Net will increase sharply, only four adaptive graph convolutional layers are used in FEM, reducing the computational cost while ensuring model performance.

[0045] like Figure 3 The diagram shows OPM. The most direct way to create SDE-Net is to directly use the output of FEM as the input to CNNs. However, this would destroy the graph topology built in FEM because the sampling regions in CNNs are implemented by the spatial order of feature maps determined by joint indices. However, this index cannot accurately represent the structural information of human joints. For example, the "right hand" and "right thumb" are naturally connected in human joints. In the NTU-RGB+D dataset, the joint indices of the "right hand" and "right thumb" are 12 and 25, respectively. If sorted according to the joint indices, the "right hand" and "right thumb" are not adjacent joints and are not in the same sampling region in the input feature map of CNNs. Therefore, it is impossible to obtain the structural information between these two joints. The output features of FEM are processed by OPM and then used as the input to CNNs, so that SDE-Net can establish the correlation between different joints between adjacent frames within the region, while also having important joint structural information.

[0046] Specifically, firstly, a fully connected matrix (Batch, N, N) is predicted from the output feature map of FEM, where N represents the number of joints. Then, a matrix multiplication is performed between the fully connected matrix and the joint coordinate tensor (Batch, V, 2). This allows the structural information between joints to be reflected in the input feature map of CNNs. The fully connected matrix used is predicted based on the features of all joints in the context. Joints like "right hand" and "right thumb," after optimization, will be arranged in adjacent regions in the CNN feature map. Even joints without natural connections in the human skeleton will be captured and reflected in the input feature map of CNNs if they are related in movement. Finally, OPM outputs a feature map of size (N, C, T, V), where T and V represent the height and width of the feature map, respectively, and C represents the number of channels.

[0047] GCNs (Graph Convolutional Networks) model the spatiotemporal dependencies of the human skeleton using graph convolutional layers and obtain structural information about the joints. Then, a CNN (Neural Network Array) with strong spatiotemporal modeling capabilities is needed to achieve complex modeling of these dependencies. Because the skeleton data is relatively small, the CNN used has only four stages, each with two bottleneck layers. The input channels for the four stages are 128, 256, 512, and 1024, respectively. After the second and third stages, average pooling is used to reduce the features by half. After OPM (Optical Programming Process), connected joints appear closer together in the image.

[0048] like Figure 4 As shown, the process of enhancing the spatiotemporal dependency between joints involves the generation of inter-frame edges in CNNs. There are four convolutional stages in CNNs. Figure 4 The second and fourth stages are illustrated. The dashed boxes show the complex dependencies established between the joints of the left leg and surrounding joints in both time and space as convolution deepens. For clarity, green, light green, blue, and orange represent joints at different distances. As convolution deepens in CNNs, SDE-Net establishes more inter-frame edges, enabling direct cross-temporal and spatial communication of joint information. Given C channels of input data, CNNs use a k×k normal 2D convolution operator, and the SDE-Net vertex V... ti The sampling area is:

[0049]

[0050] Compared to formula (2), the sampling area includes different key points of people between adjacent frames; the output of SDE-Net can be formulated as:

[0051]

[0052] Where W(m,c,q,j) are the weights of the 2D convolution, representing the weight at position (q,j) of the c-th convolution kernel in filter m; each filter corresponds to one channel, Y and X represent the output and input feature maps respectively, b is the convolution bias, and V... t+q This represents a time frame that is q frames away from the current frame t.

[0053] Formula (4) shows that at the joint point v ti The output at a given point is a weighted sum of all nodes in the spatiotemporal neighborhood. The direct information exchange between key points across spatiotemporal regions illustrates the enhanced spatiotemporal dependence of the network.

[0054] Experimental procedure:

[0055] The NTU RGB+D dataset contains 56,880 action clips covering 60 different action classes. Each clip contains the 3D coordinates of 25 key points, forming a skeleton sequence. Cross-subjects (CS) and cross-view (CV) evaluation methods were used to verify the model's generalization ability and its action recognition ability under different camera views.

[0056] The NTU-120RGB+D dataset contains 114,480 action samples across 120 different action categories. Cross-subject (CS) evaluation and cross-setting (CE) validation were used to assess the model’s generalization ability and robustness across different subjects and recording settings.

[0057] For the selection of loss function and optimization strategy, cross-entropy was used as the loss function, and stochastic gradient descent (SGD) with Nesterov momentum (0.9) was used as the optimization strategy; the batch size was set to 16, and the weight decay was set to 0.0001; the initial learning rate was set to 0.1, which was multiplied by 0.1 in stages 40 and 50 to reduce the learning rate; the entire training process lasted for 65 stages; for the NTU-RGB+D dataset, the number of skeleton frames in each sequence was adjusted to 300.

[0058] A multi-stream fusion strategy is adopted, which sums the scores of different streams in a weighted summation manner to obtain the final prediction.

[0059] Table 1 Comparison of methods on the NTU-RGB+D dataset

[0060]

[0061]

[0062] The results are shown in Tables 1 and 2. Under the CS evaluation criteria in Table 1, the recognition performance of SDE-Net of the present invention is lower than that of ICE-GCN, which may be due to improper settings of some hyperparameters in the model, which weakens the generalization ability of the model. The results presented in Table 2 are obtained from experiments on a larger dataset. A larger dataset will reduce the sensitivity of the model to hyperparameters and help improve the generalization ability of the model. Therefore, Table 2 shows good results under both the CS and CE evaluation criteria.

[0063] Ablation experiments: to verify the effectiveness of each component of SDE-Net, based on the cross-view (CV) evaluation criteria of the NTU RGB+D dataset, i.e. only the original skeleton sequence is used as the input of the model, without constructing other streams.

[0064] SDE-Net essentially connects GCN and CNN. To verify the effectiveness of this connection, the experimental accuracy of three models was compared: the baseline of FEM (AGCN network), the baseline of the convolutional neural network used in SDE-Net (ResNeSt), and SDE-Net itself. To achieve the baseline of the convolutional neural network, four 2D convolutional layers with a kernel size of 3x3 were used to extract features from the input skeleton sequence. The extracted features were then used as input to the convolutional neural network for further processing. The output channels of these four convolutional layers were 64, 64, 64, and 128, respectively. Short connections were used to connect the local features extracted by the 2D convolutional layers to the convolutional neural network. The output of the network is transmitted together to the global average pooling layer, and the final score is predicted through the fully connected layer and softmax. The results of the three models are shown in Table 3. It can be seen that the addition of the convolutional neural network significantly improves the recognition effect of the graph convolutional network. SDE-Net improves the performance of the FEM baseline from 93.7% to 95.4%. CNN can aggregate features along the time and space dimensions at the same time. However, CNN cannot utilize the topological information between joints. Therefore, the experimental results of the CNN baseline are poor. SDE-Net improves the performance of the convolutional neural network baseline by 2.3%. The experiment proves the effectiveness of connecting FEM and convolutional neural network in SDE-Net.

[0065] Table 3 Comparison of validation accuracy of different models

[0066]

[0067] Action Feature Extraction Module: Experiments explain why AGCN is used as the basic unit of graph convolution in FEM and why only four AGCN convolutional layers are used. First, the impact of graph convolution methods is studied, and two other graph convolution methods are used to replace the basic graph convolutional unit in the action feature extraction module, including CTR-GCN and DGCN. As shown in Table 4, the model with adaptive graph convolution achieves the best results. Therefore, in this work, adaptive graph convolution is used as the basic convolutional unit of the action feature extraction module by default.

[0068] Table 4 Comparison of SDE-Net validation accuracy based on different graph convolution methods

[0069]

[0070] Considering the high computational complexity of SDE-Net, the number of graph convolutional layers was adjusted in the action feature extraction module design while maintaining model performance. AGCN used 10 adaptive graph convolutional layers, while SDE-Net of this invention only uses 4. To verify the impact of the number of convolutional layers in FEM on model complexity and performance, four experiments were conducted, adjusting the number of convolutional layers to 4L, 6L, 8L, and 10L respectively. Figure 5 The data presented in the figure shows the experimental data, and the figure also provides relevant data for AGCN alone for comparison. It can be seen that as the number of convolutional layers increases, the overall computational cost of SDE-Net also increases, but the model's validation accuracy does not increase accordingly. This is because as the number of convolutional layers increases, the model may lose some important features. This is because each convolutional layer extracts and compresses the input information, and stacking convolutional layers multiple times may lose some key features, thus affecting the model's performance.

[0071] Figure 5 The network with the best performance is represented by a red pentagram. SDE-Net 4L indicates that four graph convolutional layers were used in the action feature extraction module, while two convolutional layers were always used to extract local features in the comparative experiments.

[0072] Local Features: Local features are extracted in FEM using small-sized temporal convolutional kernels and directly transmitted to the end of the model via short connections, allowing the network to effectively utilize local temporal information. To verify the importance of local features to SDE-Net, 30 action classes were randomly selected from the NTU-RGB+D dataset, and the experimental accuracy of the network on each action class after removing the local feature extraction part from FEM is shown in a bar chart. The complete results of SDE-Net are also presented for comparison. Figure 6As shown, SDE-Net suffers from a decrease in recognition accuracy for 27 action categories due to the loss of local features; however, SDE-Net performs poorly in recognizing actions such as "Walking towards" and "Walking apart". This is because the key to these two actions is the direction and distance of walking, rather than the specific gait or action details. Therefore, focusing only on global features can better capture the temporal and spatial features of the walking process.

[0073] Optimization Module: To verify the necessity of the optimization module in SDE-Net, a variant network of SDE-Net was set up. The variant network is based on SDE-Net without the optimization module, and the action feature extraction module and the convolutional neural network are directly connected, that is, the output of the former is directly used as the input of the latter. The verification accuracy comparison of SDE-Net and its variant network is shown in Table 5. The table also shows the performance of AGCN for comparison. It can be seen that the performance of the variant network is lower than that of the SDE-Net designed in this invention, but its performance is still significantly higher than AGCN by 1.3%. This once again proves the significance of connecting FEM and convolutional neural network. Although SDE-Net with optimization module only improves by 0.4% compared with the simple connected variant network, it also verifies the role of optimization module.

[0074] Table 5 Comparison of Validation Accuracy of SDE-Net and its Variant Networks

[0075]

[0076]

[0077] This invention demonstrates the feasibility of using convolutional neural networks to compensate for graph convolutional networks. SDE-Net inherits the advantages of GCNs and CNNs, possessing strong spatiotemporal dependency modeling capabilities, providing a new approach for skeleton-based action recognition. By using adaptive graph convolutional layers with temporal convolutional kernels of different sizes, local and global features are obtained respectively, and short connections are set to ensure the effective use of local temporal information. An OPM is designed to connect FEM and convolutional neural networks, solving the problem of loss of important joint structure information caused by data transmission between different networks.

[0078] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A skeleton action recognition method based on a spatiotemporal dependency enhancement network, characterized in that, Includes the following steps: Step 1: In SDE-Net, local and global features are extracted using the FEM module. Local information is fused with the output information of CNNs in the global average pooling layer through short connections. FEM has four AGCN convolutional layers. The upper network uses two AGCN convolutional layers with 1x1 and 3x1 temporal convolutional kernels to obtain local features. The lower-level network consists of two AGCN convolutional layers that use 9x1 temporal convolutional kernels to obtain global features; Predict the fully connected matrix (Batch, N, N) from the FEM output feature map; perform matrix multiplication between the fully connected matrix and the joint coordinate tensor (Batch, V, 2); after passing through OPM, they are arranged in adjacent regions in the feature map of CNNs; OPM outputs a feature map of size (N, C, T, V), where T and V represent the height and width of the feature map, respectively, and C represents the number of channels; Step 2: Global features are used as input to OPM. In OPM, the positions of joints in the grid data are adjusted so that joints with structural relationships are in adjacent positions in the grid data. Step 3: Construct grid data with height T and width V as input to CNNs. Process the grid data through CNNs to establish correlations between different joints in adjacent frames, thereby enhancing the spatiotemporal dependence of SDE-Net.

2. The skeleton action recognition method based on spatiotemporal dependency enhancement network according to claim 1, characterized in that, CNNs consist of four stages, each with two bottleneck layers; the input channels for the four stages are 128, 256, 512, and 1024 respectively; after the second and third stages, average pooling is used. 2D convolution operators.

3. The skeleton action recognition method based on spatiotemporal dependency enhancement network according to claim 1, characterized in that, SDE-Net vertices The sampling area is: in, In order to be in t On-frame key points i The sampling area Indicates in q Key points on the frame j , i and j Each represents a different joint. k It is a convolution operator.

4. The skeleton action recognition method based on spatiotemporal dependency enhancement network according to claim 1, characterized in that, The output of the SDE-Net network is: in, These are the weights of the 2D convolution, representing the weights in the filter. m Middle, No c convolution kernels Positional weight; Y and X These represent the output and input feature maps, respectively. b It is convolution bias. Represents the current frame t difference q The time frame of the frame.

5. A skeleton action recognition system based on a spatiotemporal dependency enhancement network, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the skeleton action recognition method based on a spatiotemporal dependency enhancement network as described in any one of claims 1-4.

6. A computer-readable medium storing computer program code, characterized in that, The computer program code implements the skeleton action recognition method based on spatiotemporal dependency enhancement network as described in any one of claims 1-4 when executed by a processor.

Images