Human Action Recognition Method Based on Hierarchical Spatiotemporal Transformer Network

Abstract

A human motion recognition method based on a hierarchical spatiotemporal Transformer network comprises the following steps: collecting motion videos, processing motion videos, constructing a motion dataset, extracting training and testing branches, constructing a hierarchical spatiotemporal Transformer network, training the hierarchical spatiotemporal Transformer network, and testing the hierarchical spatiotemporal Transformer network. Based on the constructed motion dataset, this invention employs a hierarchical spatiotemporal Transformer network to extract high-order spatial feature information of joints at different levels, long- and short-term temporal dependencies, and correlation information between different joints across frames. Compared with existing skeleton motion recognition methods, this invention has advantages such as high recognition accuracy and fast recognition speed, and can be applied to the field of computer-aided human motion recognition technology.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, specifically behavior recognition methods, and involves a method for computer recognition of human actions. Background Technology

[0002] Action recognition methods refer to the automated identification and classification of human movements using computer vision, image processing, and pattern recognition technologies. The action recognition process involves extracting features from video, image, or sensor data and then categorizing them into different action types using classification methods.

[0003] Existing human action recognition methods typically employ skeleton-based approaches. For example, some researchers have proposed Spatiotemporal Graph Convolutional Networks (ST-GCN), which uses a fixed graph that conforms to the physical connectivity of the human skeleton but lacks flexibility. To better adapt to different action representations, some researchers have proposed dynamic graph convolutional networks that utilize contextual features of joints to learn the correlations between each joint pair. Considering that different actions require attention to different joints, some researchers use Transformer self-attention mechanisms to reconstruct the spatiotemporal dependencies of the human skeleton. However, current Transformer-based methods neglect the unique features of human skeleton data and fail to consider the unique physical functions of different body joints and the involvement of recurring joint groups in certain specific actions. Furthermore, they also have problems capturing the correlations between different joints between frames. Therefore, research on action recognition holds significant promise and importance. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a human action recognition method based on a hierarchical spatiotemporal Transformer network with high action recognition accuracy and fast recognition speed.

[0005] The technical solution adopted to solve the above technical problems consists of the following steps:

[0006] (1) Collect action videos

[0007] 3,100 action videos were shot using cameras, comprising 62 categories with 50 videos in each category.

[0008] (2) Processing motion video

[0009] All 3100 motion videos were processed to a uniform frame rate of 300 frames per second (384×288 resolution). The HRNet method was used to identify each video, resulting in 3100 joint data points J, each containing the coordinates and scores of 18 human keypoints across all frames in each video.

[0010] J = {J1, J2, ..., J} T}

[0011] J T ={f1,f2,...,f b}

[0012] f b =(x,y,z)

[0013] Where T is the number of video frames, f b Let f represent key points of the human body, b∈[1,18], f1 represents the nose, f2 represents the left eye, f3 represents the right eye, f4 represents the left ear, f5 represents the right ear, f6 represents the left shoulder, f7 represents the right shoulder, f8 represents the left elbow, f9 represents the right elbow, and f 10 Indicates the left wrist, f 11 Indicates the right wrist, f 12 Indicates left hip, f 13 Indicates the right hip, f 14 Indicates the left knee joint, f 15 Indicates the right knee joint, f 16 Indicates left foot, f 17 Indicates the right foot, f 18 Let (x, y, z) represent the spine, and (x, y, z) be the three-dimensional coordinates of key points in the human body.

[0014] (3) Constructing an action dataset

[0015] The 3100 joint data points J were randomly shuffled across 62 categories to form the motion dataset Y:

[0016] Y = {u1, u2, ..., u} r}

[0017] u r ={J1,J2,...,J 50}

[0018] Among them, u r Indicates the action category, r∈[1,62], u1 represents alternating flower, u2 represents front kick, u3 represents back kick, u4 represents cross step, u5 represents butterfly flower, u6 represents single arm flower, u7 represents clamshell flower, u8 represents inward slicing flower, u9 represents vertical flower, u 10 Indicates top-turning flower, u 11 Indicates a jump kick, u 12 Indicates flexion-extension step, u 13 Indicates the side step, u 14 Indicates forward progress, u 15 Indicates a hook kick, u 16 Indicates swinging the hip, u 17 Indicates the marker step, u 18To indicate a bowing gesture, u 19 Indicates taking a step, u 20 Indicates flat step, u 21 Indicates single step, u 22 Indicates a long diagonal step, u 23 Indicates three points in one step, u 24 Indicates the hook step, u 25 Indicates a minor tremor, u 26 Indicates a stepped retreat step, u 27 Indicates a step-like ascent, u 28 Indicates dragging the foot, u 29 Indicates the first basic step, u 30 Indicates a knockback step, u 31 Indicates the second basic step, u 32 Indicates a hard shoulder, u 33 Indicates soft shoulders, u 34 Indicates wrapping around the shoulder, u 35 To indicate a shrug, u 36 Indicates a broken shoulder, u 37 Indicates soft shoulder, u 38 This indicates alternating lifting and pressing of the wrist, u 39 Indicates double wrist press, u 40 Indicates a soft arm, u 41 Indicates the plucking hand, u 42 Indicates a retreat step, u 43 Indicates horse stance, u 44 Indicates cradle walk, u 45 Indicates neck displacement, u 46 This means the cat is washing its face, u 47 It indicates that the palm is wrapped around the wrist, u 48 Indicates one step, u 49 Indicates the side point of a squat, u 50 It means lifting every three steps, u 51 Indicates a side step, u 52 Indicates forward and backward movement, u 53 Indicates an evasive step, u 54 To show a salute, u 55 Indicates a domed hand, u 56 Indicates the hand that has removed its hat, u 57 Indicates a lookout, u 58 Indicates the propulsion hand, u 59 Indicates waist circumference, u 60 Indicates a rower, u 61 Indicates flat step, u 62 Indicates sync.

[0019] The action dataset Y is divided into a training set and a test set in a 7:3 ratio, with no overlap between the training set and the test set.

[0020] (4) Extract training and testing branches

[0021] Joint J, bone B, and joint motion M are determined from the joint data J in the training and test sets, respectively. J Skeletal Motion M B Four training and testing branches.

[0022] (5) Constructing a hierarchical spatiotemporal Transformer network

[0023] The hierarchical spatiotemporal Transformer network is composed of a spatial attention network, a temporal attention network, and a classifier network connected in series.

[0024] The spatial attention network consists of six interconnected spatial self-attention layers.

[0025] The temporal attention network is composed of a temporal convolutional network, a Transformer encoding layer, and a temporal tuple self-attention layer connected in parallel; the temporal tuple self-attention layer is composed of a partitioning tuple layer and a Transformer self-attention layer connected in series.

[0026] (6) Training a hierarchical spatiotemporal Transformer network

[0027] The loss function M of the hierarchical spatiotemporal Transformer network is determined by the following formula:

[0028]

[0029] Where n is the total number of input data, and n takes the value of a finite positive integer. Y is the predicted category for the i-th data point. i Let be the true category of the i-th data point.

[0030] The training data from the four training branches—joints, bones, joint motion, and bone motion—were input into the hierarchical spatiotemporal Transformer network for training. The server used for training had an RTX 2080 graphics card, an initial learning rate of 0.025, 200 training epochs, and a batch size of 8. Training continued until the network loss function M converged.

[0031] (7) Test-level spatiotemporal Transformer network

[0032] The data extracted from the test set, including data on joints, bones, joint motion, and bone motion, are input into the trained hierarchical spatiotemporal Transformer network to identify and classify human movements, outputting 62 corresponding human movement categories. r .

[0033] In step (4) of the present invention, the joint J, bone B, and joint motion M are determined from the joint data J of the training set and the test set, respectively. J Skeletal Motion M B The methods for the four training and testing branches are as follows:

[0034] J = J

[0035] B = {J} 1e -J 1s J 2e -J 2s ,...,J Te -J Ts}

[0036] M J ={J2-J1,J3-J2,...,J T -J T-1}

[0037] M B ={B2-B1,B3-B2,...,B T -B T-1}

[0038] Among them, J e For joints located far from the center of gravity of the skeleton, J s For joints located near the center of gravity of the skeleton, T represents the number of frames in the video.

[0039] In step (5) of this invention, the method for constructing the hierarchical spatial self-attention layer is as follows:

[0040] 1) The key points of the human body are divided into three different layers according to different standards: joints, components, and joint groups. The joint layer has 18 key points, represented as f1, f2, ..., f 18 The component layer has 7 components: p1, p2, ..., p7. p1 includes {f1, f2, f3, f4, f5}, p2 includes {f6, f7, f8}, ..., p7}. 18}, p3 includes {f9, f 11}, p4 includes {f8, f 10}, p5 includes {f 12 ,f 13}, p6 includes {f 15 ,f 17}, p7 includes {f 14 ,f 16 The joint group layer has four joint groups: g1, g2, g3, and g4. g1 includes {f1, f2, f3, f4, f5}, g2 includes {f6, f7, f8}, and g9 includes {f1, f2, f3, f4, f5}. 12 ,f 13 ,f 18}, g3 includes {f8, f9, f 10 ,f 11}, g4 includes {f 14 ,f 15 ,f 16 ,f 17}, thus obtaining the adjacency matrix A of the joint layer. J Adjacency matrix A of component layer P Adjacency matrix A of joint group layers G ;

[0041]

[0042]

[0043]

[0044] in, N=18 represents the number of key points on the human body;

[0045] 2) Aggregate the input features X of the hierarchical spatial self-attention layer with the adjacency matrices of the three levels respectively to obtain the feature vectors X of the joint level. J Component-level feature vector X P , Feature vector X at the joint group level G ;

[0046] X J =X

[0047]

[0048]

[0049] in, A P 'and A G 'A respectively P and A G The inverse matrix, and A respectively P and A G The normalized matrix is ​​given by T, where T is the number of frames, N is the number of human keypoints, and C is the number of channels.

[0050] 3) Obtain the attention coefficient matrix A of the spatial self-attention layer at this level using the following formula:

[0051] A = (X J ×W1)×(X J ×W2)+(X J ×W1)×(X P ×W3)+(X J ×W1)×(X G×W4)

[0052] in, This is the training parameter matrix, initialized with randomly initialized parameter values;

[0053] 4) Obtain the output feature X of the hierarchical spatial self-attention layer using the following formula. H :

[0054]

[0055] Where s is the normalization function and C is the number of channels. It is the training parameter matrix, initialized with randomly initialized parameter values.

[0056] In step (5) of this invention, the method for constructing the tuple layer is as follows:

[0057] The feature vector X is divided into X1, X2, ..., X... w The vector is a finite positive integer, where w is a component vector. k = T / w.

[0058] This invention constructs a dataset of 62 human motion categories, providing reliable data for the field of computer-aided human motion recognition. It builds a hierarchical spatiotemporal Transformer network to extract the spatial structure-dependent relationships of human motion points, the long-short time-series information of long and short time frames, and the correlations between different key points between time-series frames. This transforms the input 3D keypoint coordinates into human motion categories, enabling computer-aided human motion category recognition. This invention has advantages such as high recognition accuracy and fast recognition speed, and can be applied to the field of computer-aided human motion recognition technology. Attached Figure Description

[0059] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0060] Figure 2 This is a schematic diagram of the structure of a hierarchical spatiotemporal Transformer network.

[0061] Figure 3 yes Figure 2 A schematic diagram of the structure of a mid-space attention network.

[0062] Figure 4 yes Figure 2 A schematic diagram of the structure of a time-attention network. Specific implementation methods

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0064] Example 1

[0065] like Figure 1 As shown, the human action recognition method based on a hierarchical spatiotemporal Transformer network in this embodiment consists of the following steps:

[0066] (1) Collect action videos

[0067] 3,100 action videos were shot using cameras, comprising 62 categories with 50 videos in each category.

[0068] (2) Processing motion video

[0069] All 3100 motion videos were processed to a uniform frame rate of 300 frames per second, with a resolution of 384×288 and a frame rate of 30 frames per second. The HRNet method was used to identify each video, resulting in 3100 joint data J containing the coordinates and scores of 18 human keypoints in each frame of the video.

[0070] J = {J1, J2, ..., J} T}

[0071] J T ={f1,f2,...,f b}

[0072] f b =(x,y,z)

[0073] Where T is the number of video frames, f b Let f represent key points of the human body, b∈[1,18], f1 represents the nose, f2 represents the left eye, f3 represents the right eye, f4 represents the left ear, f5 represents the right ear, f6 represents the left shoulder, f7 represents the right shoulder, f8 represents the left elbow, f9 represents the right elbow, and f 10 Indicates the left wrist, f 11 Indicates the right wrist, f 12 Indicates left hip, f 13 Indicates the right hip, f 14 Indicates the left knee joint, f 15 Indicates the right knee joint, f 16 Indicates left foot, f 17 Indicates the right foot, f 18 Let (x, y, z) represent the spine, and (x, y, z) be the three-dimensional coordinates of key points in the human body.

[0074] (3) Constructing an action dataset

[0075] The 3100 joint data points J were randomly shuffled across 62 categories to form the motion dataset Y:

[0076] Y = {u1, u2, ..., u} r}

[0077] u r ={J1,J2,...,J 50}

[0078] Among them, u r Indicates the action category, r∈[1,62], u1 represents alternating flower, u2 represents front kick, u3 represents back kick, u4 represents cross step, u5 represents butterfly flower, u6 represents single arm flower, u7 represents clamshell flower, u8 represents inward slicing flower, u9 represents vertical flower, u 10 Indicates top-turning flower, u 11 Indicates a jump kick, u 12 Indicates flexion-extension step, u 13 Indicates the side step, u 14 Indicates forward progress, u 15 Indicates a hook kick, u 16 Indicates swinging the hip, u 17 Indicates the marker step, u 18 To indicate a bowing gesture, u 19 Indicates taking a step, u 20 Indicates flat step, u 21 Indicates single step, u 22 Indicates a long diagonal step, u 23 Indicates three points in one step, u 24 Indicates the hook step, u 25 Indicates a minor tremor, u 26 Indicates a stepped retreat step, u 27 Indicates a step-like ascent, u 28 Indicates dragging the foot, u 29 Indicates the first basic step, u 30 Indicates a knockback step, u 31 Indicates the second basic step, u 32 Indicates a hard shoulder, u 33 Indicates soft shoulders, u 34 Indicates wrapping around the shoulder, u 35 To indicate a shrug, u 36 Indicates a broken shoulder, u 37 Indicates soft shoulder, u 38 This indicates alternating lifting and pressing of the wrist, u 39 Indicates double wrist press, u 40 Indicates a soft arm, u 41 Indicates the plucking hand, u 42 Indicates a retreat step, u 43 Indicates horse stance, u 44 Indicates cradle walk, u 45 Indicates neck displacement, u 46 This means the cat is washing its face, u 47 It indicates that the palm is wrapped around the wrist, u 48 Indicates one step, u 49 Indicates the side point of a squat, u 50It means lifting every three steps, u 51 Indicates a side step, u 52 Indicates forward and backward movement, u 53 Indicates an evasive step, u 54 To show a salute, u 55 Indicates a domed hand, u 56 Indicates the hand that has removed its hat, u 57 Indicates a lookout, u 58 Indicates the propulsion hand, u 59 Indicates waist circumference, u 60 Indicates a rower, u 61 Indicates flat step, u 62 Indicates sync.

[0079] The action dataset Y is divided into a training set and a test set in a 7:3 ratio, with no overlap between the training set and the test set.

[0080] (4) Extract training and testing branches

[0081] Joint J, bone B, and joint motion M are determined from the joint data J in the training and test sets, respectively. J Skeletal Motion M B Four training and testing branches.

[0082] In this embodiment, joint J, bone B, and joint motion M are determined from the joint data J in the training set and test set, respectively. J Skeletal Motion M B The methods for the four training and testing branches are as follows:

[0083] J = J

[0084] B = {J} 1e -J 1s J 2e -J 2s ,...,J Te -J Ts}

[0085] M J ={J2-J1,J3-J2,...,J T -J T-1}

[0086] M B ={B2-B1,B3-B2,...,B T -B T-1}

[0087] Among them, J e For joints located far from the center of gravity of the skeleton, J s For joints located near the center of gravity of the skeleton, T represents the number of frames in the video.

[0088] (5) Constructing a hierarchical spatiotemporal Transformer network

[0089] Figure 2 A schematic diagram of the hierarchical spatiotemporal Transformer network is given. Figure 2 In this embodiment, the hierarchical spatiotemporal Transformer network is composed of a spatial attention network, a temporal attention network, and a classifier network connected in series.

[0090] Figure 3 Given Figure 2 A schematic diagram of the structure of a mid-space attention network. Figure 3 In this embodiment, the spatial attention network consists of six layers of spatial self-attention connected in series.

[0091] The method for constructing the hierarchical spatial self-attention layer in this embodiment is as follows:

[0092] 1) The key points of the human body are divided into three different layers according to different standards: joints, components, and joint groups. The joint layer has 18 key points, represented as f1, f2, ..., f 18 The component layer has 7 components: p1, p2, ..., p7. p1 includes {f1, f2, f3, f4, f5}, p2 includes {f6, f7, f8}, ..., p7}. 18}, p3 includes {f9, f 11}, p4 includes {f8, f 10}, p5 includes {f 12 ,f 13}, p6 includes {f 15 ,f 17}, p7 includes {f 14 ,f 16 The joint group layer has four joint groups: g1, g2, g3, and g4. g1 includes {f1, f2, f3, f4, f5}, g2 includes {f6, f7, f8}, and g9 includes {f1, f2, f3, f4, f5}. 12 ,f 13 ,f 18}, g3 includes {f8, f9, f 10 ,f 11}, g4 includes {f 14 ,f 15 ,f 16 ,f 17}, thus obtaining the adjacency matrix A of the joint layer. J Adjacency matrix A of component layer P Adjacency matrix A of joint group layers G .

[0093]

[0094]

[0095]

[0096] in, N=18 represents the number of key points on the human body.

[0097] 2) Aggregate the input features X of the hierarchical spatial self-attention layer with the adjacency matrices of the three levels respectively to obtain the feature vectors X of the joint level. J Component-level feature vector X P , Feature vector X at the joint group level G ;

[0098] X J =X

[0099]

[0100]

[0101] in, A P 'and A G 'A respectively P and A G The inverse matrix, and A respectively P and A G The normalized matrix is ​​given by T, where T is the number of frames, N is the number of human keypoints, and C is the number of channels.

[0102] 3) Obtain the attention coefficient matrix A of the spatial self-attention layer at this level using the following formula:

[0103] A = (X J ×W1)×(X J ×W2)+(X J ×W1)×(X P ×W3)+(X J ×W1)×(X G ×W4)

[0104] in, It is the training parameter matrix, initialized with randomly initialized parameter values.

[0105] 4) Obtain the output feature X of the hierarchical spatial self-attention layer using the following formula. H :

[0106]

[0107] Where s is the normalization function and C is the number of channels. It is the training parameter matrix, initialized with randomly initialized parameter values.

[0108] Figure 4 Given Figure 2 A schematic diagram of the structure of a time-attention network. Figure 4 In this embodiment, the temporal attention network consists of a temporal convolutional network, a Transformer encoding layer, and a temporal tuple self-attention layer connected in parallel. The temporal tuple self-attention layer is composed of a partitioning tuple layer and a Transformer self-attention layer connected in series.

[0109] The method for constructing the tuple layer in this embodiment is as follows:

[0110] The feature vector X is divided into X1, X2, ..., X... w The vector is a finite positive integer, where w is a component vector. k = T / w.

[0111] The classifier network in this embodiment is a prior art technology. It consists of a global average pooling layer, a linear layer, and a normalized exponential layer connected in series. The output of the global average pooling layer is connected to the linear layer, and the output of the linear layer is connected to the normalized exponential layer.

[0112] This embodiment constructs a hierarchical spatiotemporal Transformer network, extracts high-order dependencies such as components and joint groups that are independent of keypoint coordinates in the human motion spatial structure, long-term and short-term temporal information and the correlation between different keypoints between temporal frames, and converts the input three-dimensional keypoint coordinates into human motion categories for recognition, thus realizing computer recognition of human motion categories.

[0113] (6) Training a hierarchical spatiotemporal Transformer network

[0114] The loss function M of the hierarchical spatiotemporal Transformer network is determined by the following formula:

[0115]

[0116] Where n is the total number of input data, and n takes the value of a finite positive integer. Y is the predicted category for the i-th data point. i Let be the true category of the i-th data point.

[0117] The training data from the four training branches—joints, bones, joint motion, and bone motion—were input into the hierarchical spatiotemporal Transformer network for training. The server used for training had an RTX 2080 graphics card, an initial learning rate of 0.025, 200 training epochs, and a batch size of 8. Training continued until the network loss function M converged.

[0118] (7) Test-level spatiotemporal Transformer network

[0119] The data extracted from the test set, including data on joints, bones, joint motion, and bone motion, are input into the trained hierarchical spatiotemporal Transformer network to identify and classify human movements, outputting 62 corresponding human movement categories. r .

[0120] A method for human motion recognition based on a hierarchical spatiotemporal Transformer network was developed.

[0121] To verify the beneficial effects of this invention, the inventors conducted comparative experiments using the human action recognition method based on a hierarchical spatiotemporal Transformer network as described in Embodiment 1 of this invention, along with the human action recognition methods of spatiotemporal graph convolutional networks (hereinafter referred to as ST-GCN), two-stream adaptive graph convolutional networks (hereinafter referred to as 2S-AGCN), multi-stream adaptive graph convolutional networks (hereinafter referred to as MS-AAGCN), spatiotemporal tuple Transformer networks (hereinafter referred to as STTFormer), and decoupled spatiotemporal attention networks (hereinafter referred to as DSTA-Net). The experimental details are as follows:

[0122] The evaluation metrics are defined as follows: Accuracy B1 is the accuracy rate of matching the first-ranked category among the output human action categories with the actual action category, and Accuracy B5 is the accuracy rate of including the actual action category among the top five categories among the output human action categories. The magnitudes of Accuracy B1 and Accuracy B5 are used as evaluation metrics for human action recognition accuracy. The larger the Accuracy B1 and Accuracy B5, the higher the accuracy of human action recognition.

[0123] The experimental results of the four test branches of the method in Example 1, namely joint, bone, joint movement, and bone movement, are shown in Table 1.

[0124] Table 1. Human motion recognition rates B1 and B5 of the method in Example 1

[0125] Evaluation indicators joint skeleton Joint movement Skeletal movement Example 1 Accuracy B1 (%) 68.01 70.42 65.59 68.33 78.46 Accuracy rate B5 (%) 90.19 92.93 86.66 89.71 95.98

[0126] As shown in Table 1, the accuracy rates B1 and B5 of the method in Example 1 are 78.46% and 95.98%, respectively.

[0127] The results of the comparative experiments are shown in Table 2.

[0128] Table 2. Experimental results of the method in Example 1 and the comparative experimental method.

[0129]

[0130] As shown in Table 2, the accuracy B1 of the method in Example 1 is 78.46%, which is higher than that of the ST-GCN, 2S-AGCN, MS-AAGCN, STTFormer, and DSTA-Net methods; the accuracy B5 of the method in Example 1 is 95.98%, which is also higher than that of the ST-GCN, 2S-AGCN, MS-AAGCN, STTFormer, and DSTA-Net methods. Experiments demonstrate that the method of the present invention has a higher accuracy in human motion recognition.

Claims

1. A method for human action recognition based on hierarchical spatiotemporal Transformer networks, characterized in that... It consists of the following steps: (1) Collect action videos 3,100 action videos were shot using cameras, comprising 62 categories, with 50 videos in each category; (2) Processing motion video All 3100 motion videos were processed to a uniform frame rate of 300 frames per second (384×288 resolution). The HRNet method was then used to identify each video, resulting in 3100 joint data points containing the coordinates and scores of 18 human keypoints from all frames in each video. : Where T is the number of video frames. Indicates key points of the human body, , It refers to the nose. Indicates the left eye. Indicates the right eye. Indicates the left ear. Indicates the right ear. Indicates left shoulder. Indicates the right shoulder. Indicates the left elbow. Indicates the right elbow. Indicates the left wrist. Indicates the right wrist. Indicates the left hip. Indicates the right buttock. Indicates the left knee joint. Indicates the right knee joint. Indicates the left foot. Indicates the right foot. It refers to the spine. Three-dimensional coordinates of key points on the human body; (3) Constructing an action dataset Data from 3100 joints The 62 categories are randomly shuffled to form the action dataset Y: in, Indicates the action category, , Indicates alternating flowers, It means a front kick. It means lifting the leg back. Indicates a crossover step. It represents a butterfly flower. Indicates a single-armed flower. It represents a oyster shell flower. Indicates an inscribed flower pattern. Indicates vertical flower pattern, Indicates a top-turning pattern. Indicates a jump kick. Indicates a flexion-extension step. Indicates a side step. Indicates progress. Indicates a hook kick. Indicates swinging one's hips. Indicates the marked step, To indicate a bowing gesture Indicates taking a step. Indicates flat step, Indicates single step, Indicates a long, oblique step. Indicates three points in one step. Indicates a hook step. Indicates a minor tremor. This indicates a stepped retreat. Indicates a step-by-step ascent. It means to drag one's steps. This indicates the first basic step. Indicates a knockback step. Indicates the second basic step. Indicates a stiff shoulder. Indicates soft shoulders, Indicates wrapping around the shoulder. To shrug, Indicates a broken shoulder. Indicates soft shoulders. This indicates alternating lifting and pressing of the wrist. This indicates double wrist press. Indicates soft arm, Indicates the hand that plucks feathers. Indicates a retreat step. Indicates horse stance. Indicates cradle walk, Indicates a displaced neck position. This means the cat is washing its face. It means the palm is wrapped around the wrist. Indicates one step. Indicates the side point of a squat. It means lifting every three steps. Indicates a side step. Indicates moving forward and backward. Indicates an evasive step. A gesture of salute. Indicates a dome-shaped hand. Indicates the hand that has removed its hat. Indicates a lookout. Indicates the driving hand, Indicates waist circumference. Indicates rower, Indicates flat step, Indicates sync; The action dataset Y is divided into a training set and a test set in a 7:3 ratio, with no overlap between the training set and the test set. (4) Extract training and test branches Joint data from training and test sets The training and testing branches are defined separately for joints, bones, joint movement, and skeletal movement. (5) Constructing a hierarchical spatiotemporal Transformer network The hierarchical spatiotemporal Transformer network is composed of a spatial attention network, a temporal attention network, and a classifier network connected in series. The spatial attention network is composed of six layers of spatial self-attention connected in series. The temporal attention network is composed of a temporal convolutional network, a Transformer encoding layer, and a temporal tuple self-attention layer connected in parallel; the temporal tuple self-attention layer is composed of a partitioning tuple layer and a Transformer self-attention layer connected in series. (6) Training a hierarchical spatiotemporal Transformer network The loss function M of the hierarchical spatiotemporal Transformer network is determined by the following formula: Where n is the total number of input data, and n takes the value of a finite positive integer. For the predicted category of the i-th data point, Let i be the true category of the i-th data point; The four training branches of the training set—joints, bones, joint motion, and bone motion—were input into the hierarchical spatiotemporal Transformer network for training. The server used for training had an RTX 2080 graphics card, an initial learning rate of 0.025, 200 training epochs, and a batch size of 8. Training continued until the network loss function M converged. (7) Test layer spatiotemporal Transformer network The data from the four test branches extracted from the test set—joints, bones, joint motion, and bone motion—are respectively input into the trained hierarchical spatiotemporal Transformer network to identify and classify human movements, outputting 62 corresponding human movement categories. .

2. The human action recognition method based on hierarchical spatiotemporal Transformer network according to claim 1, characterized in that: In step (5), the method for constructing the hierarchical spatial self-attention layer is as follows: 1) The key points of the human body are divided into three different layers according to different standards: joints, components, and joint groups. The joint layer has 18 key points, which are represented as follows: ; Component layer has 7 parts include , include , include , include , include , include , include Joint group layers have 4 joint groups include , include , include , include The adjacency matrix of the joint layer is obtained. Adjacency matrix of component layer Adjacency matrix of joint group layers ; in, , , N=18 represents the number of key points on the human body; 2) Input features of the hierarchical spatial self-attention layer Aggregate the features with the adjacency matrices of the three levels respectively to obtain the feature vectors of the joint level. Feature vectors at the component level Feature vectors at the joint group level ; in, , , , , and They are respectively and The inverse matrix, and They are respectively and The normalized matrix, where T is the number of frames, N is the number of human keypoints, and C is the number of channels; 3) Obtain the attention coefficient matrix of the spatial self-attention layer at this level using the following formula. : in, , , , This is the training parameter matrix, initialized with randomly initialized parameter values; 4) Obtain the output features of the hierarchical spatial self-attention layer using the following formula. : Where s is the normalization function and C is the number of channels. It is the training parameter matrix, initialized with randomly initialized parameter values.

Images