A human gait recognition method based on dynamic spatio-temporal graph convolutional neural network

Abstract

The application discloses a human gait recognition method based on a dynamic space-time graph convolutional neural network, and comprises the following steps: performing human posture estimation on a CASIA-B dataset to generate a heat map; inputting the heat map into an HM-C module to generate human skeleton sequence information; extracting joint, skeleton and speed features; inputting into a local attention module to obtain a feature vector containing weight information of different body parts in the whole action sequence; inputting into a main network dynamic GCN module to generate a spatial feature map; inputting the human skeleton sequence information and the spatial feature into a dynamic TCN module to generate a rich time feature map; inputting the time feature map into a joint skeleton dynamic fusion module to fuse the motion modes of the joint level and the skeleton level in different time ranges, and generating more rich and more discriminative joint-skeleton motion features. The application solves the problem that the baseline GCN method is seriously dependent on manually defined skeleton topology.

Description

Technical Field

[0001] This invention relates to a skeleton-based human gait recognition method, belonging to the interdisciplinary fields of computer vision, image processing, pose recognition, and gait recognition. Background Technology

[0002] Gait recognition is a promising video-based biometric identification method. Compared to other unique biometrics such as faces, fingerprints, and irises, gait offers significant advantages for long-distance identification and does not require subject cooperation or intrusion. Therefore, it opens up enormous potential for applications such as social security, access control, and forensic identification. However, gait can be sensitive to surface type, clothing, carried items, and clutter or occlusion in the scene. These represent challenges in processing gait recognition tasks to learn unique and invariant features from human gait.

[0003] In recent years, most gait recognition methods have used human contours to extract gait features. However, in cluttered and rapidly changing real-world environments, contour images may lose fine-grained spatial information and be affected by external factors such as occlusion. Skeleton-based gait recognition, on the other hand, focuses on skeletal structure and can capture the depth and spatial features of movement, making it more robust to interference from factors such as lighting, shadows, occlusion, and clothing. With the development of deep learning-based pose estimation, a new era of skeleton-based and model-based gait recognition has begun.

[0004] A method based on graph convolutional networks (GCNs) has emerged. Yan et al. first proposed ST-GCN to model skeletal motion patterns using spatiotemporal graphs. Two-stream adaptive graph convolutional network (2sAGCN) introduces an adaptive adjacency matrix and pre-computed skeletal information as second-order inputs. SGN has conducted a more in-depth analysis of these second-order features.

[0005] However, while Graph Convolutional Networks (GCNs) have shone brightly in the new era of skeleton-based and model-based gait recognition, demonstrating good effectiveness, these methods still heavily rely on manually defined skeleton topologies, requiring arduous customization for different datasets. Dynamic GCNs, extending from the baseline models of traditional GCNs, are better suited for modeling complex joint dependencies because their execution relies on intrinsic cooperation between joints and consists entirely of learnable coefficient matrices rather than a prescribed graphical structure. Furthermore, previous GCN-based methods mostly utilized temporal convolutions with fixed receptive fields to model joint-level motion within a specific time range, neglecting the benefits of modeling multi-level (i.e., joint-level + skeleton-level) motion patterns within a dynamic temporal receptive field. Therefore, dynamic spatiotemporal graph convolutional network models are urgently needed to address the limitations of existing work. Summary of the Invention

[0006] The purpose of this invention is to provide a human gait recognition method based on dynamic spatiotemporal graph convolutional neural networks, in order to solve the problem that existing baseline GCN methods heavily rely on manually defined skeleton topologies.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for human gait recognition based on dynamic spatiotemporal graph convolutional neural networks includes the following steps:

[0009] Step 1: Input the CASIA-B dataset into the human pose extraction module, use the HRNet model to estimate human pose, and generate a heatmap.

[0010] Step 2: Input the heat map obtained in Step 1 into the HM-C module. HM-C uses the distribution-aware coordinate representation method to re-encode the key point coordinates, and finally generates a vector sequence containing key point location information, which is the human skeleton sequence information.

[0011] Step 3: Extract joint, bone, and velocity features from the human skeleton sequence information obtained in Step 2;

[0012] Step 4: Input the joint, bone, and velocity features obtained in Step 3 into the local attention module to obtain a feature vector containing the weight information of different body parts in the entire action sequence.

[0013] Step 5: Input the human skeleton sequence information obtained in Step 2 into the backbone network dynamic GCN module to model in the spatial dimension, fuse spatial features, and generate a spatial feature map.

[0014] Step 6: Input the human skeleton sequence information obtained in Step 2 and the spatial features obtained in Step 6 into the dynamic TCN module to aggregate temporal features, learn the temporal patterns between adjacent frames, and combine the spatially captured information with temporal information. By comprehensively considering the spatial and temporal information of the action, a richer temporal feature map is generated.

[0015] Step 7: Input the temporal feature map obtained in Step 7 into the joint-skeleton dynamic fusion module to fuse the joint-level and skeleton-level motion patterns in different time ranges, generating richer and more discriminative joint-skeleton motion features.

[0016] Step 1 includes:

[0017] Step 11: Input the original images from the CASIA-B dataset into the HRNet model to generate high-resolution feature maps in the high-resolution sub-network. The high-resolution sub-network receives the high-resolution feature maps from the previous stage as input and generates the high-resolution feature maps for the current stage.

[0018] Step 12: The low-resolution sub-network of the HRNet model receives the low-resolution feature map from the previous stage as input and generates the low-resolution feature map for the current stage. High and low resolution sub-networks are added step by step to form more stages; each stage contains a high-resolution sub-network and a low-resolution sub-network.

[0019] Step 13: Connect all the stages obtained in step 12 in parallel and process the input image simultaneously; the high-resolution feature map and low-resolution feature map of each stage are sent to the next stage.

[0020] Step 14: Fuse the high-resolution feature maps and low-resolution feature maps obtained in the same stage in Step 13 together, and integrate the feature maps obtained in different stages in Step 13 using multi-scale fusion technology.

[0021] Step 15: Input the fused feature map obtained in step 14 into the pose estimation module. After passing through multiple convolutional and pooling layers, extract features and reduce the size of the feature map to output a heatmap of human pose.

[0022] Step 2 includes:

[0023] Step 21: Modulate the heatmap distribution of the heatmap obtained in Step 1, and use a Gaussian kernel K with the same variation as the training data to smooth the influence of multiple peaks in the heatmap R. The expression is as follows:

[0024] R = K * R

[0025] Here, * represents the convolution operation. To preserve the size of the original heatmap, R is ultimately scaled to 0 so that its maximum activation equals the maximum activation of R, as shown in the following expression:

[0026]

[0027] The max() and min() functions return the maximum and minimum values ​​of the input matrix, respectively.

[0028] Step 22: Using sub-pixel precision Taylor expansion for distribution-aware joint localization, assuming the predicted heatmap follows a 2D Gaussian distribution, the same as the ground-based heatmap, the predicted heatmap expression is as follows:

[0029]

[0030] Where w is the pixel position in the predicted heatmap, ū is the Gaussian mean (center) corresponding to the joint position to be estimated; the covariance ∑ is a diagonal matrix, the same matrix used in coordinate encoding, and its expression is as follows:

[0031]

[0032] Where σ is the standard deviation in both directions;

[0033] Step 23: Given the heatmap R predicted by the trained model, first determine the coordinates of the maximum (m) and second maximum (s) activations. The joint positions will be predicted as follows:

[0034]

[0035] Here, || defines the size of the vector, which means that the prediction is the maximum activation, where 0.25 pixels (i.e., sub-pixels) are offset to the second maximum activation in the heatmap space;

[0036] Step 24: Convert the predicted joint positions into coordinates in the original image space. The final coordinate prediction in the original image is calculated as follows:

[0037] P r * =δP r

[0038] Where δ is the resolution reduction ratio;

[0039] Step 25: Draw the connections between key points as skeletal information, and finally generate a vector sequence containing key point position information, which is the human skeletal sequence information.

[0040] Step 3 includes:

[0041] Step 31: Obtain the original three-dimensional coordinate set X = {x∈R} from the human skeleton sequence information obtained in Step 2. C ×T×V}, where C, T, and V represent coordinates, frames, and joints, respectively. Then, the relative position of each joint at each time step is calculated, resulting in a set of relative positions R = {γ}. i |i=1,2…,V}, where for each joint i, its relative position γ i It is obtained by subtracting the coordinates of the central joint from its coordinates, and the two sets are concatenated into a single sequence to obtain the relative positions of the joints;

[0042] Step 32: Calculate the velocity of each joint at each time step from the human skeletal sequence information obtained in Step 2 to obtain two sets of motion velocities, represented as follows:

[0043] S={st |t=1,2…,T}

[0044] S″={s″ t |t=1,2…,T}

[0045] Among them, S ‘ S represents the displacement of each joint between the current time step and the next time step, S' represents the displacement of each joint between the current time step and the previous time step, t represents a certain moment in the time series, and T represents the total length of the time series. Finally, by calculating the S' of each joint... ‘ The speed of motion is obtained by cascading with "S".

[0046] Step 33: Calculate the Euclidean distance between joint i and its parent joint from the human skeleton sequence information obtained in Step 2 to obtain the bone length; calculate the angle between joint i, its parent joint, and its child joint to obtain the bone angle, as shown in the following expression:

[0047]

[0048] Among them, l i θ represents bone length. i The x, y, and z axes represent the x-axis, y-axis, and z-axis in a 3D coordinate system, respectively, and the final bone length l is obtained. i The set and the skeleton angle θ i The set of features represents the overall skeletal characteristics.

[0049] Step 4 includes:

[0050] Step 41: Select the corresponding joints, obtain six individual body parts from the bone features obtained in Step 3, and then connect the features of all parts together.

[0051] Step 42: Perform average pooling and convolution operations on the partial feature sequences obtained in step 41 in the time dimension;

[0052] Step 43: Input the vector output from step 42 into a fully connected layer with a BatchNorm layer and a ReLU function;

[0053] Step 44: The vector output from step 43 is processed through six fully connected layers to calculate the attention matrix, and the Softmax function is used to determine the most important body parts.

[0054] Step 45: Connect the features of the six parts obtained in Step 44 into an overall skeleton representation with different attention weights. The formula for the local attention module is:

[0055]

[0056] Among them, f in This represents the input feature map, where p represents the index of the body part. Learnable parameters shared by all components. The global context feature map is specific to each part used to compute the final attention weights, obtained through average pooling over the entire time series.

[0057] Step 5 includes:

[0058] Step 51: Initialize the coefficient matrix A randomly using a normal distribution. The coefficient matrix A consists of N different components, and N can be set to any value to complete the design of multiple sets in the dynamic graph convolution module.

[0059] Step 52: The skeleton sequence information with C channels obtained in Step 2 is used as the input to the spatial convolution submodule. First, it is processed using a 1×1 convolution to generate N feature groups, each feature group having… One channel;

[0060] Step 53: Given the input feature F, first use time pooling to eliminate dimension T; then apply two separate 1×1 convolutions to F to obtain F1 and F2 respectively;

[0061] Step 54: Use the Softmax activation function to obtain the dynamic term from F1 and F2 obtained in step 53. Tanh activation function to obtain dynamic terms with a specific channel. and The expression is as follows:

[0062]

[0063] Step 55, take the result from step 54 and With static items The weighted sum is used as the final coefficient matrix A. i A i The expression is as follows:

[0064]

[0065] Here, α and β are two learnable parameters;

[0066] Step 56, take A obtained in step 55 i The N feature groups obtained in step 52 are modeled independently, and then the N feature groups are cascaded along the channel dimension;

[0067] Step 57: The concatenated results of the N feature groups obtained in step 56 are processed by a 1×1 convolution to generate a spatial feature map.

[0068] Step 6 includes:

[0069] Step 61: Input the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 5 into the bottleneck module in the TCN module, and generate M feature groups through a 1×1 convolution. The multiple TCNs are composed of multiple branches with different receptive fields.

[0070] Step 62: Split the sequence of length T obtained in step 2 into M substrings of equal length, then randomly select a frame from each substring and concatenate them into a new subsequence;

[0071] Step 63: Perform convolution and pooling operations along the channel dimension on the M feature groups obtained in step 61 and the M sample sequences obtained in step 62 respectively, and then concatenate the results.

[0072] Step 7 includes:

[0073] Step 71: Further model the parallel execution time of the results obtained in Step 6 at the joint and skeleton levels. First, model the V joint-level features {X1, ..., X...} V |X i ∈R C×T Perform average pooling to obtain skeleton-level features S;

[0074] Step 72, combine the skeleton-level features S and V joint features X obtained in step 71. i Perform parallel processing to obtain S * and X i * ;

[0075] Step 73, take the S obtained in step 72 * Merge into X i * middle;

[0076] Step 74, each TCN instance contains a learning parameter. After adaptive joint skeleton fusion, the features of joint i are:

[0077] Step 75: Perform a 1×1 convolution operation on the adaptive joint skeleton fusion feature sequence obtained in step 74 to obtain the output of the TCN module.

[0078] Beneficial effects: Compared with the prior art, the present invention has the following technical effects:

[0079] This invention first obtains a vector sequence containing keypoint location information, i.e., human skeleton sequence information, through HRNet and HW-C modules. Then, a local attention module is used to discover the importance of different body parts in the entire action sequence. Next, a dynamic GCN module is used to dynamically model the human pose of each frame. Then, a TCN network is used to learn temporal features. Next, a joint and skeleton dynamic fusion module is used to explicitly model and dynamically fuse joint-level and skeleton-level features. Temporal modeling is performed in parallel at the joint and skeleton levels. Finally, the final prediction result is obtained through pooling layers and fully connected layers.

[0080] Specifically:

[0081] (1) The present invention uses HRNet and HW-C modules to extract human skeleton sequence information, which is beneficial to obtain joint, bone and velocity features in human figure outline.

[0082] (2) This invention introduces an attention mechanism to highlight the importance of different body parts in the entire action sequence.

[0083] (3) The present invention performs spatial and temporal modeling through dynamic GCN module and TCN module to solve the problem that the baseline GCN method proposed in the background art above relies heavily on manually defined skeleton topology.

[0084] (4) The joint-skeleton dynamic fusion module integrates joint-level and skeleton-level motion patterns in different time ranges to generate richer and more discriminative joint-skeleton motion features. Attached Figure Description

[0085] Figure 1 This is a flowchart of the overall method of the present invention;

[0086] Figure 2 This is a diagram of the dynamic GCN module structure;

[0087] Figure 3 This is a diagram of the dynamic TCN module structure;

[0088] Figure 4 This is a diagram of the local attention module structure. Detailed Implementation

[0089] The invention will now be further explained with reference to the accompanying drawings.

[0090] A method for human gait recognition based on dynamic spatiotemporal graph convolutional neural networks includes the following steps:

[0091] Step 1: Input the CASIA-B dataset into the human pose extraction module, use the HRNet model to estimate human pose, and generate a heatmap; specifically including:

[0092] Step 11: Input the original images from the CASIA-B dataset into the HRNet model to generate high-resolution feature maps in the high-resolution sub-network. The high-resolution sub-network receives the high-resolution feature maps from the previous stage as input and generates the high-resolution feature maps for the current stage.

[0093] Step 12: The low-resolution sub-network of the HRNet model receives the low-resolution feature map from the previous stage as input and generates the low-resolution feature map for the current stage. High and low resolution sub-networks are added step by step to form more stages; each stage contains a high-resolution sub-network and a low-resolution sub-network.

[0094] Step 13: Connect all the stages obtained in step 12 in parallel and process the input image simultaneously; the high-resolution feature map and low-resolution feature map of each stage are sent to the next stage.

[0095] Step 14: Fuse the high-resolution feature maps and low-resolution feature maps obtained in the same stage in Step 13 together, and integrate the feature maps obtained in different stages in Step 13 using multi-scale fusion technology to produce more accurate results.

[0096] Step 15: Input the fused feature map obtained in step 14 into the pose estimation module. After passing through multiple convolutional and pooling layers, extract features and reduce the size of the feature map to output a heatmap of human pose.

[0097] Step 2: Input the heatmap obtained in Step 1 into the HM-C module. HM-C uses a distribution-aware coordinate representation method to re-encode the keypoint coordinates, ultimately generating a vector sequence containing keypoint location information, which is the human skeleton sequence information; specifically including:

[0098] Step 21: Modulate the heatmap distribution of the heatmap obtained in Step 1, and use a Gaussian kernel K with the same variation as the training data to smooth the influence of multiple peaks in the heatmap R. The expression is as follows:

[0099] R = K * R

[0100] Here, * represents the convolution operation. To preserve the size of the original heatmap, R is ultimately scaled to 0 so that its maximum activation equals the maximum activation of R, as shown in the following expression:

[0101]

[0102] The max() and min() functions return the maximum and minimum values ​​of the input matrix, respectively.

[0103] Step 22: Using sub-pixel precision Taylor expansion for distribution-aware joint localization, assuming the predicted heatmap follows a 2D Gaussian distribution, the same as the ground-based heatmap, the predicted heatmap expression is as follows:

[0104]

[0105] Where w is the pixel position in the predicted heatmap, ū is the Gaussian mean (center) corresponding to the joint position to be estimated; the covariance ∑ is a diagonal matrix, the same matrix used in coordinate encoding, and its expression is as follows:

[0106]

[0107] Where σ is the standard deviation in both directions;

[0108] Step 23: Given the heatmap R predicted by the trained model, first determine the coordinates of the maximum (m) and second maximum (s) activations. The joint positions will be predicted as follows:

[0109]

[0110] Here, || defines the size of the vector, which means that the prediction is the maximum activation, where 0.25 pixels (i.e., sub-pixels) are offset to the second maximum activation in the heatmap space;

[0111] Step 24: Convert the predicted joint positions into coordinates in the original image space. The final coordinate prediction in the original image is calculated as follows:

[0112] P r * =δP r

[0113] Where δ is the resolution reduction ratio;

[0114] Step 25: Draw the connections between key points as skeletal information, and finally generate a vector sequence containing key point position information, which is the human skeletal sequence information.

[0115] Step 3: Extract joint, bone, and velocity features from the human skeletal sequence information obtained in Step 2; specifically including:

[0116] Step 31: Obtain the original three-dimensional coordinate set X = {x∈R} from the human skeleton sequence information obtained in Step 2. C ×T×V}, where C, T, and V represent coordinates, frames, and joints, respectively. Then, the relative position of each joint at each time step is calculated, resulting in a set of relative positions R = {γ}. i |i=1,2…,V}, where for each joint i, its relative position γ iIt is obtained by subtracting the coordinates of the central joint from its coordinates, and the two sets are concatenated into a single sequence to obtain the relative positions of the joints;

[0117] Step 32: Calculate the velocity of each joint at each time step from the human skeletal sequence information obtained in Step 2 to obtain two sets of motion velocities, represented as follows:

[0118] S={s t |t=1,2…,T}

[0119] S″={s″ t |t=1,2…,T}

[0120] Among them, S ‘ S represents the displacement of each joint between the current time step and the next time step, S' represents the displacement of each joint between the current time step and the previous time step, t represents a certain moment in the time series, and T represents the total length of the time series. Finally, by calculating the S' of each joint... ‘ The speed of motion is obtained by cascading with "S".

[0121] Step 33: Calculate the Euclidean distance between joint i and its parent joint from the human skeleton sequence information obtained in Step 2 to obtain the bone length; calculate the angle between joint i, its parent joint, and its child joint to obtain the bone angle, as shown in the following expression:

[0122]

[0123] Among them, l i θ represents bone length. i The x, y, and z axes represent the x-axis, y-axis, and z-axis in a 3D coordinate system, respectively, and the final bone length l is obtained. i The set and the skeleton angle θ i The set of features represents the overall skeletal characteristics.

[0124] Step 4: Input the joint, bone, and velocity features obtained in Step 3 into the local attention module to obtain a feature vector containing weight information of different body parts in the entire action sequence; specifically including:

[0125] Step 41: Select the corresponding joints, obtain six individual body parts from the bone features obtained in Step 3, and then connect the features of all parts together.

[0126] Step 42: Perform average pooling and convolution operations on the partial feature sequences obtained in step 41 in the time dimension;

[0127] Step 43: Input the vector output from step 42 into a fully connected layer with a BatchNorm layer and a ReLU function;

[0128] Step 44: The vector output from step 43 is processed through six fully connected layers to calculate the attention matrix, and the Softmax function is used to determine the most important body parts.

[0129] Step 45: Connect the features of the six parts obtained in Step 44 into an overall skeleton representation with different attention weights. The formula for the local attention module is:

[0130]

[0131] Among them, f in This represents the input feature map, where p represents the index of the body part. Learnable parameters shared by all components. The global context feature map is specific to each part used to compute the final attention weights, obtained through average pooling over the entire time series.

[0132] Step 5: Input the human skeletal sequence information obtained in Step 2 into the backbone network dynamic GCN module to perform spatial modeling, fuse spatial features, and generate a spatial feature map; specifically including:

[0133] Step 51: Initialize the coefficient matrix A randomly using a normal distribution. The coefficient matrix A consists of N different components, and N can be set to any value to complete the design of multiple sets in the dynamic graph convolution module.

[0134] Step 52: The skeleton sequence information with C channels obtained in Step 2 is used as the input to the spatial convolution submodule. First, it is processed using a 1×1 convolution to generate N feature groups, each feature group having… One channel;

[0135] Step 53: Given the input feature F, first use time pooling to eliminate dimension T; then apply two separate 1×1 convolutions to F to obtain F1 and F2 respectively;

[0136] Step 54: Use the Softmax activation function to obtain the dynamic term from F1 and F2 obtained in step 53. Tanh activation function to obtain dynamic terms with a specific channel. and The expression is as follows:

[0137]

[0138] Step 55, take the result from step 54 and With static items The weighted sum is used as the final coefficient matrix A. i Ai The expression is as follows:

[0139]

[0140] Here, α and β are two learnable parameters;

[0141] Step 56, take A obtained in step 55 i The N feature groups obtained in step 52 are modeled independently, and then the N feature groups are cascaded along the channel dimension;

[0142] Step 57: The concatenated results of the N feature groups obtained in step 56 are processed by a 1×1 convolution to generate a spatial feature map.

[0143] Step 6 involves inputting the human skeleton sequence information obtained in Step 2 and the spatial features obtained in Step 6 into the dynamic TCN module to aggregate temporal features, learn the temporal patterns between adjacent frames, and combine spatially captured information with temporal information. This comprehensive consideration of both spatial and temporal information of the action generates a richer temporal feature map. Specifically, this includes:

[0144] Step 61: Input the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 5 into the bottleneck module in the TCN module, and generate M feature groups through a 1×1 convolution. The multiple TCNs are composed of multiple branches with different receptive fields.

[0145] Step 62: Split the sequence of length T obtained in step 2 into M substrings of equal length, then randomly select a frame from each substring and concatenate them into a new subsequence; this uniform sampling method can generate more diverse samples while maintaining the integrity of each sample.

[0146] Step 63: Perform convolution and pooling operations along the channel dimension on the M feature groups obtained in step 61 and the M sample sequences obtained in step 62 respectively, and then concatenate the results.

[0147] Step 7: Input the temporal feature map obtained in Step 7 into the joint-skeleton dynamic fusion module to fuse joint-level and skeleton-level motion patterns across different time ranges, generating richer and more discriminative joint-skeleton motion features; specifically including:

[0148] Step 71: Further model the parallel execution time of the results obtained in Step 6 at the joint and skeleton levels. First, model the V joint-level features {X1, ..., X...} V |X i ∈R C×T Perform average pooling to obtain skeleton-level features S;

[0149] Step 72, combine the skeleton-level features S and V joint features X obtained in step 71. i Perform parallel processing to obtain S * and X i * ;

[0150] Step 73, take the S obtained in step 72 * Merge into X i * middle;

[0151] Step 74, each TCN instance contains a learning parameter. After adaptive joint skeleton fusion, the features of joint i are:

[0152] Step 75: Perform a 1×1 convolution operation on the adaptive joint skeleton fusion feature sequence obtained in Step 74 to obtain the output of the TCN module.

[0153] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for human gait recognition based on dynamic spatiotemporal graph convolutional neural networks, characterized in that: Includes the following steps: Step 1: Input the CASIA-B dataset into the human pose extraction module, use the HRNet model to estimate human pose, and generate a heatmap. Step 2: Input the heat map obtained in Step 1 into the HM-C module. HM-C uses the distribution-aware coordinate representation method to re-encode the key point coordinates, and finally generates a vector sequence containing key point location information, which is the human skeleton sequence information. Step 3: Extract joint, bone, and velocity features from the human skeleton sequence information obtained in Step 2; Step 4: Input the joint, bone, and velocity features obtained in Step 3 into the local attention module to obtain a feature vector containing the weight information of different body parts in the entire action sequence. Step 5: Input the human skeleton sequence information obtained in Step 2 into the backbone network dynamic GCN module to model in the spatial dimension, fuse spatial features, and generate a spatial feature map. Step 5 includes: Step 51: Initialize the coefficient matrix A randomly using a normal distribution. The coefficient matrix A consists of N different components, and N can be set to any value to complete the design of multiple sets in the dynamic graph convolution module. Step 52: The skeleton sequence information with C channels obtained in Step 2 is used as the input to the spatial convolution submodule. First, it is processed using a 1×1 convolution to generate N feature groups, each feature group having… One channel; Step 53: Given the input feature F, first use time pooling to eliminate dimension T; then apply two separate 1×1 convolutions to F to obtain F1 and F2 respectively; Step 54: Use the Softmax activation function to obtain the dynamic term from F1 and F2 obtained in step 53. Tanh activation function to obtain dynamic terms with a specific channel. and The expression is as follows: Step 55, take the result from step 54 and With static items The weighted sum is used as the final coefficient matrix A. i A i The expression is as follows: Here, α and β are two learnable parameters; Step 56, take A obtained in step 55 i The N feature groups obtained in step 52 are modeled independently, and then the N feature groups are cascaded along the channel dimension; Step 57: The concatenated results of the N feature groups obtained in step 56 are processed by a 1×1 convolution to generate a spatial feature map. Step 6: Input the human skeleton sequence information obtained in Step 2 and the spatial feature map obtained in Step 5 into the dynamic TCN module to aggregate temporal features, learn the temporal patterns between adjacent frames, and combine the spatially captured information with temporal information. By comprehensively considering the spatial and temporal information of the action, a richer temporal feature map is generated. Step 6 includes: Step 61: Input the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 5 into the bottleneck module in the TCN module, and generate M feature groups through a 1×1 convolution. The multiple TCNs are composed of multiple branches with different receptive fields. Step 62: Split the sequence of length T obtained in step 2 into L substrings of equal length, then randomly select a frame from each substring and concatenate them into a new subsequence; Step 63: Perform convolution and pooling operations along the channel dimension on the M feature groups obtained in step 61 and the L sample sequences obtained in step 62 respectively, and then concatenate the results. Step 7: Input the temporal feature map obtained in Step 6 into the joint-skeleton dynamic fusion module to fuse the joint-level and skeleton-level motion patterns in different time ranges, generating richer and more discriminative joint-skeleton motion features. Step 7 includes: Step 71: Further model the parallel execution time of the results obtained in Step 6 at the joint and skeleton levels. First, model the V joint-level features {X1, ..., X...} V |X i ∈R C×T Perform average pooling to obtain skeleton-level features S; Step 72, combine the skeleton-level features S and V joint features X obtained in step 71. i Perform parallel processing to obtain S * and X i * ; Step 73, take the S obtained in step 72 * Merge into X i * middle; Step 74, each TCN instance contains a learning parameter. After adaptive joint skeleton fusion, the features of joint i are: Step 75: Perform a 1×1 convolution operation on the adaptive joint skeleton fusion feature sequence obtained in step 74 to obtain the output of the TCN module.

2. The method for human gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: Step 1 includes: Step 11: Input the original images from the CASIA-B dataset into the HRNet model to generate high-resolution feature maps in the high-resolution sub-network. The high-resolution sub-network receives the high-resolution feature maps from the previous stage as input and generates the high-resolution feature maps for the current stage. Step 12: The low-resolution sub-network of the HRNet model receives the low-resolution feature map from the previous stage as input and generates the low-resolution feature map for the current stage. High and low resolution sub-networks are added step by step to form more stages; each stage contains a high-resolution sub-network and a low-resolution sub-network. Step 13: Connect all the stages obtained in step 12 in parallel and process the input image simultaneously; the high-resolution feature map and low-resolution feature map of each stage are sent to the next stage. Step 14: Fuse the high-resolution feature maps and low-resolution feature maps obtained in the same stage in Step 13 together, and integrate the feature maps obtained in different stages in Step 13 using multi-scale fusion technology. Step 15: Input the fused feature map obtained in step 14 into the pose estimation module. After passing through multiple convolutional and pooling layers, extract features and reduce the size of the feature map to output a heatmap of human pose.

3. The method for human gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: Step 2 includes: Step 21: Modulate the heatmap distribution of the heatmap obtained in Step 1, and use a Gaussian kernel K with the same variation as the training data to smooth the influence of multiple peaks in the heatmap R. The expression is as follows: R′=K*R Here, * represents the convolution operation. To preserve the size of the original heatmap, R is ultimately scaled to 0 so that its maximum activation equals the maximum activation of R, as shown in the following expression: The max() and min() functions return the maximum and minimum values ​​of the input matrix, respectively. Step 22: Using sub-pixel precision Taylor expansion for distribution-aware joint localization, assuming the predicted heatmap follows a 2D Gaussian distribution, the same as the ground-based heatmap, the predicted heatmap expression is as follows: Where w is the pixel position in the predicted heatmap, ū is the Gaussian mean corresponding to the joint position to be estimated, and covariance ∑ is a diagonal matrix, the same matrix used in coordinate encoding, expressed as follows: Where σ is the standard deviation in both directions; Step 23: Given the heatmap R predicted by the trained model, first determine the coordinates of the maximum m and the second maximum s activations. The joint positions will then be predicted as follows: Here, || defines the size of the vector, which means that the prediction is the maximum activation, with 0.25 pixels offset to the second maximum activation in the heatmap space; Step 24: Convert the predicted joint positions into coordinates in the original image space. The final coordinate prediction in the original image is calculated as follows: P r * =δP r Where δ is the resolution reduction ratio; Step 25: Draw the connections between key points as skeletal information, and finally generate a vector sequence containing key point position information, which is the human skeletal sequence information.

4. The method for human gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: Step 3 includes: Step 31: Obtain the original three-dimensional coordinate set X = {x∈R} from the human skeleton sequence information obtained in Step 2. C×T×V }, where C, T, and V represent coordinates, frames, and joints, respectively. Then, the relative position of each joint at each time step is calculated, resulting in a set of relative positions R = {γ}. i |i=1,2…,V}, where for each joint i, its relative position γ i It is obtained by subtracting the coordinates of the central joint from its coordinates, and the two sets are concatenated into a single sequence to obtain the relative positions of the joints; Step 32: Calculate the velocity of each joint at each time step from the human skeletal sequence information obtained in Step 2 to obtain two sets of motion velocities, represented as follows: S′={s′ t |t=1,2…,T} S″={s″ t |t=1,2…,T} Where S' represents the displacement of each joint between the current time step and the next time step, S” represents the displacement of each joint between the current time step and the previous time step, t represents a certain moment in the time series, and T represents the total length of the time series. Finally, by calculating the S of each joint... ‘ The speed of motion is obtained by cascading with "S". Step 33: Calculate the Euclidean distance between joint i and its parent joint from the human skeleton sequence information obtained in Step 2 to obtain the bone length; calculate the angle between joint i, its parent joint, and its child joint to obtain the bone angle, as shown in the following expression: Among them, l i θ represents bone length. i The x, y, and z axes represent the x-axis, y-axis, and z-axis in a 3D coordinate system, respectively, and the final bone length l is obtained. i The set and the skeleton angle θ i The set of features represents the overall skeletal characteristics.

5. The method for human gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: Step 4 includes: Step 41: Select the corresponding joints, obtain six individual body parts from the bone features obtained in Step 3, and then connect the features of all parts together. Step 42: Perform average pooling and convolution operations on the partial feature sequences obtained in step 41 in the time dimension; Step 43: Input the vector output from step 42 into a fully connected layer with a BatchNorm layer and a ReLU function; Step 44: The vector output from step 43 is processed through six fully connected layers to calculate the attention matrix, and the Softmax function is used to determine the most important body parts. Step 45: Connect the features of the six parts obtained in Step 44 into an overall skeleton representation with different attention weights. The formula for the local attention module is: Among them, f in This represents the input feature map, where p represents the index of the body part. Learnable parameters shared by all components. Each part used to compute the final attention weights is a global context feature map obtained through average pooling over the entire time series.

Images