Character gait recognition method based on dynamic space-time diagram convolutional neural network

Abstract

The invention discloses a figure gait recognition method based on a dynamic space-time diagram convolutional neural network, and the method comprises the following steps: carrying out the human body posture estimation of a CASIA-B data set, and generating a thermodynamic diagram; inputting the thermodynamic diagram into an HM-C module to generate human skeleton sequence information; extracting joint, skeleton and speed features; inputting into a local attention module to obtain feature vectors containing weight information of different body parts in the whole action sequence; inputting the data into a backbone network dynamic GCN module to generate a spatial feature map; inputting the human skeleton sequence information and the spatial features into a dynamic TCN module to generate a rich time feature map; and inputting the time feature map into a joint and skeleton dynamic fusion module, fusing joint-level and skeleton-level motion modes in different time ranges, and generating joint-skeleton motion features which are richer and stronger in discrimination. According to the method, the problem that a baseline GCN method seriously depends on a manually defined skeleton topology is solved.

Description

Technical Field

[0001] The present invention relates to a skeleton-based human gait recognition method, and belongs to the cross-technical fields of computer vision, image processing, posture recognition, gait recognition, etc. Background Art

[0002] Gait recognition is a promising video-based biometric approach that offers significant advantages over other unique biometric features such as face, fingerprint, and iris in recognition at a distance and does not require cooperation or intrusion of the subject. It thus opens up great potential for applications such as social security, access control, and forensics. However, gait can be sensitive to surface type, clothing, items carried, and clutter or occlusions in the scene. These represent challenges in tackling the gait recognition task to learn unique and invariant features from human gait.

[0003] In recent years, most gait recognition methods use human silhouettes to extract gait features. However, in real-world scenes with cluttered and rapidly changing environments, silhouette images may lose fine-grained spatial information and be affected by external factors such as self-occlusion. Skeleton-based gait recognition focuses on the bone structure and can capture the depth and spatial characteristics of movement. Therefore, it is more resistant to interference factors such as light, shadow, occlusion, and wearing different clothing. With the development of posture estimation based on deep learning, a new era of skeleton-based gait recognition and model-based gait recognition has been ushered in.

[0004] A method based on graph convolutional networks (GCNs) came into being. Yan et al. first proposed ST-GCN to model skeletal motion patterns using spatiotemporal graphs, and the two-stream adaptive graph convolutional network (2sAGCN), which introduced an adaptive adjacency matrix and pre-computed skeletal information as second-order inputs. SGN performed a deeper analysis of these second-order features.

[0005] However, graph convolutional networks (GCNs), which have made great strides in the new era of skeleton-based and model-based gait recognition, have proven good effectiveness, but these methods still rely heavily on manually defined skeleton topologies and require painstaking customization for different datasets. The network architecture of dynamic GCN, which is extended from the traditional GCN baseline model, is more suitable for modeling complex joint correlations because it is an intrinsic cooperation between joints when performing actions, and is a completely learnable coefficient matrix rather than a prescribed graph structure. At the same time, most previous GCN-based methods use temporal convolutions with fixed receptive fields to model joint-level motion within a specific time range, ignoring the benefits of modeling multi-level (i.e., joint-level + skeleton-level) motion patterns within a dynamic temporal receptive field, so dynamic spatiotemporal graph convolutional network models are urgently needed to address the limitations of existing work. Summary of the invention

[0006] The purpose of the present invention is to provide a human gait recognition method based on a dynamic spatiotemporal graph convolutional neural network to solve the problem that the existing baseline GCN method heavily relies on manually defined skeleton topology.

[0007] To achieve the above object, the present invention adopts the following technical solution:

[0008] A method for character gait recognition based on a dynamic spatiotemporal graph convolutional neural network comprises the following steps:

[0009] Step 1: Input the CASIA-B dataset into the human posture extraction module, use the HRNet model to estimate the human posture, and generate a heat map;

[0010] Step 2: Input the heat map obtained in step 1 into the HM-C module. HM-C uses the distribution-aware coordinate representation method to re-encode the key point coordinates, and finally generates a vector sequence containing the key point position information, which is the human skeleton sequence information;

[0011] Step 3, extracting joint, bone and speed features from the human skeleton sequence information obtained in step 2;

[0012] Step 4: Input the joint, bone and velocity features obtained in step 3 into the local attention module to obtain a feature vector containing the weight information of different body parts in the entire action sequence;

[0013] Step 5: Input the human skeleton sequence information obtained in step 2 into the backbone network dynamic GCN module to model in the spatial dimension, fuse the spatial features, and generate a spatial feature map;

[0014] Step 6: Input the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 6 into the dynamic TCN module to aggregate the temporal features, learn the temporal patterns between adjacent frames, and combine the spatially captured information with the temporal information, comprehensively consider the spatial and temporal information of the action, and generate a richer temporal feature map;

[0015] Step 7, input the time feature map obtained in step 7 into the joint-skeleton dynamic fusion module, fuse the motion patterns at the joint level and the skeleton level in different time ranges, and generate richer and more discriminative joint-skeleton motion features.

[0016] The step 1 comprises:

[0017] Step 11, input the original image in the CASIA-B dataset into the HRNet model, generate a high-resolution feature map in the high-resolution sub-network, the high-resolution sub-network receives the high-resolution feature map of the previous stage as input, and generates a high-resolution feature map of the current stage;

[0018] Step 12: The low-resolution subnetwork of the HRNet model receives the low-resolution feature map of the previous stage as input, generates the low-resolution feature map of the current stage, and gradually adds high- and low-resolution subnetworks to form more stages; each stage contains a high-resolution subnetwork and a low-resolution subnetwork;

[0019] Step 13, all the stages obtained in step 12 are connected in parallel to process the input image simultaneously; the high-resolution feature map and the low-resolution feature map of each stage are sent to the next stage;

[0020] Step 14, fusing the high-resolution feature map and the low-resolution feature map in the same stage obtained in step 13, and integrating the feature maps of different stages obtained in step 13 using a multi-scale fusion technique;

[0021] In step 15, the fused feature map obtained in step 14 is input into the posture estimation module, and after passing through multiple convolutional layers and pooling layers, the features are extracted and the size of the feature map is reduced to output a heat map of the human body posture.

[0022] The step 2 comprises:

[0023] Step 21, the heat map obtained in step 1 is modulated by heat map distribution, and the influence of multiple peaks in the heat map R is smoothed using a Gaussian kernel K with the same variation as the training data. The expression is as follows:

[0024] R=K*R

[0025] Among them, * is the convolution operation. In order to maintain the size of the original heat map, R is finally scaled to 0 so that its maximum activation is equal to the maximum activation of R. The expression is as follows:

[0026]

[0027] Among them, max() and min() return the maximum and minimum values ​​of the input matrix respectively;

[0028] Step 22, distribution-aware joint localization is performed through Taylor expansion with sub-pixel accuracy. Assuming that the predicted heat map follows a 2D Gaussian distribution, which is the same as the ground truth heat map, the predicted heat map expression is as follows:

[0029]

[0030] Where w is the pixel location in the predicted heatmap, ū is the Gaussian mean (center) corresponding to the joint location to be estimated; the covariance ∑ is a diagonal matrix, the same as the matrix used in coordinate encoding, expressed as follows:

[0031]

[0032] where σ is the standard deviation in both directions;

[0033] Step 23, given the heatmap R predicted by the trained model, first determine the coordinates of the maximum (m) and second maximum (s) activations, and the position of the joint will be predicted as:

[0034]

[0035] where || defines the magnitude of the vector, meaning that the prediction is the largest activation with a 0.25 pixel (i.e. sub-pixel) shift towards the second largest activation in the heatmap space;

[0036] Step 24, convert the predicted joint positions into coordinates in the original image space, and the final coordinate prediction in the original image is calculated as:

[0037] P r * =δP r

[0038] Where δ is the resolution reduction ratio;

[0039] Step 25, draw the connection between the key points as the skeleton information, and finally generate a vector sequence containing the key point position information, which is the human skeleton sequence information.

[0040] The step 3 comprises:

[0041] Step 31, the original three-dimensional coordinate set is obtained from the human skeleton sequence information obtained in step 2 as X = {x∈R C ×T×V}, where C, T and V represent coordinates, frames and joints respectively, and then calculate the relative position of each joint at each time step to obtain a relative position set R = {γ i |i=1,2…,V}, where for each joint i, its relative position γ i It is obtained by subtracting the coordinates of the central joint from its coordinates, and these two sets are concatenated into a single sequence to obtain the relative position of the joint;

[0042] Step 32, calculating the speed of each joint at each time step from the human skeleton sequence information obtained in step 2 to obtain two sets of motion speeds, which are respectively expressed as:

[0043] S={st |t=1,2…,T}

[0044] S″={s″ t |t=1,2…,T}

[0045] Among them, S ‘ represents the displacement of each joint between the current time step and the next time step, S' represents the displacement of each joint between the current time step and the previous time step, t represents a moment in the time series, T represents the total length of the time series, and finally the S of each joint is calculated. ‘ and S” cascade to obtain the movement speed;

[0046] Step 33, from the human skeleton sequence information obtained in step 2, the Euclidean distance between joint i and its parent joint is calculated to obtain the bone length, and the angle between joint i, its parent joint and its child joint is calculated to obtain the bone angle, which is expressed as follows:

[0047]

[0048] Among them, l i represents the bone length, θ i represents the bone angle, x, y, z represent the x-axis, y-axis and z-axis in the 3D coordinates respectively, and the final bone length l i The collection and bone angle θ i The collection of is the total bone features.

[0049] The step 4 comprises:

[0050] Step 41, select the corresponding joints, obtain six separate body parts from the bone features obtained in step 3, and then connect the features of all parts;

[0051] Step 42, performing average pooling and convolution operations on the partial feature sequence obtained in step 41 in the time dimension;

[0052] Step 43, input the vector output from step 42 into a fully connected layer having a BatchNorm layer and a ReLU function;

[0053] Step 44, the vector output from step 43 is used to calculate the attention matrix using six fully connected layers, and the Softmax function is used to determine the most important body parts;

[0054] Step 45, the features of the six parts obtained in step 44 are connected into an overall skeleton representation with different attention weights. The formula of the local attention module is:

[0055]

[0056] Among them, f in represents the input feature map, p represents the index of the body part, are learnable parameters shared by all components, A global context feature map specific to each part used to compute the final attention weights is obtained by average pooling over the entire time series.

[0057] The step 5 comprises:

[0058] Step 51, using normal distribution to randomly initialize the coefficient matrix A, the coefficient matrix A is composed of N different components, N can be set to any value, and multiple groups of designs in the dynamic graph convolution module are completed;

[0059] Step 52, the skeleton sequence information with C channels obtained in step 2 is used as the input of the spatial convolution submodule, and is first processed by 1×1 convolution to generate N feature groups, each of which has channels;

[0060] Step 53, given the input feature F, first use temporal pooling to eliminate the dimension T; then apply two separate 1×1 convolutions to F to obtain F1 and F2 respectively;

[0061] Step 54: Use the Softmax activation function to obtain the dynamic term F1 and F2 obtained in step 53 Tanh activation function to obtain dynamic terms with specific channels and The expression is as follows:

[0062]

[0063] Step 55: convert the and With static items The weighted sum is the final coefficient matrix A i , A i The expression is as follows:

[0064]

[0065] Among them, α and β are two learnable parameters;

[0066] Step 56: A obtained in step 55 i Modeling is performed independently from the N feature groups obtained in step 52, and then the N feature groups are cascaded along the channel dimension;

[0067] In step 57, the result of concatenating the N feature groups obtained in step 56 is processed through a 1×1 convolution to generate a spatial feature map.

[0068] The step 6 comprises:

[0069] Step 61, inputting the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 5 into the bottleneck module in the TCN module, and processing them through a 1×1 convolution to generate M feature groups, and the multiple TCN groups are composed of multiple branches with different receptive fields;

[0070] Step 62, splitting the sequence of length T obtained in step 2 into M substrings of equal length, and then randomly selecting a frame from each substring and concatenating them into a new subsequence;

[0071] In step 63, the M feature groups obtained in step 61 and the M sample sequences obtained in step 62 are convolved and pooled along the channel dimension respectively, and the results are connected.

[0072] The step 7 comprises:

[0073] Step 71, the result obtained in step 6 is further modeled in parallel at the joint level and the skeleton level. First, V joint-level features {X1, ..., X V |X i ∈R C×T}Perform average pooling to obtain skeleton-level features S;

[0074] Step 72: The skeleton-level features S and V joint features X obtained in step 71 are i Perform parallel processing to obtain S * and X i * ;

[0075] Step 73, the S obtained in step 72 * Merge to X i * middle;

[0076] Step 74, each TCN instance contains a learning parameter After adaptive joint skeleton fusion, the characteristics of joint i are

[0077] In step 75, a 1×1 convolution operation is performed on the feature sequence after the adaptive joint skeleton fusion obtained in step 74 to obtain the output of the TCN module.

[0078] Beneficial effects: Compared with the prior art, the present invention has the following technical effects:

[0079] The present invention first obtains a vector sequence containing key point position information, namely, human skeleton sequence information, through the HRNet and HW-C modules, discovers the importance of different body parts in the entire action sequence through the local attention module, and then dynamically models the human posture of each frame in space through the dynamic GCN module, and then enters the TCN network to learn temporal features, followed by the joint-skeleton dynamic fusion module for explicit modeling and dynamic fusion of joint-level and skeleton-level features, and performs temporal modeling in parallel at the joint level and skeleton level, and finally obtains the final prediction result through the pooling layer and the fully connected layer.

[0080] Specifically:

[0081] (1) The present invention uses HRNet and HW-C modules to extract human skeleton sequence information, which is conducive to obtaining joint, bone and speed features in the human contour.

[0082] (2) The present invention introduces an attention mechanism to realize the importance of different body parts in the entire action sequence.

[0083] (3) The present invention performs spatial and temporal modeling through a dynamic GCN module and a TCN module to solve the problem that the baseline GCN method proposed in the above background technology heavily relies on manually defined skeleton topology.

[0084] (4) The joint-skeleton dynamic fusion module fuses the motion patterns at the joint level and the skeleton level in different time ranges to generate richer and more discriminative joint-skeleton motion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is a flow chart of the overall method of the present invention;

[0086] Figure 2 This is the structure diagram of the dynamic GCN module;

[0087] Figure 3 It is the structure diagram of the dynamic TCN module;

[0088] Figure 4 It is the structure diagram of the local attention module. DETAILED DESCRIPTION

[0089] The present invention will be further explained below in conjunction with the accompanying drawings.

[0090] A method for character gait recognition based on a dynamic spatiotemporal graph convolutional neural network comprises the following steps:

[0091] Step 1: Input the CASIA-B dataset into the human posture extraction module, use the HRNet model to estimate the human posture, and generate a heat map; specifically, it includes:

[0092] Step 11, input the original image in the CASIA-B dataset into the HRNet model, generate a high-resolution feature map in the high-resolution sub-network, the high-resolution sub-network receives the high-resolution feature map of the previous stage as input, and generates a high-resolution feature map of the current stage;

[0093] Step 12: The low-resolution subnetwork of the HRNet model receives the low-resolution feature map of the previous stage as input, generates the low-resolution feature map of the current stage, and gradually adds high- and low-resolution subnetworks to form more stages; each stage contains a high-resolution subnetwork and a low-resolution subnetwork;

[0094] Step 13, all the stages obtained in step 12 are connected in parallel to process the input image simultaneously; the high-resolution feature map and the low-resolution feature map of each stage are sent to the next stage;

[0095] Step 14, fusing the high-resolution feature map and the low-resolution feature map in the same stage obtained in step 13, and integrating the feature maps of different stages obtained in step 13 using a multi-scale fusion technique to produce a more accurate result;

[0096] In step 15, the fused feature map obtained in step 14 is input into the posture estimation module, and after passing through multiple convolutional layers and pooling layers, the features are extracted and the size of the feature map is reduced to output a heat map of the human body posture.

[0097] Step 2: Input the heat map obtained in step 1 into the HM-C module. HM-C uses the distribution-aware coordinate representation method to re-encode the key point coordinates, and finally generates a vector sequence containing the key point position information, which is the human skeleton sequence information; specifically, it includes:

[0098] Step 21, the heat map obtained in step 1 is modulated by heat map distribution, and the influence of multiple peaks in the heat map R is smoothed using a Gaussian kernel K with the same variation as the training data. The expression is as follows:

[0099] R=K*R

[0100] Among them, * is the convolution operation. In order to maintain the size of the original heat map, R is finally scaled to 0 so that its maximum activation is equal to the maximum activation of R. The expression is as follows:

[0101]

[0102] Among them, max() and min() return the maximum and minimum values ​​of the input matrix respectively;

[0103] Step 22, distribution-aware joint localization is performed through Taylor expansion with sub-pixel accuracy. Assuming that the predicted heat map follows a 2D Gaussian distribution, which is the same as the ground truth heat map, the predicted heat map expression is as follows:

[0104]

[0105] Where w is the pixel location in the predicted heatmap, ū is the Gaussian mean (center) corresponding to the joint location to be estimated; the covariance ∑ is a diagonal matrix, the same as the matrix used in coordinate encoding, expressed as follows:

[0106]

[0107] where σ is the standard deviation in both directions;

[0108] Step 23, given the heatmap R predicted by the trained model, first determine the coordinates of the maximum (m) and second maximum (s) activations, and the position of the joint will be predicted as:

[0109]

[0110] where || defines the magnitude of the vector, meaning that the prediction is the largest activation with a 0.25 pixel (i.e. sub-pixel) shift towards the second largest activation in the heatmap space;

[0111] Step 24, convert the predicted joint positions into coordinates in the original image space, and the final coordinate prediction in the original image is calculated as:

[0112] P r * =δP r

[0113] Where δ is the resolution reduction ratio;

[0114] Step 25, draw the connection between the key points as the skeleton information, and finally generate a vector sequence containing the key point position information, which is the human skeleton sequence information.

[0115] Step 3, extracting joint, bone and speed features from the human skeleton sequence information obtained in step 2; specifically including:

[0116] Step 31, the original three-dimensional coordinate set is obtained from the human skeleton sequence information obtained in step 2 as X = {x∈R C ×T×V}, where C, T and V represent coordinates, frames and joints respectively, and then calculate the relative position of each joint at each time step to obtain a relative position set R = {γ i |i=1,2…,V}, where for each joint i, its relative position γ iIt is obtained by subtracting the coordinates of the central joint from its coordinates, and these two sets are concatenated into a single sequence to obtain the relative position of the joint;

[0117] Step 32, calculating the speed of each joint at each time step from the human skeleton sequence information obtained in step 2 to obtain two sets of motion speeds, which are respectively expressed as:

[0118] S={s t |t=1,2…,T}

[0119] S″={s″ t |t=1,2…,T}

[0120] Among them, S ‘ represents the displacement of each joint between the current time step and the next time step, S' represents the displacement of each joint between the current time step and the previous time step, t represents a moment in the time series, T represents the total length of the time series, and finally the S of each joint is calculated. ‘ and S” cascade to obtain the movement speed;

[0121] Step 33, from the human skeleton sequence information obtained in step 2, the Euclidean distance between joint i and its parent joint is calculated to obtain the bone length, and the angle between joint i, its parent joint and its child joint is calculated to obtain the bone angle, which is expressed as follows:

[0122]

[0123] Among them, l i represents the bone length, θ i represents the bone angle, x, y, z represent the x-axis, y-axis and z-axis in the 3D coordinates respectively, and the final bone length l i The collection and bone angle θ i The collection of is the total bone features.

[0124] Step 4: Input the joint, bone and velocity features obtained in step 3 into the local attention module to obtain a feature vector containing the weight information of different body parts in the entire action sequence; specifically, including:

[0125] Step 41, select the corresponding joints, obtain six separate body parts from the bone features obtained in step 3, and then connect the features of all parts;

[0126] Step 42, performing average pooling and convolution operations on the partial feature sequence obtained in step 41 in the time dimension;

[0127] Step 43, input the vector output from step 42 into a fully connected layer having a BatchNorm layer and a ReLU function;

[0128] Step 44, the vector output from step 43 is used to calculate the attention matrix using six fully connected layers, and the Softmax function is used to determine the most important body parts;

[0129] Step 45, the features of the six parts obtained in step 44 are connected into an overall skeleton representation with different attention weights. The formula of the local attention module is:

[0130]

[0131] Among them, f in represents the input feature map, p represents the index of the body part, are learnable parameters shared by all components, A global context feature map specific to each part used to compute the final attention weights is obtained by average pooling over the entire time series.

[0132] Step 5: Input the human skeleton sequence information obtained in step 2 into the backbone network dynamic GCN module to model in the spatial dimension, fuse the spatial features, and generate a spatial feature map; specifically including:

[0133] Step 51, using normal distribution to randomly initialize the coefficient matrix A, the coefficient matrix A is composed of N different components, N can be set to any value, and multiple groups of designs in the dynamic graph convolution module are completed;

[0134] Step 52, the skeleton sequence information with C channels obtained in step 2 is used as the input of the spatial convolution submodule, and is first processed by 1×1 convolution to generate N feature groups, each of which has channels;

[0135] Step 53, given the input feature F, first use temporal pooling to eliminate the dimension T; then apply two separate 1×1 convolutions to F to obtain F1 and F2 respectively;

[0136] Step 54: Use the Softmax activation function to obtain the dynamic term F1 and F2 obtained in step 53 Tanh activation function to obtain dynamic terms with specific channels and The expression is as follows:

[0137]

[0138] Step 55: convert the and With static items The weighted sum is the final coefficient matrix A i , Ai The expression is as follows:

[0139]

[0140] Among them, α and β are two learnable parameters;

[0141] Step 56: A obtained in step 55 i Modeling is performed independently from the N feature groups obtained in step 52, and then the N feature groups are cascaded along the channel dimension;

[0142] In step 57, the result of concatenating the N feature groups obtained in step 56 is processed through a 1×1 convolution to generate a spatial feature map.

[0143] Step 6: Input the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 6 into the dynamic TCN module to aggregate the temporal features, learn the temporal patterns between adjacent frames, and combine the spatially captured information with the temporal information, comprehensively consider the spatial and temporal information of the action, and generate a richer temporal feature map; specifically, it includes:

[0144] Step 61, inputting the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 5 into the bottleneck module in the TCN module, and processing them through a 1×1 convolution to generate M feature groups, and the multiple TCN groups are composed of multiple branches with different receptive fields;

[0145] Step 62, splitting the sequence of length T obtained in step 2 into M substrings of equal length, and then randomly selecting a frame from each substring and concatenating them into a new subsequence; this uniform sampling method can generate more diverse samples while maintaining the integrity of each sample;

[0146] In step 63, the M feature groups obtained in step 61 and the M sample sequences obtained in step 62 are convolved and pooled along the channel dimension respectively, and the results are connected.

[0147] Step 7, input the time feature graph obtained in step 7 into the joint-skeleton dynamic fusion module, fuse the joint-level and skeleton-level motion patterns in different time ranges, and generate richer and more discriminative joint-skeleton motion features; specifically including:

[0148] Step 71, the result obtained in step 6 is further modeled in parallel at the joint level and the skeleton level. First, V joint-level features {X1, ..., X V |X i ∈R C×T}Perform average pooling to obtain skeleton-level features S;

[0149] Step 72: The skeleton-level features S and V joint features X obtained in step 71 are i Perform parallel processing to obtain S * and X i * ;

[0150] Step 73, the S obtained in step 72 * Merge to X i * middle;

[0151] Step 74, each TCN instance contains a learning parameter After adaptive joint skeleton fusion, the characteristics of joint i are

[0152] Step 75, perform a 1×1 convolution operation on the adaptive joint skeleton fusion feature sequence obtained in step 74 to obtain the output of the TCN module

[0153] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network, characterized in that: The following steps are involved: Step 1: Input the CASIA-B dataset into the human posture extraction module, use the HRNet model to estimate the human posture, and generate a heat map; Step 2: Input the heat map obtained in step 1 into the HM-C module. HM-C uses the distribution-aware coordinate representation method to re-encode the key point coordinates, and finally generates a vector sequence containing the key point position information, which is the human skeleton sequence information; Step 3, extracting joint, bone and speed features from the human skeleton sequence information obtained in step 2; Step 4: Input the joint, bone and velocity features obtained in step 3 into the local attention module to obtain a feature vector containing the weight information of different body parts in the entire action sequence; Step 5: Input the human skeleton sequence information obtained in step 2 into the backbone network dynamic GCN module to model in the spatial dimension, fuse the spatial features, and generate a spatial feature map; Step 6: Input the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 6 into the dynamic TCN module to aggregate the temporal features, learn the temporal patterns between adjacent frames, and combine the spatially captured information with the temporal information, comprehensively consider the spatial and temporal information of the action, and generate a richer temporal feature map; Step 7, input the time feature map obtained in step 7 into the joint-skeleton dynamic fusion module, fuse the motion patterns at the joint level and the skeleton level in different time ranges, and generate richer and more discriminative joint-skeleton motion features.

2. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 1 comprises: Step 11, input the original image in the CASIA-B dataset into the HRNet model, generate a high-resolution feature map in the high-resolution sub-network, the high-resolution sub-network receives the high-resolution feature map of the previous stage as input, and generates a high-resolution feature map of the current stage; Step 12: The low-resolution subnetwork of the HRNet model receives the low-resolution feature map of the previous stage as input, generates the low-resolution feature map of the current stage, and gradually adds high- and low-resolution subnetworks to form more stages; each stage contains a high-resolution subnetwork and a low-resolution subnetwork; Step 13, all the stages obtained in step 12 are connected in parallel to process the input image simultaneously; the high-resolution feature map and the low-resolution feature map of each stage are sent to the next stage; Step 14, fusing the high-resolution feature map and the low-resolution feature map in the same stage obtained in step 13, and integrating the feature maps of different stages obtained in step 13 using a multi-scale fusion technique; In step 15, the fused feature map obtained in step 14 is input into the posture estimation module, and after passing through multiple convolutional layers and pooling layers, the features are extracted and the size of the feature map is reduced to output a heat map of the human body posture.

3. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 2 comprises: Step 21, the heat map obtained in step 1 is modulated by heat map distribution, and the influence of multiple peaks in the heat map R is smoothed using a Gaussian kernel K with the same variation as the training data. The expression is as follows: R′=K*R Among them, * is the convolution operation. In order to maintain the size of the original heat map, R is finally scaled to 0 so that its maximum activation is equal to the maximum activation of R. The expression is as follows: Among them, max() and min() return the maximum and minimum values ​​of the input matrix respectively; Step 22, distribution-aware joint localization is performed through Taylor expansion with sub-pixel accuracy. Assuming that the predicted heat map follows a 2D Gaussian distribution, which is the same as the ground truth heat map, the predicted heat map expression is as follows: Where w is the pixel location in the predicted heatmap, ū is the Gaussian mean (center) corresponding to the joint location to be estimated; the covariance ∑ is a diagonal matrix, the same as the matrix used in coordinate encoding, expressed as follows: where σ is the standard deviation in both directions; Step 23, given the heatmap R predicted by the trained model, first determine the coordinates of the maximum (m) and second maximum (s) activations, and the position of the joint will be predicted as: where || defines the magnitude of the vector, meaning that the prediction is the largest activation with a 0.25 pixel (i.e. sub-pixel) shift towards the second largest activation in the heatmap space; Step 24, convert the predicted joint positions into coordinates in the original image space, and the final coordinate prediction in the original image is calculated as: P r * =δP r Where δ is the resolution reduction ratio; Step 25, draw the connection between the key points as the skeleton information, and finally generate a vector sequence containing the key point position information, which is the human skeleton sequence information.

4. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 3 comprises: Step 31, the original three-dimensional coordinate set is obtained from the human skeleton sequence information obtained in step 2 as X = {x∈R C×T×V }, where C, T and V represent coordinates, frames and joints respectively, and then calculate the relative position of each joint at each time step to obtain a relative position set R = {γ i |i=1,2…,V}, where for each joint i, its relative position γ i It is obtained by subtracting the coordinates of the central joint from its coordinates, and these two sets are concatenated into a single sequence to obtain the relative position of the joint; Step 32, calculating the speed of each joint at each time step from the human skeleton sequence information obtained in step 2 to obtain two sets of motion speeds, which are respectively expressed as: S′={s′ t |t=1,2…,T} S″={s″ t |t=1,2…,T} Among them, S' represents the displacement of each joint between the current time step and the next time step, S" represents the displacement of each joint between the current time step and the previous time step, t represents a moment in the time series, T represents the total length of the time series, and finally the motion speed is obtained by cascading S' and S" of each joint; Step 33, from the human skeleton sequence information obtained in step 2, the Euclidean distance between joint i and its parent joint is calculated to obtain the bone length, and the angle between joint i, its parent joint and its child joint is calculated to obtain the bone angle, which is expressed as follows: Among them, l i represents the bone length, θ i represents the bone angle, x, y, z represent the x-axis, y-axis and z-axis in the 3D coordinates respectively, and the final bone length l i The collection and bone angle θ i The collection of is the total bone features.

5. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 4 comprises: Step 41, select the corresponding joints, obtain six separate body parts from the bone features obtained in step 3, and then connect the features of all parts; Step 42, performing average pooling and convolution operations on the partial feature sequence obtained in step 41 in the time dimension; Step 43, input the vector output from step 42 into a fully connected layer having a BatchNorm layer and a ReLU function; Step 44, the vector output from step 43 is used to calculate the attention matrix using six fully connected layers, and the Softmax function is used to determine the most important body parts; Step 45, the features of the six parts obtained in step 44 are connected into an overall skeleton representation with different attention weights. The formula of the local attention module is: Among them, f in represents the input feature map, p represents the index of the body part, are learnable parameters shared by all components, A global context feature map specific to each part used to compute the final attention weights is obtained by average pooling over the entire time series.

6. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 5 comprises: Step 51, using normal distribution to randomly initialize the coefficient matrix A, the coefficient matrix A consists of N different components, N can be set to any value, and multiple groups of designs in the dynamic graph convolution module are completed; Step 52, the skeleton sequence information with C channels obtained in step 2 is used as the input of the spatial convolution submodule, and is first processed by 1×1 convolution to generate N feature groups, each of which has channels; Step 53, given the input feature F, first use temporal pooling to eliminate the dimension T; then apply two separate 1×1 convolutions to F to obtain F1 and F2 respectively; Step 54: Use the Softmax activation function to obtain the dynamic term F1 and F2 obtained in step 53 Tanh activation function to obtain dynamic terms with specific channels and The expression is as follows: Step 55: convert the and With static items The weighted sum is the final coefficient matrix A i , A i The expression is as follows: Among them, α and β are two learnable parameters; Step 56: A obtained in step 55 i Modeling is performed independently from the N feature groups obtained in step 52, and then the N feature groups are cascaded along the channel dimension; In step 57, the result of concatenating the N feature groups obtained in step 56 is processed through a 1×1 convolution to generate a spatial feature map.

7. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 6 comprises: Step 61, inputting the human skeleton sequence information obtained in step 2 and the spatial features obtained in step 5 into the bottleneck module in the TCN module, and processing them through a 1×1 convolution to generate M feature groups, and the multiple TCN groups are composed of multiple branches with different receptive fields; Step 62, splitting the sequence of length T obtained in step 2 into M substrings of equal length, and then randomly selecting a frame from each substring and concatenating them into a new subsequence; In step 63, the M feature groups obtained in step 61 and the M sample sequences obtained in step 62 are respectively convolved and pooled along the channel dimension, and the results are connected.

8. The method for character gait recognition based on dynamic spatiotemporal graph convolutional neural network according to claim 1, characterized in that: The step 7 comprises: Step 71, the result obtained in step 6 is further modeled in parallel at the joint level and the skeleton level. First, V joint-level features {X1, ..., X V |X i ∈R C×T }Perform average pooling to obtain skeleton-level features S; Step 72: The skeleton-level features S and V joint features X obtained in step 71 are i Perform parallel processing to obtain S * and X i * ; Step 73, the S obtained in step 72 * Merge to X i * middle; Step 74, each TCN instance contains a learning parameter After adaptive joint skeleton fusion, the characteristics of joint i are In step 75, a 1×1 convolution operation is performed on the feature sequence after the adaptive joint skeleton fusion obtained in step 74 to obtain the output of the TCN module.

Images