3D human body posture estimation method for abnormal gaits of cerebral apoplexy based on space-time structure modeling

Abstract

The invention discloses a 3D human body posture estimation method for abnormal gaits of cerebral apoplexy based on space-time structure modeling, and the method comprises the steps: firstly obtaining a gait video of a cerebral apoplexy patient, extracting skeleton point tracks and key points covering joints of the whole body, and obtaining 2D joint coordinates of the video; secondly, for 2D joint coordinates, constructing a multi-dimensional neighborhood feature vector of each joint point to obtain an embedded feature tensor; and then, based on the embedded feature tensor, introducing an attention mechanism for modeling a dynamic structure dependency relationship among the joints, and outputting features after space enhancement. And finally, based on the spatially enhanced features, modeling a relative time relationship between different frames in the frame sequence to obtain comprehensive spatial-temporal features, and performing three-dimensional key point prediction on the comprehensive spatial-temporal features to obtain a 3D human body posture estimation result. According to the invention, the structure expression ability of the model in a complex stroke gait scene is improved. And fine modeling and three-dimensional attitude estimation of abnormal gaits of the stroke patient are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a 3D human posture estimation method for stroke abnormal gait based on space-time structure modeling, which is used for extracting three-dimensional time sequence information of key skeleton points from gait videos of stroke patients. BACKGROUND

[0002] Stroke, commonly known as apoplexy, is a disease caused by damage to the brain blood vessels, which in turn causes local or whole brain tissue damage. It is the most common type of neurological disease. Stroke not only has a high mortality rate, but also often has serious sequelae such as limb dysfunction, language impairment and cognitive impairment, which seriously affect the quality of life of patients. Because stroke patients often have complex functional impairment, the rehabilitation treatment cycle is long, so it is particularly important to assess the rehabilitation progress of patients in a timely and accurate manner during the rehabilitation process.

[0003] The diagnosis and rehabilitation assessment of stroke have long relied on medical imaging technologies such as computed tomography (CT) and magnetic resonance imaging (MRI), as well as clinical scales such as the Fugl-Meyer Motor Function Assessment Scale, the modified Rankin Scale (mRS) and the Gait Abnormality Rating Scale (GARS). Although these methods have advantages in accuracy, they are subject to high equipment costs, complex operation procedures and subjectivity of assessment results, making it difficult to meet the growing needs of efficient and personalized rehabilitation. Gait abnormality is one of the most common motor dysfunctions in stroke patients, characterized by slow walking speed, reduced stride length, asymmetric gait and other features, which directly affects walking stability and energy utilization efficiency. Therefore, gait analysis is gradually becoming an important technical path for rehabilitation assessment. With the development of computer vision and machine learning, skeleton point extraction technology based on video is becoming an important direction for intelligent rehabilitation assessment due to its advantages of convenient data collection, low hardware dependency, high computing efficiency and strong system scalability. Compared with traditional methods, this technology not only improves the objectivity and automation level of assessment, but also provides technical support for dynamic function monitoring and precise intervention for stroke patients, showing broad application prospects in actual clinical rehabilitation scenarios.

[0004] In the tasks of skeleton-based action recognition and pose modeling, the mainstream methods include recurrent neural networks (RNN), graph convolutional networks (GCN) and Transformer architecture, etc. RNN model is good at capturing temporal dynamic changes and can effectively reflect the time continuity of action, but is limited by gradient disappearance and insufficient long-range dependency modeling capability. GCN models the spatial relationship through the topological structure between skeleton joints, has strong spatial dependency capturing ability, and is suitable for expressing the structural characteristics of human skeleton, but the fixed graph structure limits the adaptability to dynamic changes in the movement process, making it difficult to handle the atypical compensatory actions of stroke patients. Transformer can capture long-range spatio-temporal dependencies with global attention mechanism, improve the recognition performance of complex action patterns, and has strong parallel computing advantage, but the lack of structural priori leads to insufficient sensitivity to local action details, and the high computational complexity is not conducive to lightweight deployment. SUMMARY

[0005] Based on the above discussion, the present application proposes a human skeleton extraction method based on spatio-temporal feature hierarchical decoupling network. The human skeleton motion data is extracted by video acquisition of patient gait process. The key point modeling capability is enhanced by multi-scale convolution and dynamic attention mechanism, and the skeleton extraction precision and stability in complex scenes such as abnormal gait and slow motion are improved. The model proposed by the present application optimizes the problems of insufficient adaptability of abnormal skeleton structure and incomplete spatio-temporal dependency capture of existing methods. Traditional methods often cannot effectively model atypical skeleton connection and complex time series changes when dealing with special groups such as stroke patients. To solve this problem, the present model introduces a learnable coupling probability matrix to enhance the perception ability of abnormal connection between joints; at the same time, through relative position embedding and multi-scale hollow convolution module, the modeling capability of spatio-temporal dependency is improved, especially in dealing with the non-periodicity and local pause of stroke gait. In addition, depth separable convolution and structural sparse regularization further optimize the computational efficiency and robustness of the model, which can more accurately reflect the gait abnormal pattern and provide efficient support for clinical rehabilitation evaluation. In order to achieve the above purposes, the method of the present application mainly includes the following steps:

[0006] Step (1). Data acquisition and preprocessing.

[0007] The video used in the present application is a gait video of a stroke patient acquired by standardized acquisition. The shooting device is fixed in the required direction of the gait channel to ensure complete recording of the whole walking process. The human pose estimation tool is used to extract the skeleton point trajectory covering the main joints of the body, and the key points are extracted according to the international standard skeleton topological structure to obtain the 2D joint coordinates of the video. Data preprocessing includes abnormal frame elimination, trajectory smoothing, posture normalization, time series completion and alignment, and feature enhancement.

[0008] Step (2). Input features and multi-scale dynamic feature extraction.

[0009] For the 2D joint coordinates of the video, a multi-dimensional neighborhood feature vector is constructed for each joint point to capture dynamic information in the spatiotemporal neighborhood, resulting in an embedded feature tensor. Assume the gait video has T frames, and each frame contains J joint points. This represents the position of each keypoint j in frame t. The input 2D skeleton keypoint sequence is: X∈{x t,j |t=1,...,T;j=1,...,J}.

[0010] A linear projection mapping is performed on the sequence of key points of the 2D skeleton onto the static feature space to obtain the vector of each single joint and single frame. Z static It is the vector of each single joint and single frame. A 3D tensor stacked by time frames and joint dimensions:

[0011]

[0012] Among them, W s It is a linear transformation weight matrix that maps two-dimensional joint coordinates to the feature space, b s It is the corresponding bias vector.

[0013] To enhance the model's ability to capture spastic movements such as slow dragging steps, multi-scale dilated convolution is introduced. Let the set of convolution dilation rates be: D = {d1, d2, ..., d...} K}, where different expansion rates d i i = 1, 2, ..., K correspond to different temporal receptive fields, and K represents the number of multi-scale branches (i.e. the number of dilation rates).

[0014] For each scale d i ∈D, perform dilated convolution operations independently on each two-dimensional skeleton keypoint sequence in the time dimension:

[0015]

[0016] in, The expansion rate is d i The dilated convolution operation has C output channels. d .

[0017] The outputs of convolutional branches at different scales are concatenated, and finally all joints are combined to obtain multi-scale dynamic features.

[0018]

[0019] The three-dimensional tensor Z staticConcatenate with the extracted dynamic features to obtain

[0020] To fuse static and dynamic information and map the number of channels to the target embedding dimension C, a depthwise separable convolution is used to transform Z... concat Channel dimension (C+K·C) d Projecting onto C, we get

[0021] To enhance the perception of location information in time series, a learnable temporal location code is introduced. With Z fusion By adding element by element, we finally obtain the embedded feature tensor. As input for subsequent spatial modeling modules.

[0022] Step (3). Modeling the spatial structure

[0023] Based on the embedded feature tensor, an attention mechanism is introduced to model the dynamic structural dependencies between joints. First, attention triples are constructed from the embedded feature tensor representation of each joint; the query, key, and value are generated through linear projection.

[0024]

[0025] in These represent the Query, Key, and Value vectors of the h-th attention head, respectively, used to calculate the structural dependency weights between different joints; Z is the training projection matrix for the h-th attention head; l-1 The input features of the previous layer (when l is 1, i.e., the initial input, Z here) l-1 The embedded feature Z output from step 2 ins H represents the number of attention heads, and d represents the number of attention heads. h Dimensions for attention calculation:

[0026] To guide attention to more important joint pairs, a learnable coupling probability matrix P is introduced here. j,k ∈[0,1] J×J P j,k This represents the degree of structural coupling between the j-th and k-th joints. The coupling probability is defined as:

[0027]

[0028] in, σ represents the original unnormalized coupling weights learned by the model; c is a temperature scaling factor, a positive constant used to control the "steepness" of the activation function; σ is the activation function.

[0029] Dynamic attention distribution is adjusted, and the probability matrix is shared in each attention head. The probability matrix is directly applied to the attention weight calculation, using Logit mapping to enhance the importance of key structure pairs. The attention weight of the hth attention head is calculated as follows:

[0030]

[0031] where, is to strengthen the key connection item, and λ is a learnable weight.

[0032] Multi-head attention splicing and residual connection, a total of H attention heads. Each attention head calculates the attention weight and generates the output, and then splices in the channel dimension:

[0033] MSA(Q s ,K s ,V s )=Concat(head1,...,head H )W o

[0034] head h =AttnS h

[0035] where, represents the output of the hth attention head. is a linear mapping matrix, which is used to uniformly map the aggregated features after splicing to the target dimension. The final splicing result is connected in residual connection with the input feature to keep the information flow and gradient stable.

[0036] On this basis, an L-layer stacked encoder structure is constructed. The bottom layer (input layer e = 1) is the embedding feature tensor Z ins output by step 2, that is, the feature sent to the first layer after embedding. In the encoder structure, let the output of the (l-1)th layer be Z -1 (l>1), which represents the feature representation updated by the previous layer attention and feedforward network, containing the context information of the joint in the time and space dimensions. The feature is taken as the input of the lth layer, and the dependency relationship is recalculated through layer normalization and multi-head self-attention module, and a new representation is obtained through residual connection. After the feedforward network and again residual connection, the output of the lth layer Z l is obtained, which has the same dimension as Z l-1 , ensuring the continuity of feature transmission between layers.

[0037] Specifically, first, the output Z l-1Layer normalization is performed to stabilize the training process. The input is fed into the multi-head self-attention module to calculate the correlation between frames in the time dimension. After the random depth layer, part of the path is randomly discarded for regularization to improve the generalization ability of the model. Finally, the output Z l-1 Residual connection is made to prevent gradient disappearance and help the model train deeper.

[0038] The output Z' of the multi-head self-attention sub-layer is l-1 Layer normalization is performed first. Nonlinear feature transformation is performed through the multi-layer perception module to enhance the expression ability. Similarly, path random dropout regularization is performed through the random depth layer. The result is combined with the multi-head attention output Z' l-1 Residual connection is made to ensure stable gradient transmission.

[0039] After the previous L-layer encoder stack, the output features of the last layer are obtained

[0040] Step (4). Modeling temporal dependencies

[0041] The spatial module output The joint embedding features of each frame are flattened into a vector, and the joint feature of the i-th frame is The flattened features of T frames are vertically concatenated to construct the time dimension feature matrix Z F .

[0042] To ensure the sensitivity of the model to the frame sequence, a learnable position encoding matrix E TPos2 is introduced. F Element-wise addition is performed to obtain

[0043] In traditional time modeling, the triplet vector is usually obtained through full connection mapping, ignoring the continuity and local structure within the frame sequence. Stroke gait abnormalities often manifest in rhythm misalignment and low-amplitude motion changes between consecutive frames. Therefore, the model introduces one-dimensional convolution, and the attention triplet generated by the h-th attention head is used to enhance the dynamic modeling capability within the time window. For the l-th time encoder, the input is the output Z l-1 of the previous layer (when l = 1, the bottom layer should input Z int ).

[0044] To model the relative temporal relationship between different frames in the frame sequence, a relative temporal position bias matrix R is introduced, where each element R i,j represents the weight adjustment of the relative temporal position of frame i to frame j, to enhance or suppress the attention between frames of different time intervals, so that the model can better capture short-term motion changes and long-term temporal dependencies. The attention weight of the h-th attention head is The calculation is as follows:

[0045]

[0046] The attention residual connection is consistent with the design of the spatial modeling module, including: layer normalization is applied before the multi-head attention and the feedforward network, and the random dropout sub-layer path is used in the backbone path to realize stable deep residual connection; the feedforward module is composed of two fully connected layers and activation, the output dimensions are consistent, and efficient nonlinear modeling and information integration of cross-frame dynamic features are realized.

[0047] After L-layer encoder stacking, the output feature obtained by the last layer becomes the input of the module for regressing three-dimensional joint positions. t Each row corresponds to the integrated spatio-temporal features of a frame, and each column corresponds to the combined features of the joint channel dimension.

[0048] Step (5). Output prediction of regressing three-dimensional joint positions

[0049] After completing feature encoding and modeling, the output integrated spatio-temporal features Z t are sent to the pose regression module for three-dimensional key point prediction, y is the aggregated center frame joint feature vector, which integrates the context information of the entire time window. To enhance the flexibility of inter-frame modeling, a learnable frame weight is introduced for the time series features of each key point to perform weighted averaging to obtain y.

[0050] The weighted aggregated features are input into the MLP module for coordinate regression, and y is mapped from the feature space to the three-dimensional coordinate space of the joint to obtain

[0051] is reshaped into the three-dimensional coordinate form of each joint, wherein, represents the three-dimensional space coordinate prediction of each key point in the current time center frame, and the three-dimensional coordinates of each key point are represented as

[0052] The standard mean joint position error (MPJPE) loss function is used for training.

[0053] Compared with the existing many skeleton extraction methods, the present application has the following characteristics:

[0054] ​Since spastic gait in stroke patients typically exhibits complex movement characteristics such as slowness, asymmetry, and unstable rhythm, this invention constructs a 3D posture modeling method for stroke gait based on abnormal movement conditions. By extracting 2D keypoint trajectories from input video sequences and combining this with a multi-scale dilated convolution module to extract local dynamic features within different temporal receptive fields, the model's ability to perceive low-frequency anomalies such as dragging and pauses is enhanced. Simultaneously, to improve feature representation efficiency, this invention uses depthwise separable convolution to map and fuse input features, reducing the number of parameters and computational overhead.

[0055] In the spatial modeling module, addressing the insufficient ability of traditional attention mechanisms to model structural anomalies, this invention introduces a learnable coupling probability matrix. Combined with a structural prior modulation mechanism, it enhances the spatial attention's ability to perceive anomalous connections between joints through Logit mapping. This mechanism can adaptively adjust the dependency strength between joint pairs based on training data, improving the model's structural representation ability in complex stroke gait scenarios.

[0056] This invention introduces inter-frame relative temporal offset information into the temporal modeling module to enhance the sensitivity of the attention mechanism to short-term and long-term dependencies in frame sequences. This design can effectively identify non-periodic temporal structures commonly found in stroke gait, such as unilateral delays and local pauses, thereby enabling precise modeling and three-dimensional posture estimation of gait abnormalities in stroke patients, providing structured and quantitative support for the clinical rehabilitation process. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the process of the present invention;

[0058] Figure 2 This is a schematic diagram illustrating the application of the training model in gait skeletal extraction from real stroke patients. Detailed Implementation

[0059] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings: This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0060] like Figure 1 As shown, this embodiment provides a 3D human pose estimation method for abnormal gait in stroke patients based on spatiotemporal structure modeling, applicable to motion detection scenarios based on skeleton sequences, and specifically includes the following steps:

[0061] Step (1). Data collection and preprocessing. The Human3.6M public dataset is used as the experimental basis in the present application, which contains 15 types of daily action data performed by 11 healthy subjects (6 males and 5 females), and the data collection uses a Vicon optical motion capture system with a sampling frequency of 50 Hz. The three-dimensional space motion trajectory is obtained by laying 31 reflective markers on the whole body. In the data preprocessing stage, 17 key joints in the human body structure (including the head, neck, torso, limbs and other core skeletal parts) are selected, and the three-dimensional coordinates are uniformly converted to the local coordinate system with the hip joint as the origin, so as to enhance the translation invariance and structural consistency of the data. The training set is composed of S1, S5, S6, S7 and S8 subject data, and S9 and S11 subject data are used as the test set.

[0062] Step (2). Input feature and multi-scale dynamic feature extraction.

[0063] In the present embodiment, the input data is the two-dimensional human skeleton key point sequence provided in the Human3.6M dataset, and the format is where T is the number of time frames (here, 27 frames), J represents the number of joints in the skeleton (here, 17 joints), and 2 represents the two-dimensional plane coordinates (x, y) of each joint.

[0064] The two-dimensional coordinates of each joint are projected to a high-dimensional feature space through a trainable linear mapping to obtain the original static position feature that is, the vector of each single-joint single-frame, and the dimension C is 32 in the implementation.

[0065] To improve the modeling ability of the model for motion features under different time scales, a multi-scale dilated convolution mechanism is used to locally model the time series of each joint in the input stage. The specific operation is as follows: the historical and future coordinates of each joint j in the time window are extracted, and K multi-scale branches (i.e. K dilated convolutions with different dilation rates d i , i = 1, 2,..., K) are used to extract time dynamic features respectively. Then, the outputs of each branch are spliced in the scale dimension to form a multi-scale dynamic feature representation of each joint, thereby enhancing the model's ability to perceive changes in actions within different time ranges.

[0066]

[0067] In a specific implementation, for each joint time series, a multi-scale dilated convolution branch is used to capture dynamic features at different temporal scales. The multi-scale parameter configuration is dilation = [2, 4, 6, 8], K = 4; each branch contains two layers of one-dimensional dilated convolution, with GELU activation and BatchNorm regularization; the number of output channels of each branch is 32, there are four branches in total, and the total number of channels is 128; then, the original static position features are concatenated with the multi-scale dynamic features , fused through depth separable convolution, and a learnable position encoding vector is added to obtain the final embedded input features

[0068] Z ins = Z fusion + E Tpos1

[0069] This feature representation not only retains the static spatial information of each joint, but also enhances its dynamic semantic ability at multiple time scales. The final output is an embedded feature tensor

[0070] Through the above multi-scale dynamic feature extraction mechanism, the model has good local dynamic modeling capability in the process of encoding the input skeleton sequence.

[0071] Step (3). Model the spatial structure.

[0072] In this embodiment, the spatial structure modeling module aims to deeply encode the joint features of each frame to capture the spatial dependency between joints in the skeleton. The input of this module is the embedded feature sequence obtained in step (2), and the joint relationship within the frame is modeled.

[0073] The model takes the embedded feature tensor of each frame skeleton sequence as input, and each frame joint embedding feature is regarded as a group of token sequences, which is sent to the spatial attention module for modeling. In this module, a multi-head attention mechanism (number of heads H = 8) is used to model the dependency between joints, and a learnable structure modulation matrix is combined to enhance the modeling ability of the skeleton anatomical structure prior.

[0074] The hth attention head generates query vectors, key vectors and value vectors through three groups of learnable linear transformation matrices respectively, as follows:

[0075]

[0076] The output dimension is maintained as C, and the multi-head attention divides the C dimension into H subspaces for independent attention calculation and then splices to form the overall attention output.

[0077] Introducing a learnable coupling probability matrix For any two joints j and k, the connection probability is P j,k , and the unnormalized connection strength learned by the model is The temperature coefficient c in the embodiment is 1.1, and its connection probability is calculated by the following formula:

[0078]

[0079] In implementation, it is introduced into the attention scoring stage, and the learnable weight λ is set to 1.0 initially. The formula for calculating the attention weight is:

[0080]

[0081] There are H attention heads, and the output of the hth attention head is After splicing the H attention heads, a linear transformation is performed through W o to map it to the target dimension:

[0082] MSA(Q s ,K s ,V s )=Concat(head1,...,head H )W o

[0083] Introducing DropPath random depth mechanism in the spatial attention module, combined with standard residual connection and layer normalization operation, can improve the stability of model training and suppress overfitting. The process is as follows:

[0084] First, the spatial feature representation Z l-1 of the previous layer is subjected to layer normalization to enhance numerical stability. The normalized features are fed into the multi-head attention module to obtain the attention output features MSA(LN(Z l-1 )), and the DropPath regularization mechanism is introduced to control the effective probability of the residual path. In the embodiment, r is set to 0.9.

[0085] Where, when in the bottom layer (i.e. l = 1 input layer), the input is the output embedding feature tensor Z ins of step 2.

[0086] Z' l-1 =Z l-1 +DropPath(MSA(LN(Z l-1 ))),l=2,...,L

[0087]

[0088] where r is the probability of reserved path, i.e., the probability that the output of this layer is reserved and scaled; (1-r) is the probability of discarded path, i.e., the probability that the output of this layer is completely discarded.

[0089] Z' l-1 First, layer normalization is performed, then non-linear feature transformation is performed through a multi-layer perception, and random depth regularization is adopted, and finally, Z' l-1 is connected in residual to enhance the expression ability and ensure stable gradient transmission, forming the updated feature of the l-th layer:

[0090] Z l = Z' l-1 + DropPath(FFN(LN(Z' l-1 )), l = 2,..., L

[0091] where Z l-1 and Z l are input and output features respectively; LN(·) represents layer normalization operation; MSA(·) is a multi-head self-attention mechanism, which is used to capture the spatial dependence relationship between joints; DropPath(·) is random depth, which randomly discards part of the sub-layer path to improve the model generalization ability and reduce overfitting; FFN(·) is a feedforward fully connected network, which includes a non-linear activation.

[0092] After L-layer encoder, the output feature Z L of the last layer is obtained, and in the embodiment, L is set to 4, and the output is as follows:

[0093] Z s = LN(Z L )

[0094] Here, is the global perception updated feature, and the spatial enhanced feature is used as the input of the temporal attention.

[0095] Step (4). Modeling time dependence

[0096] In order to model the time dependence of the same joint between different frames, the embodiment independently models the dynamic trajectory of each joint changing over time, and adopts a relative position enhanced convolution attention mechanism.

[0097] First, the spatial output of each frame of Z s is flattened, and each frame is represented as a vector with a length of J·C:

[0098]

[0099] To enhance the model's perception of temporal regularities such as frame order and action rhythm, a learnable temporal position embedding is introduced:

[0100]

[0101] A one-dimensional convolution is used instead of a linear mapping to generate the attention triplets To improve the efficiency of local modeling:

[0102] Q h = Conv Q (Z l-1 )K h = Conv K (Z l-1 )V h = Conv V (Z l-1 )

[0103] Inter-frame relative position encoding is introduced to adjust attention scoring:

[0104]

[0105] The attention residual connection, feedforward module design, and the spatial modeling module of the embodiment remain consistent. After L layers of encoder, the output features of the last layer are obtained As the input of step (5).

[0106] Step (5). Output prediction of regression three-dimensional joint position

[0107] After completing spatial modeling and temporal modeling, this embodiment obtains a global feature representation that integrates spatial and temporal information, and then regresses the corresponding three-dimensional coordinate position of each joint in the current action sequence. A set of learnable frame weights α t is introduced to represent the importance of each frame in the final three-dimensional coordinate prediction, and finally form the full-body prediction result Its calculation method is:

[0108]

[0109] Reshape into the form of three-dimensional coordinates of each joint:

[0110]

[0111] represents the predicted three-dimensional coordinates of the jth joint, is its true value.

[0112] In this scheme, a standard MPJPE loss function is used in the training process. The loss function quantifies the prediction accuracy of the model by calculating the Euclidean distance between the predicted skeleton and the real skeleton of each joint three-dimensional coordinates, and taking the average of all joints. The goal of model training is to minimize MPJPE, so that the predicted skeleton position is as close as possible to the real label, and to improve the accuracy and robustness of spatiotemporal feature modeling.

[0113]

[0114] wherein, is the 3D coordinate prediction result of the jth key point, represents the real three-dimensional coordinates of the jth key point.

[0115] The number of training rounds epoch in this embodiment is set to 100 rounds. We use the following standard indicators to evaluate the performance of the model: mean per joint position error (MPJPE), which measures the average Euclidean distance between the predicted 3D joint position and the real label; protocol-adjusted MPJPE (P-MPJPE) and standardized MPJPE (N-MPJPE), which normalize the error and consider the influence of different body types and scales, ensuring consistency in evaluation between different subjects; mean per joint velocity error (MPJVE), which quantifies the average error of joint velocity and provides the performance of the model in dynamic motion tasks. The training results are shown in Table 1:

[0116] Table 1 Model effect evaluation of Human3.6M dataset

[0117]

[0118]

[0119] The average indicators of the trained model are shown in Table 2:

[0120] Table 2 Average error of the model

[0121] Indicators Mean error (mm) MPJPE 48.1 P-MPJPE 37.4 N-MPJPE 46.3 Velocity (MPJVE) 3.03

[0122] From Table 1 and Table 2, it can be seen that the trained model has good effect in extracting the skeleton, and has high applicability in various scenarios. The cross-action verification results show that the model has excellent spatiotemporal modeling capability in handling abnormal gait similar to stroke features. For example, in actions such as "sitting down" with unstable rhythm and significant scale fluctuations, the structure normalization strategy effectively alleviates the joint drift problem. For low dynamic, variable step frequency scenarios such as "walking" and "walking side by side", the model can stably keep MPJPE less than 37mm and MPJVE less than 3.3mm, indicating that it has spatiotemporal consistency expression capability, and can still achieve accurate three-dimensional pose reconstruction in non-steady state motion.

[0123] The application effect of the model trained in the embodiment in a real scene is as shown in the following table. Figure 2 The example video used is from a real stroke patient video of the rehabilitation department of Zhoushan City Hospital.

[0124] As can be seen intuitively from the video test result, the model performs well in capturing the spastic gait of stroke patients (for example, complex motion features such as slow movement, asymmetric gait, unstable rhythm, etc.), and has high sensitivity and consistency in overall skeleton extraction, and good actual application effect.

Claims

1. A 3D human pose estimation method for stroke abnormal gait based on spatio-temporal structure modeling, characterized in that, The method comprises the following steps: Step 1. Obtain the gait video of a stroke patient, extract the bone point trajectory covering the whole body joint by using a human pose estimation tool, extract the key points, and obtain the 2D joint coordinates of the video; Step 2. For the 2D joint coordinates of the video, a multi-dimensional neighborhood feature vector of each joint is constructed to capture dynamic information in the spatial and temporal neighborhood, and an embedded feature tensor is obtained; Step 3. Based on the embedded feature tensor, an attention mechanism is introduced to model the dynamic structural dependency relationship between joints, and a spatial enhanced feature is output; Step 4. Based on the spatial enhanced feature, a time dimension feature matrix is constructed to model the relative time relationship between different frames in the frame sequence, and a comprehensive spatio-temporal feature is obtained; Step 5. The comprehensive spatio-temporal feature is used for three-dimensional key point prediction to obtain a 3D human pose estimation result.

2. The 3D human pose estimation method for stroke abnormal gait based on spatio-temporal structure modeling according to claim 1, characterized in that, The specific implementation process of step 2 is as follows: Assume that the gait video has T frames, each frame contains J joint points, x t,j represents the position of each joint j in the t frame, the input two-dimensional skeleton key point sequence is: X ∈ {x t,j |t = 1,...,T; j = 1,...,J} Linear projection mapping the two-dimensional skeleton key point sequence to the static feature space to obtain the vector of each single-joint single frame The vector of each single-joint single frame will be stacked in time frame and joint dimension to obtain a three-dimensional tensor The vector of each single-joint single frame will be stacked in time frame and joint dimension to obtain a three-dimensional tensor A multi-scale dilated convolution is introduced, and a set of convolution dilation rates is defined as: D = {d1, d2,..., dK}, where d1 K} and dK i , i = 1, 2,..., K correspond to different dilation rates, and a dilated convolution operation is independently performed on each two-dimensional skeleton key point sequence in the time dimension; the outputs of different scale convolution branches are spliced, and finally all joints are combined to obtain multi-scale dynamic features The three-dimensional tensor Z static is spliced with the dynamic feature to obtain The depth separable convolution is used to map the channel number of Z concat to the target embedding dimension C, to obtain Z fusion ; the learnable time position coding E Tpos1 is introduced and added to Z fusion element by element, to obtain the embedding feature tensor 3. The 3D human pose estimation method for stroke abnormal gait based on spatio-temporal structure modeling according to claim 2, characterized in that, The specific implementation process of step 3 is as follows: First, an attention triple is constructed from the embedded feature tensor representation of each joint, and the query, key and value are generated by linear projection: wherein respectively represent the Query, Key and Value vectors of the h-th attention head for calculating the structural dependency weight between different joints; is the training projection matrix of the h-th attention head; Z l-1 is the input feature of the previous layer, when l is 1, i.e. the initial input, Z l-1 is the embedding feature Z ins outputted in step 2; H is the number of attention heads, d h is the dimension of attention calculation; Introducing a learnable coupling probability matrix P e [0, 1] J×J , P j,k represents the degree of structural coupling between the jth and the kth joint; dynamically adjusting the attention distribution, the probability matrix is shared in each attention head; The probability matrix is directly applied to the attention weight calculation, using Logit mapping, the attention weight of the hth attention head The calculation is as follows: Where λ is a learnable weight; there are H attention heads in total, each of which calculates an attention weight and generates an output, and then the outputs are concatenated in the channel dimension: the concatenation result is connected with the input feature by residual connection; On this basis, an L-layer stacked encoder structure is constructed; the bottom layer is an embedded feature tensor Z ins , that is, the feature sent to the first layer after embedding; in the encoder structure, let the output of the (l-1)th layer be Z l-1 , l>1, which represents the feature representation updated by the attention and feedforward network of the previous layer, which is taken as the input of the lth layer, and after layer normalization and multi-head self-attention module, the dependency relationship is recalculated, and a new representation is obtained through a residual connection; after the feedforward network and the residual connection again, the output Z l of the lth layer is obtained, which has the same dimension as Z l-1 ; after the L-layer encoder stacking, the output feature of the last layer is obtained , which is the feature after space enhancement after normalization output 4. The 3D human pose estimation method for stroke abnormal gait based on spatio-temporal structure modeling according to claim 3, characterized in that, The specific implementation process of step 4 is as follows: Z s All joint embedding features of each frame are flattened into vectors, and the joint features of the i-th frame are... The flattened features of T frames are vertically stitched together to construct the time-dimensional feature matrix Z. F Introducing a learnable positional encoding matrix E TPos2 With Z F Adding element by element, we get For Z int , a one-dimensional convolution is introduced to obtain the attention triplet generated by the hth attention head For the lth layer of time encoder, the input is Z l-1 When l = 1, the bottom layer should input Z int , a relative time position bias is introduced Each element R i,j represents the weight adjustment of the relative time position of frame i to frame j; the attention weight of the hth attention head is calculated as follows: The attention residual connection and the feedforward module are designed in the same way as step 3, including: layer normalization is applied to the multi-head attention and the feedforward network in advance, and the main path adopts a random dropout sub-layer path to realize stable deep residual connection; the feedforward module is composed of two fully connected layers and an activation, and the output dimensions remain the same; After L-layer encoder stack, the output features from the last layer become the input to the next stage of regression to 3D joint positions;Z t Each row corresponds to the spatio-temporal features of a frame, and each column corresponds to the combined features of a joint channel dimension.

5. The 3D human pose estimation method for stroke abnormal gait based on spatio-temporal structure modeling according to claim 4, characterized in that, The specific implementation process of step 5 is as follows: Output comprehensive spatiotemporal features Z t The vectors are fed into the pose regression module for 3D keypoint prediction. y is the aggregated joint feature vector of the center frame, and Z is the comprehensive spatiotemporal feature of each keypoint. t Introducing learnable frame weights A weighted average is performed to obtain y; the weighted aggregated features y are then input into the MLP for coordinate regression, mapping y from the feature space to the three-dimensional coordinate space of the joints. Will The reshaping is done in 3D coordinate form for each joint, and the 3D coordinates of each keypoint are represented as follows: That is, the 3D human pose estimation results are used for training with the average joint position error loss function.

Images