Whole body three-dimensional pose estimation method based on spatio-temporal dual-path hybrid expert

Abstract

A whole body three-dimensional pose estimation method based on space-time dual-path hybrid expert, comprising data preprocessing, forward diffusion noise adding, constructing a space-time dual-path hybrid expert denoising network, training the space-time dual-path hybrid expert denoising network, and testing the space-time dual-path hybrid expert denoising network. The method obtains the original three-dimensional pose sequence of 133 key points from the H3WB dataset, performs forward diffusion noise adding to generate a noisy three-dimensional pose sequence; constructs a denoising network composed of a space processing branch and a time processing branch in series to respectively realize key point spatial relationship modeling, part-level feature enhancement, continuous frame time sequence dependence modeling and time feature enhancement; the trained denoising network performs denoising prediction on the test data and outputs the whole body three-dimensional pose sequence containing body, face, left hand and right hand key points. Experimental results show that the local alignment error of the method is reduced to 20.9 mm, and the parameter amount is reduced to 11.27M, which is suitable for whole body three-dimensional pose estimation in resource limited environment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically involving a method for full-body 3D pose estimation based on spatiotemporal dual-path hybrid experts. Background Technology

[0002] Three-dimensional whole-body pose estimation recovers the three-dimensional spatial coordinates of key points on the human torso, hands, and face from two-dimensional pose sequences or video detection results. It has broad application prospects in motion capture, virtual reality, human-computer interaction, and medical rehabilitation. Currently, methods based on diffusion probability models, with their multi-hypothesis modeling capabilities, have become an important research direction in the field of three-dimensional pose estimation.

[0003] When faced with full-body 3D pose data containing 133 key points, existing technologies still have the following limitations:

[0004] Because the human torso differs significantly in feature scale from fine-grained parts such as fingers and face, current part modeling often employs physical splitting and multi-stream independent modeling strategies, processing different human body parts separately. While this method of segmenting whole-body data can enhance local modeling capabilities to some extent, it easily leads to redundant model parameters, complex deployment, and technical problems such as disproportionate generated poses or unnatural transitions between parts.

[0005] If a conventional architecture is used directly, the gradients of small-scale, fine-grained features such as fingers are easily masked by main features such as the torso due to the lack of effective feature routing and decoupling mechanisms, resulting in insufficient local fine-grained modeling capabilities.

[0006] When processing large-scale whole-body data, the computational overhead of the existing Transformer architecture increases dramatically with the number of key points, making it difficult to achieve lightweight deployment in low-computing environments.

[0007] In the field of human pose estimation technology, a pressing technical problem is to develop a full-body 3D pose estimation method that does not require physical decomposition, can balance global spatial consistency with fine-grained anatomical modeling capabilities, and has high parameter efficiency. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a full-body 3D pose estimation method based on spatiotemporal dual-path hybrid experts with high lightweightness, strong global consistency and high local key point alignment accuracy.

[0009] The technical solution adopted to solve the above technical problems consists of the following steps:

[0010] (1) Data preprocessing

[0011] A total of 133 keypoints covering the human torso, face, hands, and feet were annotated in the Human3.6M 3D WholeBody full-body pose dataset. Following the principle of subject independence, subjects S1, S5, S6, and S7 were selected as the training set, and subject S8 was selected as the test set. The video sequence was segmented into fixed-length segments to obtain continuous frame pose data. Each frame contains 133 full-body keypoints, and each keypoint corresponds to a keypoint index n. The 133 key points of the whole body are divided into body key points, face key points, left hand key points and right hand key points according to the predefined part index. The body key points correspond to index 0 to 22, and the body key points include the human torso, limbs and feet key points; the face key points correspond to index 23 to 90; the left hand key points correspond to index 91 to 111; and the right hand key points correspond to index 112 to 132.

[0012] (2) Forward diffusion noise

[0013] The 3D pose dataset is used to generate a noisy 3D pose sequence using the following formula. :

[0014]

[0015]

[0016] Where τ is the current diffusion time step, τ∈{1,2,...,T} d}, T d T is the total number of time steps in the diffusion process. d The value can be a finite number of positive integers; The dimensions are T×133×3. This represents the cumulative signal retention coefficient from the first diffusion time step to the τth diffusion time step. Represents the original 3D pose sequence Gaussian noise of the same dimension.

[0017] (3) Construct a spatiotemporal dual-path hybrid expert denoising network

[0018] The spatiotemporal dual-path hybrid expert denoising network consists of a spatial processing branch and a temporal processing branch connected in series.

[0019] The spatial processing branch is composed of a spatial Transformer coding module and a spatial location perception hybrid expert module connected in series.

[0020] The time processing branch is composed of a time Transformer encoding module and a time dynamic mode hybrid expert module connected in series.

[0021] (4) Training a spatiotemporal dual-path hybrid expert denoising network

[0022] 1) Construct the loss function L

[0023]

[0024] In the formula, This represents the pose reconstruction loss function. Let represent the load balancing loss function, and λ represent the weighting coefficient of the load balancing loss. ∈(0,1).

[0025] Construct the pose reconstruction loss function as follows :

[0026]

[0027] In the formula, B represents the training batch size; T represents the number of frames contained in each fixed-length segment, with T taking the value of 27; N represents the number of full-body keypoints in each frame, with N taking the value of 133; b represents the sample index; r represents the frame index in the fixed-length segment; and n represents the keypoint index. This represents the three-dimensional coordinates of the b-th sample, the r-th frame, and the n-th keypoint predicted by the denoising network. Represents the corresponding three-dimensional pose true coordinates; represents The Euclidean distance between the coordinates of two 3D keypoints.

[0028] Construct the load balancing loss function as follows :

[0029]

[0030]

[0031]

[0032] In the formula, This represents the load balancing loss of the average routing probability distribution of spatial experts relative to a uniform target distribution. This represents the load balancing loss of the time-expert average routing probability distribution relative to a uniform target distribution. The total number of experts, The value can be an integer from 2 to 4; Indicates the first The average routing probability corresponding to a spatial expert network; Indicates the first The average routing probability corresponding to a time-based expert network; Indicates the first The target probability corresponding to each expert network. ∈(0,1).

[0033] 2) Training a spatiotemporal dual-path hybrid expert denoising network

[0034] The training set was input into a spatiotemporal dual-path hybrid expert denoising network for training. The training parameters were: 400 training epochs, batch size of 1024, and initial learning rate of 0.00006. An exponential decay strategy was used to adjust the learning rate, and the AdamW optimizer was selected. Training was continued until the loss function L converged.

[0035] 3) Save the model and select the optimal weights

[0036] After each training round, the optimal loss record is updated according to the loss function L. When the current loss function value is better than the historical optimal loss function value, the current denoising network weights are saved, and the saved optimal network weights are used as the pre-trained model for the final backdiffusion denoising inference stage.

[0037] (5) Testing the spatiotemporal dual-path hybrid expert denoising network

[0038] The test set is input into the trained spatiotemporal dual-path hybrid expert denoising network for testing, and the output is the full-body 3D pose with the corresponding indices 0-22 for body key points, 23-90 for facial key points, 91-111 for left hand key points, and 112-132 for right hand key points.

[0039] In step (3) of the present invention, the spatiotemporal dual-path hybrid expert denoising network is constructed by sequentially connecting a part mask layer, a part mask pooling layer, a Top-K sparse gating layer, a weighted broadcasting layer, and a residual fusion layer.

[0040] In step (3) of the present invention, the spatiotemporal dual-path hybrid expert denoising network is constructed by sequentially connecting the time dynamic mode hybrid expert module with the time routing layer, the Top-K sparse gating layer, the gating constraint layer, and the residual fusion layer.

[0041] The method for constructing the part mask layer of the present invention is as follows:

[0042] The part masking layer constructs a priori mask matrix M for human body parts based on predefined key point indices, as follows:

[0043]

[0044] In the formula, P is the number of preset body parts, and P takes the value of 4; N is the number of key points on the whole body, and N takes the value of 133. Represents the prior mask matrix of the said location The p-th row, i.e., the part prior mask vector corresponding to the p-th human body part. for:

[0045]

[0046] The mask value in the p-th row and n-th column of the prior mask matrix M Determine by the following formula:

[0047]

[0048] This represents the set of keypoint indices corresponding to the p-th human body part; the human body parts include the body, face, left hand, and right hand, and their keypoint indices are as follows:

[0049]

[0050] in, , , These correspond to the body, face, left hand, and right hand, respectively.

[0051] The method for constructing the part mask pooling layer of the present invention is as follows:

[0052] The feature vector of the p-th part is determined by the following formula. :

[0053]

[0054] In the formula, , The mask value in the p-th row and n-th column of the part prior mask matrix represents the mask value. This represents the spatial features corresponding to the nth key point; N represents the total number of key points in the body. The preset lower limit threshold of the denominator. .

[0055] Because this invention employs a spatiotemporal dual-path hybrid expert denoising network, while maintaining the overall topology of 133 key points across the entire body, it utilizes a predefined prior mask matrix for feature aggregation and routing allocation for different body parts. This enables key points from different body parts to achieve feature decoupling and body-level collaborative enhancement within a unified whole-body modeling framework, thus balancing global spatial consistency with fine-grained local modeling capabilities. Furthermore, this invention reduces the proportion of irrelevant experts involved in computation by setting a Top-K sparse gating mechanism, a lightweight expert network, and load balancing auxiliary loss, alleviating the problem of unbalanced expert usage and reducing the number of model parameters and computational overhead. Comparative experiments using this invention on the H3WB dataset with existing mainstream methods show that the local alignment error is reduced to 20.9 mm, and the total number of parameters is reduced from 34.91M to 11.27M, a reduction of 67.7%. This improves parameter efficiency while maintaining high 3D pose estimation accuracy, making it suitable for whole-body 3D pose estimation tasks under low-computing-power environments. Attached Figure Description

[0056] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0057] Figure 2 This is a schematic diagram of the spatiotemporal dual-path hybrid expert denoising network.

[0058] Figure 3 yes Figure 2 A schematic diagram of the structure of the mid-space processing branch.

[0059] Figure 4 yes Figure 2 A schematic diagram of the time processing branch.

[0060] Figure 5 yes Figure 3 A schematic diagram of the structure of the hybrid expert module for mid-space location perception.

[0061] Figure 6 yes Figure 4 A schematic diagram of the structure of the hybrid expert module in the time-dynamic mode. Detailed Implementation

[0062] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0063] Example 1

[0064] exist Figure 1 In this embodiment, the whole-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert consists of the following steps:

[0065] (1) Data preprocessing

[0066] A total of 133 keypoints covering the human torso, face, hands, and feet were annotated in the Human3.6M 3D WholeBody full-body pose dataset. Following the principle of subject independence, subjects S1, S5, S6, and S7 were selected as the training set, and subject S8 was selected as the test set. The video sequence was segmented into fixed-length segments to obtain continuous frame pose data. Each frame contains 133 full-body keypoints, and each keypoint corresponds to a keypoint index n. The 133 key points of the whole body are divided into body key points, face key points, left hand key points and right hand key points according to the predefined part index. The body key points correspond to index 0 to 22, and the body key points include the human torso, limbs and feet key points; the face key points correspond to index 23 to 90; the left hand key points correspond to index 91 to 111; and the right hand key points correspond to index 112 to 132.

[0067] (2) Forward diffusion noise

[0068] The 3D pose dataset is used to generate a noisy 3D pose sequence using the following formula. :

[0069]

[0070]

[0071] Where τ is the current diffusion time step, τ∈{1,2,...,T} d}, T d T is the total number of time steps in the diffusion process. d The value can be a finite number of positive integers; The dimensions are T×133×3. This represents the cumulative signal retention coefficient from the first diffusion time step to the τth diffusion time step. Represents the original 3D pose sequence Gaussian noise of the same dimension.

[0072] (3) Construct a spatiotemporal dual-path hybrid expert denoising network

[0073] Figure 2 A schematic diagram of the spatiotemporal dual-path hybrid expert denoising network in this embodiment is given. Figure 2 In this embodiment, the spatiotemporal dual-path hybrid expert denoising network is composed of a spatial processing branch and a temporal processing branch connected in series.

[0074] Figure 3 Given Figure 2 A schematic diagram of the structure of the mid-space processing branch. Figure 3 In this embodiment, the spatial processing branch is composed of a spatial Transformer coding module and a spatial part perception hybrid expert module connected in series.

[0075] Figure 4 Given Figure 2 A schematic diagram of the time processing branch. Figure 4 In this embodiment, the time processing branch is composed of a time Transformer encoding module and a time dynamic mode hybrid expert module connected in series.

[0076] Figure 5 Given Figure 3 A schematic diagram of the structure of the hybrid expert module for mid-space location perception. Figure 5 In this embodiment, the spatial part perception hybrid expert module is composed of a part mask layer, a part mask pooling layer, a Top-K sparse gating layer, a weighted broadcasting layer, and a residual fusion layer connected in series.

[0077] The method for constructing the part mask layer in this embodiment is as follows:

[0078] The part masking layer constructs a priori mask matrix M for human body parts based on predefined key point indices, as follows:

[0079]

[0080] In the formula, P is the number of preset body parts, and P takes the value of 4; N is the number of key points on the whole body, and N takes the value of 133. Represents the prior mask matrix of the said location The p-th row, i.e., the part prior mask vector corresponding to the p-th human body part. for:

[0081]

[0082] The mask value in the p-th row and n-th column of the prior mask matrix M Determine by the following formula:

[0083]

[0084] This represents the set of keypoint indices corresponding to the p-th human body part; the human body parts include the body, face, left hand, and right hand, and their keypoint indices are as follows:

[0085]

[0086] in, , , These correspond to the body, face, left hand, and right hand, respectively.

[0087] The part mask layer is used to construct a part prior mask matrix based on predefined human body part keypoint indices, representing the attribution relationship between each keypoint of the whole body and the body, face, left hand, and right hand. Each row in the part prior mask matrix corresponds to a human body part, and the keypoint position belonging to that human body part is marked by a binary mask value.

[0088] The method for constructing the part mask pooling layer in this embodiment is as follows:

[0089] The feature vector of the p-th part is determined by the following formula. :

[0090]

[0091] In the formula, , The mask value in the p-th row and n-th column of the part prior mask matrix represents the mask value. This represents the spatial features corresponding to the nth key point; N represents the total number of key points in the body. The preset lower limit threshold of the denominator. .

[0092] The part-mask pooling layer is used to perform masked average pooling on the spatial features of keypoints contained in each human body part based on the part-prior mask matrix. The part-mask pooling layer aggregates multiple keypoint features contained in each human body part into a corresponding part feature vector, providing input for part-level dynamic routing.

[0093] The spatial part perception hybrid expert module is used to aggregate spatial features at the part level based on predefined human body part key point indexes, dynamically determine the routing weights of the spatial expert network based on the features of each part, enhance the corresponding key point features using the selected spatial expert network, and perform residual fusion of the spatial enhancement features with the input spatial features.

[0094] Figure 6 Given Figure 4 A schematic diagram of the structure of the hybrid expert module with time-dynamic mode. Figure 6 In this embodiment, the time dynamic mode hybrid expert module is composed of a time routing layer, a Top-K sparse selection layer, a gating constraint layer, and a residual fusion layer connected in series.

[0095] The time-dynamic mode hybrid expert module is used to dynamically calculate the routing weights of the time expert network based on the time features corresponding to each time frame, select the time expert network that matches the current motion mode, adjust the fusion intensity of the time enhancement features through learnable gating parameters, and perform residual fusion with the backbone time features.

[0096] (4) Training a spatiotemporal dual-path hybrid expert denoising network

[0097] 1) Construct the loss function L

[0098]

[0099] In the formula, This represents the pose reconstruction loss function. Let represent the load balancing loss function, and λ represent the weighting coefficient of the load balancing loss. ∈(0,1], and in this embodiment, λ is 0.5.

[0100] Construct the pose reconstruction loss function as follows :

[0101]

[0102] In the formula, B represents the training batch size; T represents the number of frames contained in each fixed-length segment, with T taking the value of 27; N represents the number of full-body keypoints in each frame, with N taking the value of 133; b represents the sample index; r represents the frame index in the fixed-length segment; and n represents the keypoint index. This represents the three-dimensional coordinates of the b-th sample, the r-th frame, and the n-th keypoint predicted by the denoising network. Represents the corresponding three-dimensional pose true coordinates; represents The Euclidean distance between the coordinates of two 3D keypoints;

[0103] Construct the load balancing loss function as follows :

[0104]

[0105]

[0106]

[0107] In the formula, This represents the load balancing loss of the average routing probability distribution of spatial experts relative to a uniform target distribution. This represents the load balancing loss of the time-expert average routing probability distribution relative to a uniform target distribution. The total number of experts, The value is an integer from 2 to 4, as in this embodiment. The value is 3; Indicates the first The average routing probability corresponding to a spatial expert network; Indicates the first The average routing probability corresponding to a time-based expert network; Indicates the first The target probability corresponding to each expert network. ∈(0,1]; This embodiment The value is 0.5.

[0108] 2) Training a spatiotemporal dual-path hybrid expert denoising network

[0109] The training set was input into a spatiotemporal dual-path hybrid expert denoising network for training. The training parameters were: 400 training epochs, batch size of 1024, and initial learning rate of 0.00006. An exponential decay strategy was used to adjust the learning rate, and the AdamW optimizer was selected. Training was continued until the loss function L converged.

[0110] 3) Save the model and select the optimal weights

[0111] After each round of training, the optimal loss record is updated according to the loss function L; when the current loss function value is better than the historical optimal loss function value, the current denoising network weights are saved, and the saved optimal network weights are used as the pre-trained model for the final back-diffusion denoising inference stage.

[0112] (5) Testing the spatiotemporal dual-path hybrid expert denoising network

[0113] The test set is input into the trained spatiotemporal dual-path hybrid expert denoising network for testing, and the output is the full-body 3D pose with the corresponding indices 0-22 for body key points, 23-90 for facial key points, 91-111 for left hand key points, and 112-132 for right hand key points.

[0114] A method for full-body 3D pose estimation based on spatiotemporal dual-path hybrid expert was developed.

[0115] Example 2

[0116] The full-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert in this embodiment consists of the following steps:

[0117] (1) Data preprocessing

[0118] The steps are the same as in Example 1.

[0119] (2) Forward diffusion noise

[0120] The steps are the same as in Example 1.

[0121] (3) Construct a spatiotemporal dual-path hybrid expert denoising network

[0122] The spatiotemporal dual-path hybrid expert denoising network structure in this embodiment is the same as that in Embodiment 1.

[0123] The method for constructing the part mask layer and the part mask pooling layer of the spatiotemporal dual-path hybrid expert denoising network in this embodiment is the same as that in Embodiment 1.

[0124] (4) Training a spatiotemporal dual-path hybrid expert denoising network

[0125] 1) Construct the loss function L

[0126]

[0127] In the formula, This represents the pose reconstruction loss function. Let represent the load balancing loss function, and λ represent the weighting coefficient of the load balancing loss. ∈(0,1], and in this embodiment, λ takes the value of 0.01.

[0128] The pose reconstruction loss function in this embodiment The expression is the same as in Example 1. Pose reconstruction loss function The parameters, variables, and value ranges are the same as in Example 1.

[0129] This embodiment's load balancing loss function The expression is the same as in Example 1. In the load balancing loss function... middle, The total number of experts, The value is an integer from 2 to 4, as in this embodiment. The value is 2; Indicates the first The target probability corresponding to each expert network. ∈(0,1], in this embodiment The value is 0.25. Other parameters, variables, and value ranges are the same as in Example 1.

[0130] 2) Training a spatiotemporal dual-path hybrid expert denoising network

[0131] The steps are the same as in Example 1.

[0132] 3) Save the model and select the optimal weights

[0133] The steps are the same as in Example 1.

[0134] The other steps are the same as in Example 1. A full-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert is thus completed.

[0135] Example 3

[0136] The full-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert in this embodiment consists of the following steps:

[0137] (1) Data preprocessing

[0138] The steps are the same as in Example 1.

[0139] (2) Forward diffusion noise

[0140] The steps are the same as in Example 1.

[0141] (3) Construct a spatiotemporal dual-path hybrid expert denoising network

[0142] The spatiotemporal dual-path hybrid expert denoising network structure in this embodiment is the same as that in Embodiment 1.

[0143] The method for constructing the part mask layer and the part mask pooling layer of the spatiotemporal dual-path hybrid expert denoising network in this embodiment is the same as that in Embodiment 1.

[0144] (4) Training a spatiotemporal dual-path hybrid expert denoising network

[0145] 1) Construct the loss function L

[0146]

[0147] In the formula, This represents the pose reconstruction loss function. Let represent the load balancing loss function, and λ represent the weighting coefficient of the load balancing loss. ∈(0,1], and in this embodiment, λ takes the value of 0.1.

[0148] The pose reconstruction loss function in this embodiment The expression is the same as in Example 1. Pose reconstruction loss function The parameters, variables, and value ranges are the same as in Example 1.

[0149] This embodiment's load balancing loss function The expression is the same as in Example 1. In the load balancing loss function... middle, The total number of experts, The value is an integer from 2 to 4, as in this embodiment. The value is 4; Indicates the first The target probability corresponding to each expert network. ∈(0,1], in this embodiment The value is 1. Other parameters, variables, and their value ranges are the same as in Example 1.

[0150] 2) Training a spatiotemporal dual-path hybrid expert denoising network

[0151] The steps are the same as in Example 1.

[0152] 3) Save the model and select the optimal weights

[0153] The steps are the same as in Example 1.

[0154] The other steps are the same as in Example 1. A full-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert is thus completed.

[0155] To verify the beneficial effects of the present invention, the three-dimensional pose estimation method based on a part-aware hybrid expert diffusion model of Embodiment 1 of the present invention (hereinafter referred to as Embodiment 1) was compared with “Pavllo D, Feichtenhofer C, Grangier D, et al. 3d human pose estimation in video with temporal convolutions and semi-supervised training[C]. Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, 2019: 7753-7762.” (hereinafter referred to as Comparative Experiment 1), “Zheng C, Zhu S, Mendieta M, et al. 3d human pose estimation with spatial and temporal transformers[C]. Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021: 11656-11665.” (hereinafter referred to as Comparative Experiment 2), and “Zhang J, Tu Z, Yang J, et al. Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video[C]. Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, 2022: 13232-13242. (referred to as Comparative Experiment 3) Samet N, Rommel C, Picard D, et al. Pafuse: Part-based diffusion for 3D whole-bodypose estimation[C] / / European Conference on Computer Vision. Cham: SpringerNature Switzerland, 2024: 151-169. (referred to as Comparative Experiment 4) Comparative experiments were conducted. The performance of fine modeling of various parts of the human body in the local coordinate system was evaluated by the average error of the parts. The experimental results are shown in Table 1.

[0156] Table 1. Experimental results of Example 1 method and comparative experiments 1-4

[0157]

[0158] As shown in Table 1, compared with comparative experiments 1-4, the average error of each part in Example 1 was reduced. The PB-MPJPE of Example 1 was reduced by approximately 24.8% compared with comparative experiment 1, approximately 3.2% compared with comparative experiment 2, approximately 0.9% compared with comparative experiment 3, and approximately 1.4% compared with comparative experiment 4.

Claims

1. A method for estimating the whole-body 3D pose based on a spatiotemporal dual-path hybrid expert, characterized in that... It consists of the following steps: (1) Data preprocessing A total of 133 keypoints covering the human torso, face, hands, and feet were annotated in the Human3.6M 3D WholeBody full-body pose dataset. Following the principle of subject independence, subjects S1, S5, S6, and S7 were selected as the training set, and subject S8 was selected as the test set. The video sequence was segmented into fixed-length segments to obtain continuous frame pose data. Each frame contains 133 full-body keypoints, and each keypoint corresponds to a keypoint index n. The 133 key points of the whole body are divided into body key points, face key points, left hand key points and right hand key points according to the predefined part index. The body key points correspond to index 0 to 22, and the body key points include the key points of the human torso, limbs and feet; the face key points correspond to index 23 to 90; the left hand key points correspond to index 91 to 111; and the right hand key points correspond to index 112 to 132. (2) Forward diffusion noise The 3D pose dataset is used to generate a noisy 3D pose sequence using the following formula. : Where τ is the current diffusion time step, τ∈{1,2,...,T} d }, T d T is the total number of time steps in the diffusion process. d The value can be a finite number of positive integers; The dimensions are T×133×3. This represents the cumulative signal retention coefficient from the first diffusion time step to the τth diffusion time step. Represents the original 3D pose sequence Gaussian noise of the same dimension; (3) Construct a spatiotemporal dual-path hybrid expert denoising network The spatiotemporal dual-path hybrid expert denoising network consists of a spatial processing branch and a temporal processing branch connected in series. The spatial processing branch is composed of a spatial Transformer coding module and a spatial location perception hybrid expert module connected in series. The time processing branch is composed of a time Transformer encoding module and a time dynamic mode hybrid expert module connected in series. (4) Training a spatiotemporal dual-path hybrid expert denoising network 1) Construct the loss function L In the formula, This represents the pose reconstruction loss function. Let represent the load balancing loss function, and λ represent the weighting coefficient of the load balancing loss. ∈(0,1]: Construct the pose reconstruction loss function as follows : In the formula, B represents the training batch size; T represents the number of frames contained in each fixed-length segment, with T taking the value of 27; N represents the number of full-body keypoints in each frame, with N taking the value of 133; b represents the sample index; r represents the frame index in the fixed-length segment; and n represents the keypoint index. This represents the three-dimensional coordinates of the b-th sample, the r-th frame, and the n-th keypoint predicted by the denoising network. Represents the corresponding three-dimensional pose true coordinates; represents The Euclidean distance between the coordinates of two 3D keypoints; Construct the load balancing loss function as follows : In the formula, This represents the load balancing loss of the average routing probability distribution of spatial experts relative to a uniform target distribution. This represents the load balancing loss of the time-expert average routing probability distribution relative to a uniform target distribution. The total number of experts, The value can be an integer from 2 to 4; Indicates the first The average routing probability corresponding to a spatial expert network; Indicates the first The average routing probability corresponding to a time-based expert network; Indicates the first The target probability corresponding to each expert network. ∈(0,1]; 2) Training a spatiotemporal dual-path hybrid expert denoising network The training set was input into a spatiotemporal dual-path hybrid expert denoising network for training. The training parameters were: 400 training epochs, batch size of 1024, and initial learning rate of 0.00006. An exponential decay strategy was used to adjust the learning rate, and the AdamW optimizer was selected. Training was continued until the loss function L converged. 3) Save the model and select the optimal weights After each round of training, the optimal loss record is updated according to the loss function L; when the current loss function value is better than the historical optimal loss function value, the current denoising network weights are saved, and the saved optimal network weights are used as the pre-trained model for the final back-diffusion denoising inference stage. (5) Testing the spatiotemporal dual-path hybrid expert denoising network The test set is input into the trained spatiotemporal dual-path hybrid expert denoising network for testing, and the output is the full-body 3D pose with the corresponding indices 0-22 for body key points, 23-90 for facial key points, 91-111 for left hand key points, and 112-132 for right hand key points.

2. The whole-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert as described in claim 1, characterized in that: In step (3), the spatiotemporal dual-path hybrid expert denoising network is constructed by sequentially connecting the part mask layer, the part mask pooling layer, the Top-K sparse gating layer, the weight broadcasting layer, and the residual fusion layer.

3. The whole-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert as described in claim 1, characterized in that: In step (3), the spatiotemporal dual-path hybrid expert denoising network is constructed. The time dynamic mode hybrid expert module is composed of a time routing layer, a Top-K sparse gating layer, a gating constraint layer, and a residual fusion layer connected in series.

4. The whole-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert as described in claim 2, characterized in that... The method for constructing the location mask layer is as follows: The part masking layer constructs a priori mask matrix M for human body parts based on predefined key point indices, as follows: In the formula, P is the number of preset body parts, and P takes the value of 4; N is the number of key points on the whole body, and N takes the value of 133. Represents the prior mask matrix of the said location The p-th row, i.e., the part prior mask vector corresponding to the p-th human body part. for: The mask value in the p-th row and n-th column of the prior mask matrix M Determine by the following formula: This represents the set of keypoint indices corresponding to the p-th human body part; the human body parts include the body, face, left hand, and right hand, and their keypoint indices are as follows: in, , , These correspond to the body, face, left hand, and right hand, respectively.

5. The whole-body 3D pose estimation method based on spatiotemporal dual-path hybrid expert as described in claim 2, characterized in that... The method for constructing the location mask pooling layer is as follows: The feature vector of the p-th part is determined by the following formula. : In the formula, , The mask value in the p-th row and n-th column of the part prior mask matrix represents the mask value. This represents the spatial features corresponding to the nth key point; N represents the total number of key points in the body. The preset lower limit threshold of the denominator. .

Images