Three-dimensional human body posture estimation method based on enhanced topology sensing network

Abstract

The invention discloses a three-dimensional human body posture estimation method based on an enhanced topology sensing network. The method comprises the following steps: acquiring a human body motion capture data set; an enhanced topology sensing network model is constructed, the model comprises a feature embedding block, an enhanced topology sensing module and a regression head which are connected in sequence, the enhanced topology sensing module is repeatedly stacked for five times, and the enhanced topology sensing module comprises a space-time double-branch Transform and a mixed constraint module; training the model by using a data set to obtain a final enhanced topology sensing network model; human body pictures or videos needing to be detected are input into the final enhanced topology sensing network model, three-dimensional coordinates corresponding to all joints are obtained, and three-dimensional human body posture estimation is completed. The generated three-dimensional attitude coordinates are closer to the real situation, and the accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional human body posture estimation method based on an enhanced topology perception network. Background Art

[0002] 3D HPE (3D Human Pose Estimation) is a core research topic in the field of computer vision, which aims to accurately restore the 3D joint positions of the human body from 2D images or videos. In recent years, thanks to the development of deep learning algorithms, the task of 3D human pose estimation has developed rapidly. The topological structure of the human body reflects the spatial and motion relationship between joints and is the essential feature of human posture. Accurately modeling this structure is crucial to improving the accuracy and robustness of the task. Transformer-based models have made significant progress in the task of 3D human pose estimation. These methods use multi-head self-attention mechanisms to model the global dependencies of joints and have advantages in capturing long-range spatiotemporal correlations between joints. However, the self-attention mechanism overemphasizes the global relationship and ignores the local topological connections and structural constraints of the human skeleton. This limitation makes it difficult for the model to fully utilize the prior knowledge of the human skeleton structure, especially in complex postures. Therefore, how to fully consider the topological structure of the human body while modeling the global spatiotemporal correlation of joints, strengthen the modeling of the topological dependencies between human joints, and generate more accurate 3D poses has become a major problem. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a three-dimensional human posture estimation method based on an enhanced topology perception network, which solves the problem of insufficient modeling of the human body topological structure.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A 3D human body posture estimation method based on an enhanced topology-aware network comprises the following steps:

[0006] S1. Obtain a human motion capture dataset.

[0007] S2: construct an enhanced topology-aware network model, and train the model using the data set in step S1 to obtain a final enhanced topology-aware network model.

[0008] S3. Input the human body picture or video to be detected into the final enhanced topology-aware network model to obtain the three-dimensional coordinates corresponding to each joint and complete the three-dimensional human body posture estimation.

[0009] Furthermore, in step S1, the two-dimensional coordinates, three-dimensional coordinates and true values ​​of the joints are obtained from the Human3.6M and MPI-INF-3DHP large-scale motion capture datasets.

[0010] Furthermore, in step S2, the enhanced topology-aware network model includes sequentially connected feature embedding blocks, an enhanced topology-aware module repeatedly stacked 5 times, and a regression head.

[0011] Among them, the enhanced topology perception module includes a spatiotemporal dual-branch Transformer and a hybrid constraint module.

[0012] The data set in step S1 is divided into a training set and a test set according to a ratio of 7:3. The training set is used to train the enhanced topology-aware network model, and the test set is used to test the trained enhanced topology-aware network model to obtain the final enhanced topology-aware network model.

[0013] Furthermore, in step S3, the estimation of the three-dimensional human body posture includes the following steps:

[0014] Use the 2D posture detector to obtain the 2D coordinates of the joints , where T and N represent the number of frames and joints in the sequence respectively, and the number 3 represents the dimension, which includes the horizontal and vertical coordinates and confidence scores of the joints; the two-dimensional coordinates are input into the final enhanced topology-aware network model, and after the feature embedding block, the two-dimensional coordinates are projected to high dimensions to obtain preliminary high-dimensional features , where C represents the dimension size; add a set of tensors, add them to the preliminary high-dimensional features and input them into the enhanced topological perception module, use the spatiotemporal dual-branch Transformer to calculate the spatiotemporal global dependencies between joints, and obtain the fused intermediate features. According to the degrees of freedom of the joints and the limb categories they belong to, use the hybrid constraint module to obtain the local topological constraints of different joints respectively, and obtain the final hybrid topological constraints through adaptive fusion. Use this constraint to perform structured guidance on the fused intermediate features to complete the operation of the enhanced topological perception module;

[0015] The operation in the enhanced topology perception module is repeated 5 times to obtain the enhanced features of the human topology structure. After the regression head, the linear layer is used to predict the final 3D posture coordinates. .

[0016] Furthermore, the shapes of the tensors are N×C and T×1×C respectively and are initialized to 0.

[0017] Furthermore, the intermediate features after fusion include the following:

[0018] The spatiotemporal dual-branch Transformer includes Transformer blocks stacked in a spatial-temporal order and Transformer blocks stacked in a temporal-spatial order; wherein the Transformer blocks stacked in a spatial-temporal order include a spatial encoder and a temporal encoder connected in sequence, and the Transformer blocks stacked in a temporal-spatial order include a temporal encoder and a spatial encoder connected in sequence.

[0019] The added tensor and preliminary high-dimensional features are passed through the spatiotemporal dual-branch Transformer to obtain the corresponding intermediate features. The specific expression is:

[0020] ;

[0021] ;

[0022] Among them, P1 represents the intermediate features obtained by stacking Transformer blocks in space-time order, P2 represents the intermediate features obtained by stacking Transformer blocks in time-space order, TTE represents the time encoder, and STE represents the space encoder.

[0023] Adaptively fuse P1 and P2 to obtain the intermediate features after fusion. The specific expression is:

[0024] ;

[0025] ;

[0026] Among them, W represents the tensor after dimension conversion, FC represents the linear layer, Concat represents the concatenation operation, F represents the intermediate feature after fusion, and W1 and W2 both represent weights.

[0027] Furthermore, the joints are grouped according to their degrees of freedom. The specific expression is:

[0028] ;

[0029] ;

[0030] ;

[0031] Among them, DoF1, DoF2, and DoF3 all represent the degree of freedom grouping of the joints, right_shoulder represents the right shoulder, left_shoulder represents the left shoulder, right_hip represents the right hip, left_hip represents the left hip, right_elbow represents the right elbow, left_elbow represents the left elbow, right_knee represents the right knee, left_knee represents the left knee, right_wrist represents the right wrist, left_wrist represents the left wrist, right_feet represents the right foot, and left_feet represents the left foot.

[0032] The joints are grouped according to the limb category they belong to. The specific expression is:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] in, , , , They represent the right arm, left arm, right leg and left leg of the human body respectively.

[0038] The expression for static joint grouping is:

[0039] ;

[0040] Among them, Static represents static joint grouping, head represents head, neck represents neck, thorax represents chest, spine represents spine, and hip represents hip joint.

[0041] Furthermore, the fused intermediate features are grouped according to the degrees of freedom of the joints and the limb categories to obtain degree of freedom grouping features and limb category grouping features.

[0042] Perform feature dimension conversion on the grouping features to obtain the converted grouping features ,in represents the i-th degree of freedom grouping feature, represents the grouping feature of the jth limb category, Represents static joint grouping features.

[0043] Concatenate the features of each degree of freedom group along the joint dimension to obtain the overall features of the degree of freedom group , after a two-dimensional convolution layer Conv2d with a convolution kernel size of 4×3 D Perform feature extraction to obtain aggregate features containing each degree of freedom grouping ,Will Split along the joint dimension to obtain the local topological constraints of the i-th degree of freedom grouping feature , the specific formula is:

[0044] ;

[0045] ;

[0046] ;

[0047] in, represents the GELU activation function, Split represents the segmentation feature along the joint dimension, represents the first degree of freedom grouping feature, represents the second degree of freedom grouping feature, Represents the third degree of freedom grouping feature.

[0048] Concatenate the features of each limb category group along the joint dimension to obtain the overall features of the limb category group , after a two-dimensional convolution Conv2d with a convolution kernel size of 3×3 p Perform feature extraction to obtain aggregate features containing each limb category grouping ,Will Split along the joint dimension to obtain the local topological constraints of the j-th limb category grouping feature , the specific formula is:

[0049] ;

[0050] ;

[0051] ;

[0052] in, Indicates the first limb category grouping feature, Indicates the second limb category grouping feature, Indicates the third limb category grouping feature, Indicates the fourth limb category grouping feature.

[0053] The static joint grouping features are processed by a two-dimensional convolution Conv2d with a convolution kernel size of 5×3. s Perform feature extraction to obtain static joint grouping aggregation features , and then obtain the local topological constraints of the kth static joint grouping feature , the specific formula is:

[0054] ;

[0055] .

[0056] The local topological constraints of the grouping features of the limb category are combined with the local topological constraints of the grouping features of different degrees of freedom, and based on the weight parameters, the final hybrid topological constraints are obtained. The specific expression is:

[0057] ;

[0058] ;

[0059] ;

[0060] in, It represents the mixed feature of the i-th degree of freedom grouping feature in the j-th limb category grouping feature, Representation and The corresponding learnable parameters are initialized to 0, R represents the final hybrid topology constraint, concat order represents the sequential splicing operation, and Y represents the result after adding the hybrid topology constraint.

[0061] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the three-dimensional human posture estimation method based on an enhanced topology-aware network when executing the computer program.

[0062] Furthermore, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, it executes the three-dimensional human posture estimation method based on an enhanced topology-aware network.

[0063] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0064] The present invention solves the problem of insufficient modeling of human topology structure by designing an enhanced topology-aware network structure and using a hybrid topology constraint module to enhance the network model's learning of human topology structure, thereby generating more accurate three-dimensional posture coordinates. The feasibility and effectiveness of the proposed method are verified on the commonly used Human3.6M and MPI-INF-3DHP datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is an overall implementation flow chart of the present invention.

[0066] Figure 2 Schematic diagram of the joint grouping method in an embodiment of the present invention.

[0067] Figure 3 It is an operation diagram of the hybrid topology constraint module in an embodiment of the present invention.

[0068] Figure 4 Result visualization diagram of the embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0070] To achieve the above object, the present invention proposes a 3D human posture estimation method based on an enhanced topology-aware network. Figure 1 As shown, the specific steps are as follows:

[0071] S1. Obtain the 2D coordinates, 3D coordinates and true values ​​of joints from the Human3.6M and MPI-INF-3DHP large-scale motion capture datasets.

[0072] S2: Build the ETANet model and train it using the data set in step S1 to obtain the final enhanced topology-aware network model. The specific contents are:

[0073] The ETANet (Enhanced Topology-Aware Network) model is built using the PyTorch deep learning framework. First, the environment is configured and data is loaded. The built-in DataLoader method is used to process the data set and divide it into sequences of a specified number of frames (such as 243 frames). Then the network structure is built, and the forward propagation process and loss function of the model are defined according to its specifications to optimize the model.

[0074] The enhanced topology-aware network model consists of sequentially connected feature embedding blocks, an enhanced topology-aware module that is repeatedly stacked 5 times, and a regression head.

[0075] Among them, the enhanced topological perception module includes the spatiotemporal dual-branch Transformer and the MTC (Mixed Topological Constraints) based on the human skeleton topology. MTC is simple, efficient, flexible and easy to use. It is a plug-and-play module. By combining with different Transformer models, it significantly improves the accuracy and robustness of 3D posture estimation. MTC aims to learn the topological prior of the human skeleton, and by constraining the feature expressions of different joints, guide the model to generate more reasonable 3D postures, further reduce errors and improve model performance. The core of MTC is to group and mix according to different logics to obtain stronger topological constraints.

[0076] The data set in step S1 is divided into a training set and a test set according to a ratio of 7:3. The training set is used to train the enhanced topology-aware network model, and the test set is used to test the trained enhanced topology-aware network model. In the test phase, the parameters in the network model are no longer updated. This phase is only used to evaluate the performance of the network model. After the training and testing phases, the final enhanced topology-aware network model is obtained.

[0077] In this embodiment, the data of participants numbered 1, 5, 6, 7, and 8 are used for training, and the data of participants numbered 9 and 11 are used for testing. NVIDIA RTX 4090 GPU is used to accelerate model training.

[0078] In terms of model training, the batch size is set to 4, the initial learning rate is set to 0.0005, and a total of 100 rounds of training are performed, with the learning rate decaying by 0.99 after each round. The AdamW optimizer is selected to optimize the model parameters and the weight decay is specified to be 0.01.

[0079] S3: Input the human body image or video to be detected into the final enhanced topology-aware network model to obtain the three-dimensional coordinates corresponding to each joint and complete the three-dimensional human body posture estimation. The specific contents are:

[0080] Use the 2D posture detector to obtain the 2D coordinates of the joints , where T and N represent the number of frames and joints in the sequence respectively, and the number 3 represents the dimension, which includes the horizontal and vertical coordinates and confidence scores of the joints; the two-dimensional coordinates are input into the final enhanced topology-aware network model, and after the feature embedding block, the two-dimensional coordinates are projected to high dimensions to obtain preliminary high-dimensional features .

[0081] In order to solve the problem that the model lacks the ability to process the order of elements in the sequence, it is also necessary to introduce the position information of each element in the sequence through position embedding, add a set of tensors with shapes of N×C and T×1×C and initialized to 0 for the model to learn, add them to the preliminary high-dimensional features and input them into the enhanced topology perception module, use the spatiotemporal dual-branch Transformer to calculate the spatiotemporal global dependency between joints, and obtain the fused intermediate features. The specific content is:

[0082] The spatiotemporal dual-branch Transformer includes Transformer blocks stacked in a spatial-temporal order and Transformer blocks stacked in a temporal-spatial order; wherein the Transformer blocks stacked in a spatial-temporal order include a spatial encoder and a temporal encoder connected in sequence, and the Transformer blocks stacked in a temporal-spatial order include a temporal encoder and a spatial encoder connected in sequence.

[0083] The two Transformer blocks consider both the correlation between different joints within the same frame and the long-distance dependencies between the same joints across frames.

[0084] The added tensor and preliminary high-dimensional features are passed through the spatiotemporal dual-branch Transformer to obtain the corresponding intermediate features. The specific expression is:

[0085] ;

[0086] ;

[0087] Among them, P1 represents the intermediate features obtained by stacking Transformer blocks in space-time order, P2 represents the intermediate features obtained by stacking Transformer blocks in time-space order, TTE represents the time encoder, and STE represents the space encoder.

[0088] Adaptively fuse P1 and P2, fully integrate temporal and spatial features while keeping the feature dimension unchanged, enhance the model's ability to model the global and local dependencies of 3D postures, and obtain the fused intermediate features. The specific expression is:

[0089] ;

[0090] ;

[0091] Among them, W represents the tensor after dimension conversion; FC represents the linear layer; Concat represents the concatenation operation; F represents the intermediate features after fusion; W1 and W2 both represent weights, which are taken from the two values ​​of the last dimension of W respectively.

[0092] The joint degrees of freedom are defined as the rotational degrees of freedom of a joint relative to its parent node. Since the joints at the end of the limbs have a larger range of motion, higher degrees of freedom, and higher pose estimation errors, more sophisticated modeling is required.

[0093] In order to match the grouping of the limb categories and distinguish it from the static joint grouping, only the 12 joints including the limbs are considered here, and the 12 joints are grouped according to the degrees of freedom of the joints, such as Figure 2 As shown in (a), the specific expression is:

[0094] ;

[0095] ;

[0096] ;

[0097] Among them, DoF1, DoF2, and DoF3 all represent the degree of freedom grouping of the joints, right_shoulder represents the right shoulder, left_shoulder represents the left shoulder, right_hip represents the right hip, left_hip represents the left hip, right_elbow represents the right elbow, left_elbow represents the left elbow, right_knee represents the right knee, left_knee represents the left knee, right_wrist represents the right wrist, left_wrist represents the left wrist, right_feet represents the right foot, and left_feet represents the left foot.

[0098] Each group contains 4 joints with the same degree of freedom. The larger the group subscript, the higher the DoF (Degree of Freedom) of the joints in the group.

[0099] Based on the basic principles of human kinematics and anatomy, the limbs are grouped for 3D human pose estimation to adapt to the motion characteristics of different limbs. Similarly, in order to coordinate with DoF grouping and distinguish static joint grouping, the 12 joints are grouped according to the limb category they belong to, such as Figure 2 As shown in (b), the specific expression is:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, , , , They represent the right arm, left arm, right leg and left leg of the human body respectively.

[0105] Since the spine-related joints are relatively rigid, they are usually used to represent the core structure and key nodes of the trunk, while the limb joint group is relatively flexible. At the same time, in order not to interfere with the mixing of the two groups of degrees of freedom and limbs, the present invention considers the spine-related joints as static joint groups separately, which helps to reduce the movement conflict between different joints, making the movement of the spine more in line with physiological reality, thereby avoiding unnatural posture prediction and improving the accuracy and stability of the model. The expression of static joint grouping is:

[0106] ;

[0107] Among them, Static represents static joint grouping, head represents head, neck represents neck, thorax represents chest, spine represents spine, and hip represents hip joint.

[0108] So far, various grouped joint groups have been obtained. Figure 3 As shown in the figure, according to the degrees of freedom of the joints and the limb categories they belong to, the hybrid constraint module is used to obtain the local topological constraints of different joints respectively, and the final hybrid topological constraints are obtained through adaptive fusion. The constraints are used to structurally guide the fused intermediate features to complete the operation of the enhanced topological perception module; the specific contents are as follows:

[0109] The fused intermediate features are grouped according to the degrees of freedom of the joints and the limb categories to obtain degree of freedom grouping features and limb category grouping features.

[0110] Perform feature dimension conversion on the grouping features to obtain the converted grouping features ,in represents the i-th degree of freedom grouping feature, represents the grouping feature of the jth limb category, Represents static joint grouping features. There is no subscript because there is only one joint group in this grouping method.

[0111] Group each degree of freedom into features Splicing along the joint dimension to obtain the overall characteristics of the degree of freedom grouping , after a two-dimensional convolutional layer Conv2d with a convolution kernel size of 4×3, padding set to (0, 1), stride set to (4, 1), and the number of feature channels kept unchanged DPerform feature extraction so that the convolution can capture the local temporal dependency between three adjacent frames while extracting the features of each group, complementing the global temporal dependency captured by the Transformer, extracting more comprehensive temporal correlation information, and obtaining aggregate features containing each degree of freedom group. ,Will Split along the joint dimension to obtain the local topological constraints of the i-th degree of freedom grouping feature , the specific formula is:

[0112] ;

[0113] ;

[0114] ;

[0115] in, represents the GELU activation function, Split represents the segmentation feature along the joint dimension, represents the first degree of freedom grouping feature, represents the second degree of freedom grouping feature, Represents the third degree of freedom grouping feature.

[0116] Concatenate the features of each limb category group along the joint dimension to obtain the overall features of the limb category group , after a two-dimensional convolution Conv2d with a convolution kernel size of 3×3, padding set to (0, 1), stride set to (3, 1), and keeping the number of feature channels unchanged p Perform feature extraction to obtain aggregate features containing each limb category grouping ,Will Split along the joint dimension to obtain the local topological constraints of the j-th limb category grouping feature , the specific formula is:

[0117] ;

[0118] ;

[0119] ;

[0120] in, Indicates the first limb category grouping feature, Indicates the second limb category grouping feature, Indicates the third limb category grouping feature, Indicates the fourth limb category grouping feature.

[0121] Since there is only one group for static joint grouping, the feature concatenation process is omitted. The static joint grouping features are subjected to a two-dimensional convolution Conv2d with a convolution kernel size of 5×3, padding set to (0, 1), a step size set to 1, and the number of feature channels kept unchanged. s Perform feature extraction to obtain static joint grouping aggregation features , and then obtain the local topological constraints of the kth static joint grouping feature , the specific formula is:

[0122] ;

[0123] .

[0124] After the limbs are grouped, the three joints in each group correspond to three different degrees of freedom. Therefore, the local topological constraints of the grouping features of the limb category are combined with the local topological constraints of the grouping features of different degrees of freedom, and based on the weight parameters, which are initialized to 0, a more refined and flexible final hybrid topological constraint is obtained, which guides the model to more comprehensively model the topological relationship of the human skeleton and predict a more reasonable three-dimensional posture. The specific expression is:

[0125] ;

[0126] ;

[0127] ;

[0128] in, It represents the mixed feature of the i-th degree of freedom grouping feature in the j-th limb category grouping feature, Representation and The corresponding learnable parameters are initialized to 0, R represents the final hybrid topology constraint, concat order represents the sequential splicing operation, and Y represents the result after adding the hybrid topology constraint.

[0129] The topological constraints formed by combining these joint features that contain both joint degree of freedom information and limb category information are added to the input, and residual connections are applied to enhance the model's learning of the human body topology.

[0130] The operation in the enhanced topology perception module is repeated 5 times to obtain the enhanced features of the human topology structure. After the regression head, the linear layer is used to predict the final 3D posture coordinates. .

[0131] Figure 4 is a comparison chart of the visualization results of the method of the present invention and the MixSTE method, where Figure 4(a) is the input two-dimensional human posture graph, Figure 4 (b) is a 3D human body posture graph generated by MixSTE method. Figure 4 (c) is a three-dimensional human body posture diagram generated by the method of the present invention, Figure 4 (d) is the real image of the 3D human posture corresponding to the input 2D posture. It can be seen that the result of the MixSTE method does not restore the posture of the human body's crossed legs very well, and the left shoulder joint of the human body is too high, and the prediction accuracy is poor. The method of the present invention restores the details of the left and right foot joints and the left shoulder joint, and the prediction of the head joint is closer to the true value, and the result is more accurate than MixSTE. Therefore, the method of the present invention can achieve better results in the task of 3D human posture estimation, and the generated 3D posture is more accurate.

[0132] The performance of different 3D human pose estimation methods is compared, as shown in Tables 1 and 2.

[0133] Table 1 Performance comparison of different 3D human pose estimation methods on Human3.6M

[0134]

[0135] In Table 1, UGCN (U-shaped Graph Convolutional Network), PoseFormer (pose transformer), StridedFormer (stride transformer), MHFormer (Multi-Hypothesis Transformer), MixSTE (Mixed Spatio-Temporal Encoder), P-STMO (Pre-trained Spatial Temporal Many-to-One), GLA-GCN (Global-local Adaptive Graph Convolutional Network), POT (Pose-oriented transformer), STCFormer (Spatio-Temporal Criss-cross attention Transformer), KTPFormer (Kinematics and Trajectory Prior Knowledge-Enhanced Transformer), NC-RetNet (Non-Causal RetNet, Non-causal preserved network), TMT (Two-step Mixed-Training strategy) and the method of the present invention are compared.

[0136] The number of frames refers to the number of video frames of input data during training. The average joint error refers to the average Euclidean distance between the predicted three-dimensional joint coordinates and the true coordinates. The joint error (true value) uses the true value of the two-dimensional coordinates instead of the value predicted by the two-dimensional detector as input, and the final predicted three-dimensional coordinates and the true value. The lower the value of the joint error, the better. When the number of frames is uniformly set to 243, compared with other methods in Table 1, the average joint error and joint error (true value) in the method proposed in the present invention have achieved the minimum value, which is 1.2 mm and 1.1 mm respectively. This shows that the posture predicted by the method proposed in the present invention is closer to the actual situation and achieves the best model performance.

[0137] Table 2 Performance comparison of different 3D human pose estimation methods on MPI-INF-3DHP

[0138]

[0139] In Table 2, VideoPose3D (Video Pose 3D), PoseFormer (Pose Transformer), MHFormer (Multi-Hypothesis Transformer), MixSTE (Mixed Spatio-Temporal Encoder), P-STMO (Pre-trained Spatial Temporal Many-to-One), D3DP (Diffusion-based 3D Pose estimation), GLA-GCN (Global-local Adaptive Graph Convolutional Network), PoseFormerV2 (Pose Transformer Version 2), STCFormer (Spatio-Temporal Criss-cross attention Transformer), and MotionAGFormer (Motion Attention-GCNFormer) are compared with the method of the present invention.

[0140] The percentage of correct joints indicates the accuracy of predicting key points within a certain threshold range, and the area under the curve is the area of ​​the correct joint percentage curve, which is used to measure the overall performance of the model under different thresholds. The larger the values ​​of these two indicators, the higher the overall accuracy. Compared with other methods in Table 2, the method proposed in the present invention achieves the highest percentage of correct key points and area under the curve, while the average joint error is the smallest, which is 0.9 mm higher than the best method. This shows that when trained with the MPI-INF-3DHP dataset, the method proposed in the present invention achieves better performance than other methods.

[0141] The embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, the specific steps of the method provided in the embodiment of the present invention are corresponding to the specific steps of the method provided in the embodiment of the present invention, and the processor has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0142] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. It should be noted that when the computer program is executed by the processor, it corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0143] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A three-dimensional human posture estimation method based on an enhanced topology-aware network, characterized in that: include: S1, obtain human motion capture dataset; S2, constructing an enhanced topology-aware network model, and training the model using the data set in step S1 to obtain a final enhanced topology-aware network model; S3. Input the human body picture or video to be detected into the final enhanced topology-aware network model to obtain the three-dimensional coordinates corresponding to each joint and complete the three-dimensional human body posture estimation.

2. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 1, characterized in that: In step S1, the two-dimensional coordinates, three-dimensional coordinates and their true values ​​of the joints are obtained from the Human3.6M and MPI-INF-3DHP large-scale motion capture datasets.

3. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 1, characterized in that: In step S2, the enhanced topology-aware network model includes a feature embedding block connected in sequence, an enhanced topology-aware module and a regression head repeatedly stacked 5 times; Among them, the enhanced topology perception module includes a spatiotemporal dual-branch Transformer and a hybrid constraint module; The data set in step S1 is divided into a training set and a test set according to a ratio of 7:

3. The training set is used to train the enhanced topology-aware network model, and the test set is used to test the trained enhanced topology-aware network model to obtain the final enhanced topology-aware network model.

4. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 3, characterized in that: In step S3, the estimation of the three-dimensional human body posture includes the following: Use the 2D posture detector to obtain the 2D coordinates of the joints , where T and N represent the number of frames and joints in the sequence respectively, and the number 3 represents the dimension, which includes the horizontal and vertical coordinates and confidence scores of the joints; the two-dimensional coordinates are input into the final enhanced topology-aware network model, and after the feature embedding block, the two-dimensional coordinates are projected to high dimensions to obtain preliminary high-dimensional features , where C represents the dimension size; add a set of tensors, add them to the preliminary high-dimensional features and input them into the enhanced topological perception module, use the spatiotemporal dual-branch Transformer to calculate the spatiotemporal global dependencies between joints, and obtain the fused intermediate features. According to the degrees of freedom of the joints and the limb categories they belong to, use the hybrid constraint module to obtain the local topological constraints of different joints respectively, and obtain the final hybrid topological constraints through adaptive fusion. Use this constraint to perform structured guidance on the fused intermediate features to complete the operation of the enhanced topological perception module; The operation in the enhanced topology perception module is repeated 5 times to obtain the enhanced features of the human topology structure. After the regression head, the linear layer is used to predict the final 3D posture coordinates. .

5. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 4, characterized in that: The shapes of the tensors are N×C and T×1×C respectively and are initialized to 0.

6. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 4, characterized in that: The intermediate features after fusion include the following: The spatiotemporal dual-branch Transformer includes Transformer blocks stacked in a spatial-temporal order and Transformer blocks stacked in a temporal-spatial order; wherein the Transformer blocks stacked in a spatial-temporal order include a spatial encoder and a temporal encoder connected in sequence, and the Transformer blocks stacked in a temporal-spatial order include a temporal encoder and a spatial encoder connected in sequence; The added tensor and preliminary high-dimensional features are passed through the spatiotemporal dual-branch Transformer to obtain the corresponding intermediate features. The specific expression is: ； ； Among them, P1 represents the intermediate features obtained by stacking Transformer blocks in space-time order, P2 represents the intermediate features obtained by stacking Transformer blocks in time-space order, TTE represents the time encoder, and STE represents the space encoder; Adaptively fuse P1 and P2 to obtain the intermediate features after fusion. The specific expression is: ； ； Among them, W represents the tensor after dimension conversion, FC represents the linear layer, Concat represents the concatenation operation, F represents the intermediate feature after fusion, and W1 and W2 both represent weights.

7. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 4, characterized in that: The joints are grouped according to their degrees of freedom. The specific expression is: ； ； ； Among them, DoF1, DoF2, and DoF3 all represent the degree of freedom grouping of joints, right_shoulder represents the right shoulder, left_shoulder represents the left shoulder, right_hip represents the right hip, left_hip represents the left hip, right_elbow represents the right elbow, left_elbow represents the left elbow, right_knee represents the right knee, left_knee represents the left knee, right_wrist represents the right wrist, left_wrist represents the left wrist, right_feet represents the right foot, and left_feet represents the left foot; The joints are grouped according to the limb category they belong to. The specific expression is: ； ； ； ； in, , , , They represent the right arm, left arm, right leg and left leg of the human body respectively; The expression for static joint grouping is: ； Among them, Static represents static joint grouping, head represents head, neck represents neck, thorax represents chest, spine represents spine, and hip represents hip joint.

8. The method for 3D human posture estimation based on enhanced topology-aware network according to claim 4, characterized in that: The fused intermediate features are grouped according to the degrees of freedom of the joints and the limb categories to which they belong, and the degrees of freedom grouping features and the limb category grouping features are obtained; Perform feature dimension conversion on the grouping features to obtain the converted grouping features ,in represents the i-th degree of freedom grouping feature, represents the grouping feature of the jth limb category, Represents static joint grouping features; Concatenate the features of each degree of freedom group along the joint dimension to obtain the overall features of the degree of freedom group , after a two-dimensional convolution layer Conv2d with a convolution kernel size of 4×3 D Perform feature extraction to obtain corresponding aggregation features ,Will Split along the joint dimension to obtain the local topological constraints of the i-th degree of freedom grouping feature , the specific formula is: ； ； ； in, represents the GELU activation function, Split represents the segmentation feature along the joint dimension, represents the first degree of freedom grouping feature, represents the second degree of freedom grouping feature, represents the third degree of freedom grouping feature, and Concat represents the concatenation operation; Concatenate the features of each limb category group along the joint dimension to obtain the overall features of the limb category group , after a two-dimensional convolution Conv2d with a convolution kernel size of 3×3 p Perform feature extraction to obtain the corresponding aggregation features ,Will Split along the joint dimension to obtain the local topological constraints of the j-th limb category grouping feature , the specific formula is: ； ； ； in, Indicates the first limb category grouping feature, Indicates the second limb category grouping feature, Indicates the third limb category grouping feature, Indicates the grouping feature of the fourth limb category; The static joint grouping features are processed by a two-dimensional convolution Conv2d with a convolution kernel size of 5×3. s Perform feature extraction to obtain static joint grouping aggregation features , and then obtain the local topological constraints of the kth static joint grouping feature , the specific formula is: ； ； The local topological constraints of the grouping features of the limb category are combined with the local topological constraints of the grouping features of different degrees of freedom, and based on the weight parameters, the final hybrid topological constraints are obtained. The specific expression is: ； ； ； in, It represents the mixed feature of the i-th degree of freedom grouping feature in the j-th limb category grouping feature, Representation and The corresponding learning parameters are initialized to 0, R represents the final hybrid topology constraint, concat order represents the sequential concatenation operation, F is the intermediate feature after fusion, and Y represents the result after adding the hybrid topological constraint.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the three-dimensional human posture estimation method based on the enhanced topology-aware network described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional human posture estimation method based on an enhanced topology-aware network described in any one of claims 1 to 8 is executed.

Images