Gcn and transformer based feature fusion 3d human pose estimation method

Abstract

The application discloses a feature fusion 3D human body posture estimation method based on GCN and a transformer, and comprises the following steps: inputting 2D posture key points with confidence into a trained posture estimation model to output corresponding predicted 3D human body posture key points; the processing steps of the posture estimation model are as follows: mapping the 2D posture key points to high dimensions; extracting features from the high-dimensional features in turn through a plurality of hierarchical feature layers to obtain corresponding space-time features; extracting global posture features through a spatial transformer and a time transformer connected in turn; extracting local posture features through a GCN block and a multi-scale transformer connected in turn; fusing the space-time features corresponding to the hierarchical feature layers through a hierarchical feature fusion layer to generate corresponding overall features; and processing the overall features through a regression head to generate corresponding predicted 3D human body posture key points. The application uses GCN to compensate for the poor performance of the transformer in the extraction of spatial features of human body postures, thereby improving the accuracy of 3D human body posture estimation.

Description

Technical Field

[0001] This invention relates to the fields of big data and internet technology, specifically to a 3D human pose estimation method based on feature fusion using GCN and Transformer. Background Technology

[0002] 3D human pose estimation requires a large amount of data to train and optimize models. This data typically includes images or videos of the human body in different poses. Therefore, large datasets provide abundant samples, enabling models to learn more features about human pose, thereby improving the accuracy and robustness of estimations. Furthermore, big data can be used to analyze the patterns of human pose changes, providing a deeper understanding of pose estimation. The internet facilitates the acquisition and sharing of big data. Data resources from around the world can be easily collected via the internet, covering different groups of people, scenes, and poses, providing more comprehensive data support for 3D human pose estimation.

[0003] 3D human pose estimation aims to estimate the 3D pose keypoints of the human body from 2D images or videos. This technology is widely used in biomedicine, sports, human-computer interaction, and other fields, and is one of the research hotspots in computer vision. Due to the widespread use of standard cameras, existing image and video data are mainly stored in RGB format. The lack of depth information in the original images makes it difficult for existing methods to accurately estimate the 3D pose of the human body. Therefore, one of the main challenges of 3D human pose estimation is to upscal the 2D pose sequence estimated from the image to 3D space.

[0004] Accurate 3D human pose estimation precisely reflects the spatiotemporal information of the human body and is crucial for in-depth analysis of pose data and understanding of human state. Recent 3D upscaling models utilize the inherent spatiotemporal features of human motion to improve the accuracy of 3D pose prediction. The Transformer is widely used in pose estimation due to its excellent feature extraction capabilities. After several years of development, Transformer-based methods have matured. However, issues such as high computational and data requirements, and the suitability of applying a global perspective to pose feature extraction, still need to be addressed and optimized. The Transformer's excellent ability to capture global dependencies gives it superior performance in image classification and segmentation tasks. However, this does not mean that the attention mechanism is optimal for 3D human pose estimation. The joints of the human body have unique spatial and temporal relationships, and without sufficient 3D human pose data, the Transformer skeleton struggles to efficiently extract features.

[0005] In recent years, research in the field of 3D human pose estimation has mainly been based on Transformer skeletons. Zheng et al. (from C. Zheng, S. Zhu, M. Mendieta, T. Yang, C. Chen, and Z. Ding, "3D human pose estimation with spatial and temporal transformers," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 11656-11665) proposed Poseformer, which for the first time applied a pure Transformer model to human pose estimation for 2D to 3D upscaling. Their method uses a spatial Transformer module to extract the spatial dependencies between joints of the human body and a temporal Transformer module to extract the temporal dependencies of a joint at different times. Li et al. (from W. Li, H. Liu, H. Tang, P. Wang, and L. Van Gool, "Mhformer: Multi-hypothesis transformer for 3d human pose estimation," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 13147-13156) proposed MHFormer, a Transformer-based multi-hypothesis method. It generates multiple feature hypotheses using features of different depths and cross-encodes the features of these hypotheses to improve prediction accuracy.

[0006] In summary, while existing pure Transformer-based methods have achieved state-of-the-art performance, their improvements over previous works are not significant. This is because the Transformer can only overcome the limitation of lacking inductive bias when trained with sufficient data, and the lack of accurate training data is one of the core problems in 3D human pose estimation, leading to poor accuracy in existing methods. Therefore, improving the accuracy of 3D human pose estimation is a pressing technical problem that needs to be solved. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a feature fusion 3D human pose estimation method based on GCN and Transformer, using GCN to compensate for the poor performance of Transformer in extracting spatial features of human pose, and at the same time establishing an information interaction channel between shallow features and deep features, using shallow features to correct deep features, thereby improving the accuracy of 3D human pose estimation.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] 3D human pose estimation methods based on feature fusion of GCN and Transformer include:

[0010] S1: Obtain 2D pose keypoints with confidence;

[0011] S2: Input the 2D pose key points with confidence into the trained pose estimation model, and output the corresponding predicted 3D human pose key points.

[0012] The processing steps for the attitude estimation model are as follows:

[0013] S201: Mapping 2D pose keypoints to a high dimension through an embedding layer to obtain high-dimensional features;

[0014] S202: High-dimensional features are extracted sequentially through multiple layers of feature layers to obtain the corresponding spatiotemporal features;

[0015] The feature layer includes global feature branches and local feature branches. The spatiotemporal features of the feature layer are generated by dynamically weighting the global pose features extracted by the global feature branches and the local pose features extracted by the local feature branches.

[0016] The global feature branch extracts global pose features by sequentially connecting a spatial Transformer and a temporal Transformer.

[0017] The local feature branch extracts local pose features through sequentially connected GCN blocks and multi-scale Transformers;

[0018] S203: The spatiotemporal features corresponding to each hierarchical feature layer are fused through the hierarchical feature fusion layer to generate the corresponding overall features;

[0019] S204: The overall features are processed by the regression head to generate corresponding predicted 3D human pose key points;

[0020] S3: Use the output predicted 3D human pose keypoints as the corresponding 3D human pose estimation results.

[0021] Preferably, in step S202, both the spatial Transformer and the temporal Transformer include a first normalized layer, an attention network layer, a second normalized layer, and a multilayer perceptron connected sequentially from end to end.

[0022] The input to the first normalized layer is the input to either the spatial or temporal Transformer; the input to the first normalized layer and the output of the attention network layer are residually connected and then used as the input to the second normalized layer; the input to the second normalized layer and the output of the multilayer perceptron are residually connected and then used as the output to either the spatial or temporal Transformer.

[0023] Preferably, 1) the attention network layer uses a multi-head self-attention mechanism, expressed by the following formula:

[0024] MSA=Concat(head1,...,headn)W O ;

[0025] head i =Attntion(Q i K i V i );

[0026] In the formula: Concat represents a chain operation; head i Indicates the i-th attention head;

[0027] Each attention head uses a scaled dot product attention mechanism to calculate the attention, as shown in the following formula:

[0028]

[0029] In the formula: W O d m A linear projection layer of dimensionality; n represents the number of key points in a spatial Transformer and the number of time frames in a temporal Transformer; d m This represents the number of feature channels, i.e., the dimension.

[0030] 2) The calculation formula for a multilayer perceptron is as follows:

[0031] MLP(x) = σ(xW1+b1)W2+b2;

[0032] In the formula: σ(·) represents the activation function GeLU: and These represent the fully connected layers of a multilayer perceptron; and Representing fully connected layers and The bias; MLP(x) represents the multilayer perceptron.

[0033] Preferably, in step S202, the GCN block includes a first layer normalization layer, a GCN network layer, a second layer normalization layer, and a multilayer perceptron connected sequentially from end to end;

[0034] The input of the first normalized layer is the input of the GCN block; the input of the first normalized layer and the output of the GCN network layer are residually connected and used as the input of the second normalized layer; the input of the second normalized layer and the output of the multilayer perceptron are residually connected and used as the output of the GCN block.

[0035] The preferred feature extraction formula for GCN blocks is as follows:

[0036] x′=GCN(LayerNorm(x))+x;

[0037] O=MLP(LayerNorm(x′))+x′;

[0038] The feature extraction formula for the GCN network layer is as follows:

[0039]

[0040] In the formula: BatchNorm represents the standardization process; Degree matrix; Represents the adjacency matrix;

[0041] The inter-layer feature propagation formula for the GCN network is as follows:

[0042]

[0043] In the formula: GCN represents the GCN network layer; LayerNorm represents the layer normalization layer; MLP represents the multilayer perceptron; W1 (l) , This represents the linear projection layer between layers; x is the inter-layer input; O is the inter-layer output; It is the adjacency matrix of an undirected graph, I N It is the identity matrix; W (l) H represents the linear projection layer between layers; σ(·) represents the ReLU activation function; (l) H is the feature matrix of each layer's input; (0) Features of the initial input; and Let i and j represent the elements in the i-th and j-th columns of the degree matrix and adjacency matrix, respectively.

[0044] Preferably, in step S202, the multi-scale Transformer includes a Reshape function, a first normalization layer, an attention network layer, a second normalization layer, a multilayer perceptron, and a Restore function connected sequentially from end to end.

[0045] The input to the Reshape function is the input to the multi-scale Transformer; the input to the first normalized layer and the output of the attention network layer are residually connected and then used as the input to the second normalized layer; the input to the second normalized layer and the output of the multilayer perceptron are residually connected and then used as the input to the Restore function; the output of the Restore function is used as the output of the multi-scale Transformer.

[0046] Preferably, the calculation formula for the Reshape function is as follows:

[0047]

[0048] In the formula: K represents the high-dimensional pose feature; S represents the scale, which gradually decreases to 1 as the depth increases.

[0049] Preferably, in step S202, the calculation formula for dynamically weighting the local pose features and the global pose features is as follows:

[0050] K i =K l ·W l +K g ·W g ;

[0051] W l W g =softmax(concat(K) l ,Kg)W);

[0052] In the formula: This indicates that the input is 2d. m A linear layer with an output of 2; W l W g K represents the weights of local and global features, respectively. i This represents the output of the i-th feature layer.

[0053] Preferably, in step S203, the hierarchical feature fusion layer performs feature fusion in one or more of the following ways:

[0054] 1) Weighted average pooling

[0055] The formula is expressed as:

[0056]

[0057] In the formula: K represents the overall characteristic; w i K represents the weight of the i-th layer; i The i-th layer represents the output feature; L represents the set of selected feature layers; w i These are trainable parameters;

[0058] 2) Softmax

[0059] The formula is expressed as:

[0060]

[0061] In the formula: w′ i This represents the weighted value after processing by the softmax() function;

[0062] 3) Transformer

[0063] The formula is expressed as:

[0064]

[0065]

[0066] In the formula: MSA represents the multi-head self-attention mechanism in Transformer;

[0067] 4) Average pooling

[0068] The formula is expressed as:

[0069]

[0070] Preferably, the loss function for training the pose estimation model is as follows:

[0071]

[0072] In the formula: This represents the average joint position error; This represents the normalized average joint position error. This represents the average joint velocity error.

[0073] Compared with existing technologies, the feature fusion 3D human pose estimation method based on GCN and Transformer in this invention has the following advantages:

[0074] In this invention, the feature layer of the pose estimation model extracts local pose features through GCN blocks and multi-scale Transformers. GCN (Graph Convolutional Network) incorporates prior graph structures during inference, allowing it to include inductive biases not found in Transformers. This means that with only a small amount of labeled information, GCN performs exceptionally well. Since the human pose skeleton itself is graph-structured data, it aligns perfectly with GCN. Simultaneously, multi-scale Transformers capture short-term correlations in human pose, enabling the capture of local spatial features using GCN and local temporal features using multi-scale Transformers. This allows GCN to compensate for the Transformer's poor performance in extracting spatial features of human pose. Furthermore, combining the excellent performance of GCN in graph inference with the Transformer's powerful global dependency capture capability improves the accuracy of pose estimation. Moreover, fusing the local pose features extracted by GCN blocks and multi-scale Transformers with the global pose features extracted by traditional (spatial / temporal) Transformers achieves even more accurate pose estimation, thereby improving the accuracy of 3D human pose estimation.

[0075] In this invention, the pose estimation model extracts spatiotemporal features through multiple layers of feature layers and fuses these features to generate overall features. The applicant discovered that shallow and deep features are equally important in pose estimation models, but existing solutions only use output layer features for pose estimation, leading to poor accuracy in 3D human pose estimation. Therefore, this invention fuses shallow, low-dimensional raw features with deep, high-dimensional abstract features as output, replacing the original single output layer features. This uses a hierarchical feature fusion module to establish an information exchange channel between shallow and deep features, using shallow features to correct deep features, thereby better learning the underlying 3D pose structure. Simultaneously, the weights assigned to each layer are dynamically updated during training. This hierarchical feature fusion method reuses the model's shallow parameters, improving parameter utilization and further enhancing the accuracy of 3D human pose estimation. Attached Figure Description

[0076] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0077] Figure 1 This is a logic block diagram of the 3D human pose estimation method of the present invention;

[0078] Figure 2 This is a diagram showing the overall and local network structure of the posture evaluation model;

[0079] Figure 3 (a) shows a commonly used human pose estimation skeleton and its corresponding adjacency matrix; Figure 3 (b) is a multi-scale Transformer;

[0080] Figure 4 This is a graph showing how the weights of each layer change with the number of training rounds.

[0081] Figure 5 Weight comparison of local and global features;

[0082] Figure 6 Comparison of 3D human pose estimation results. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0084] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not mean that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0085] The following detailed explanation illustrates the specific implementation methods:

[0086] Example:

[0087] To better introduce the technical solution of the present invention, this embodiment first analyzes the prior art as follows:

[0088] 1) 3D human pose estimation

[0089] Based on the methodology, existing 3D human pose estimation methods can be divided into end-to-end methods and 2D-to-3D upscaling methods. End-to-end methods directly estimate 3D human pose from raw RGB images and videos. The advantage of this method is that it avoids potential errors and complexities associated with intermediate steps, but it is also difficult to train and optimize due to its complexity. 2D-to-3D upscaling methods, on the other hand, first estimate the 2D pose using existing 2D pose estimation algorithms, and then upscale the 2D pose to 3D. This method has higher interpretability. Furthermore, since there is a mapping relationship between the 2D and 3D human pose data in the camera coordinate system, existing methods can improve the model's inference performance through this constraint. In this invention, we use the 2D pose of monocular video as input and employ the 2D-to-3D upscaling method.

[0090] 2) Transformer-based methods

[0091] The Transformer was initially proposed by Vaswani et al. (from A. Vaswani et al., "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017) as a natural language processing model. Its potential was quickly recognized in the field of computer vision and applied to various tasks. Zheng et al. (from C. Zheng, S. Zhu, M. Mendieta, T. Yang, C. Chen, and Z. Ding, "3d human pose estimation with spatial and temporal transformers," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 11656-11665) first applied a pure Transformer network to the field of 3D human pose estimation, demonstrating the Transformer's enormous potential in this area. The Transformer has been widely used in subsequent work. Shan et al. (from W. Shan, Z. Liu, X. Zhang, S. Wang, S. Ma, and W. Gao, "P-stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation," in European Conference on Computer Vision, 2022: Springer, pp. 461-478) proposed a pre-trained spatial temporal many-to-one (P-STMO) model for 2D to 3D human pose estimation tasks. It enhances the encoder's ability to capture spatial and temporal dependencies through pre-training. The pre-trained encoder is loaded into the STMO model and fine-tuned to improve prediction performance.Einfalt et al. (from M. Einfalt, K. Ludwig, and R. Lienhart, "Uplift and upsample: Efficient 3d human pose estimation with uplifting transformers," in Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2023, pp. 2903-2913.) proposed a Transformer-based pose enhancement scheme that operates on temporally sparse 2D pose sequences by masking frame modeling, yet still generates temporally dense 3D pose estimates. This method decouples the sampling rate of the input 2D pose from the target frame rate of the video and reduces the overall computational complexity. Zhao et al. (from Q. Zhao, C. Zheng, M. Liu, P. Wang, and C. Chen, "PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8877-8886.) proposed PoseFormerV2, which effectively expands the receptive field and improves robustness to noise by utilizing a compact representation of long skeleton sequences in the frequency domain. This method effectively integrates features from the time and frequency domains, achieving a better speed-accuracy tradeoff than its predecessor. Tang et al. (from Z. Tang, Z. Qiu, Y. Hao, R. Hong, and T. Yao, "3D Human Pose Estimation With Spatio-Temporal Criss-Cross Attention," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 4790-4799) proposed a novel Spatio-Temporal Criss-Cross Attention (STC) block. It first uniformly partitions its input features along the channel dimension into two partitions, performing spatial and temporal attention on each partition. Then, STC models the interactions between joints within the same frame and between joints within the same trajectory simultaneously by concatenating the outputs of the attention layers.These methods, using Transformer as a framework, address a wide variety of problems in the field of human pose estimation.

[0092] 3) GCN-based methods

[0093] GCN-based methods have achieved significant success in action recognition. However, in human pose estimation, they struggle to demonstrate an advantage in accuracy compared to Transformers. Ci et al. (from H.Ci, C.Wang, X.Ma, and Y.Wang, "Optimizing network structure for 3d human pose estimation," in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 2262-2271) proposed a Locally Connected Network (LCN) to enhance the representational capabilities of GCN in pose estimation. This network is naturally implemented using this general simulation. Choi et al. (from H. Choi, G. Moon, and KMLee, "Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose," in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, 2020: Springer, pp. 769-787) proposed Pose2Mesh, a novel system based on graph convolutional neural networks (GraphCNN) that can directly estimate the 3D coordinates of human mesh vertices from a 2D human pose. The 2D human pose as input provides basic human joint information and has relatively uniform geometric properties between the two domains. Wang et al. (from J. Wang, S. Yan, Y. Xiong, and D. Lin, "Motion-guided 3D pose estimation from videos," in European Conference on Computer Vision, 2020: Springer, pp. 764-780) designed a novel graph convolutional network architecture, U-shaped GCN (UGCN). It captures both short-term and long-term motion information to fully utilize the additional supervision of motion loss.Yu et al. (from BXYu, Z. Zhang, Y. Liu, S.-h. Zhong, Y. Liu, and CWChen, "Gla-gcn: global-local adaptive graph convolutional network for 3D human pose estimation from monocular video," in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 8818-8829) proposed a simple yet effective model called Global-Local Adaptive Graph Convolutional Network (GLA-GCN). GLA-GCN globally models the spatiotemporal structure through graph representations and backtracks local joint features through individually connected layers for 3D human pose estimation. While these methods maintain a relatively light memory load, their ability to represent global structures is difficult to match that of the Transformer.

[0094] Based on the above analysis, this embodiment discloses a feature fusion 3D human pose estimation method based on GCN and Transformer.

[0095] like Figure 1 As shown, the feature fusion 3D human pose estimation method based on GCN and Transformer includes:

[0096] S1: Obtain 2D pose keypoints with confidence;

[0097] In this embodiment, 2D pose key points with confidence levels are obtained using existing mature methods. 2D pose key points with confidence levels. Where F represents the number of input image frames, N represents the number of pose keypoints in each frame, and 3 represents the 2D data and confidence level contained in each keypoint.

[0098] S2: Input the 2D pose key points with confidence into the trained pose estimation model, and output the corresponding predicted 3D human pose key points.

[0099] like Figure 2 As shown in (a), the processing steps of the attitude estimation model are as follows:

[0100] S201: The 2D pose keypoints are mapped to a higher dimension (a higher dimension than the current dimension) through the embedding layer to obtain high-dimensional features;

[0101] In this embodiment, the 3D 2D pose keypoints are mapped to C-dimensionality through an embedding layer to obtain high-dimensional features.

[0102] S202: High-dimensional features are extracted sequentially through multiple layers of feature layers to obtain the corresponding spatiotemporal features;

[0103] like Figure 2 As shown in (b), the feature layer includes global feature branches and local feature branches, and the spatiotemporal features of the feature layer are generated by dynamically weighting the global pose features extracted by the global feature branches and the local pose features extracted by the local feature branches.

[0104] The global feature branch extracts global pose features by sequentially connecting a spatial Transformer and a temporal Transformer.

[0105] The local feature branch extracts local pose features through sequentially connected GCN (Graph Convolutional Network) blocks and multi-scale Transformer;

[0106] S203: The spatiotemporal features corresponding to each hierarchical feature layer are fused through the hierarchical feature fusion layer to generate the corresponding overall features;

[0107] S204: Process the overall features using a regression head to generate corresponding predicted 3D human pose keypoints; where the predicted 3D human pose keypoints are...

[0108] In this embodiment, the regression head is a well-established concept. In deep learning, a regression head is a part of a neural network structure used to perform regression tasks. The goal of a regression task is to predict continuous values ​​based on input data, such as predicting house prices or stock prices. The regression head is typically the last layer or a few layers of a neural network model, responsible for mapping the features learned by the model to a continuous numerical space.

[0109] Below is a typical neural network architecture for a simple regression task, including a regression head:

[0110] Input Layer: Features that accept input data.

[0111] Hidden Layers: Composed of multiple hidden layers, each containing multiple neurons, used to learn complex features from the input data.

[0112] Activation Functions: After the hidden layer, activation functions are often used to introduce non-linearity, such as ReLU (Rectified Linear Unit).

[0113] Output Layer: Contains the regression head. This layer typically has only one neuron because the output of a regression task is a continuous value. No activation function is used in this layer because we want the output to be the original continuous value.

[0114] At the output layer, the neural network produces a continuous numerical value representing the model's prediction for a given input. During training, the model adjusts its parameters to minimize the difference between the predicted and actual values, typically measured using mean squared error or other appropriate loss functions.

[0115] The role of the regression head is to map the features learned by the neural network onto a continuous numerical space, thereby producing the final prediction result. Through the training process, the neural network can learn patterns and features in the input data, enabling it to make accurate predictions on unseen data.

[0116] S3: Use the output predicted 3D human pose keypoints as the corresponding 3D human pose estimation results.

[0117] In this invention, the feature layer of the pose estimation model extracts local pose features through GCN blocks and multi-scale Transformers. GCN (Graph Convolutional Network) incorporates prior graph structures during inference, allowing it to include inductive biases not found in Transformers. This means that with only a small amount of labeled information, GCN performs exceptionally well. Since the human pose skeleton itself is graph-structured data, it aligns perfectly with GCN. Simultaneously, multi-scale Transformers capture short-term correlations in human pose, enabling the capture of local spatial features using GCN and local temporal features using multi-scale Transformers. This allows GCN to compensate for the Transformer's poor performance in extracting spatial features of human pose. Furthermore, combining the excellent performance of GCN in graph inference with the Transformer's powerful global dependency capture capability improves the accuracy of pose estimation. Moreover, fusing the local pose features extracted by GCN blocks and multi-scale Transformers with the global pose features extracted by traditional (spatial / temporal) Transformers achieves even more accurate pose estimation, thereby improving the accuracy of 3D human pose estimation.

[0118] In this invention, the pose estimation model extracts spatiotemporal features through multiple layers of feature layers and fuses these features to generate overall features. The applicant discovered that shallow and deep features are equally important in pose estimation models, but existing solutions only use output layer features for pose estimation, leading to poor accuracy in 3D human pose estimation. Therefore, this invention fuses shallow, low-dimensional raw features with deep, high-dimensional abstract features as output, replacing the original single output layer features. This uses a hierarchical feature fusion module to establish an information exchange channel between shallow and deep features, using shallow features to correct deep features, thereby better learning the underlying 3D pose structure. Simultaneously, the weights assigned to each layer are dynamically updated during training. This hierarchical feature fusion method reuses the model's shallow parameters, improving parameter utilization and further enhancing the accuracy of 3D human pose estimation.

[0119] In this embodiment, our model was tested on the Human3.6M and MPI-INF-3DHP datasets. The experimental results show that our method outperforms existing mainstream methods and achieves the best performance on both datasets with errors of 37.7 mm and 16.4 mm, respectively, under the MPJPE standard.

[0120] like Figure 2 As shown in (c), both the spatial Transformer and the temporal Transformer include a first layer normalization (LN), an attention network layer, a second layer normalization, and a multi-layer perceptron (MLP) connected sequentially from end to end.

[0121] The input to the first normalized layer is the input to either the spatial or temporal Transformer; the input to the first normalized layer and the output of the attention network layer are residually connected and then used as the input to the second normalized layer; the input to the second normalized layer and the output of the multilayer perceptron are residually connected and then used as the output to either the spatial or temporal Transformer.

[0122] In this embodiment, the global pose feature extraction uses traditional Transformer structures. These Transformers all employ the traditional scaled dot product attention mechanism, and their attention calculation formulas are as follows:

[0123]

[0124] in n represents the number of key points in a spatial Transformer and the number of time frames in a temporal Transformer; d m This represents the number of feature channels, i.e., the dimension.

[0125] Transformer's self-attention mechanism can efficiently capture global dependencies of features. It treats different tokens with equal importance, which is very helpful for us to capture the correlation between different key points and different frames.

[0126] To ensure that information from different subspaces is collectively focused, the attention network layer uses a multi-head self-attention mechanism, as expressed in the following formula:

[0127] MSA = Concat(head1,...,head) n W O ;

[0128] head i =Attention(Q) i ,K i V i );

[0129] Among them W O d m A linear projection layer in dimensionality; Concat represents the concatenation operation; head i This represents the i-th attention head.

[0130] In this embodiment, MSA is an important component of the Transformer model, used to capture the dependencies between positions in the input sequence. Specifically, MSA allows the model to compare the representation of each position with the representations of other positions, enabling the model to pay attention to the correlations between different positions when processing the input sequence. Multi-Head Self-Attention refers to projecting the input vector onto multiple attention heads to learn multiple different representations in parallel. In the multi-head self-attention layer of the Transformer model, the outputs of multiple attention heads are usually concatenated. This concatenation operation is to preserve and combine information from different attention representations among multiple attention heads, thereby enriching the model's ability to represent the input sequence. In the multi-head self-attention mechanism, the model typically uses multiple attention heads to learn multiple different attention representations. iIn this context, "i" represents the i-th attention head, which typically indicates a specific attention pattern or representation learned by this head. Multi-head self-attention allows the model to learn multiple different attention representations in parallel, with each head potentially focusing on different features or relationships in the sequence.

[0131] Following this, a Multilayer Perceptron (MLP) is used. An MLP is a basic feedforward neural network model consisting of at least three layers of neurons: an input layer, at least one or more hidden layers, and an output layer. Each layer of an MLP consists of multiple neurons, which are fully connected between adjacent layers. The hidden layers of an MLP introduce non-linearity through non-linear activation functions (such as ReLU, sigmoid, or tanh), enabling the network to learn non-linear patterns and features. The input layer of an MLP receives the raw feature data, which is then combined and transformed by the hidden layers to ultimately produce the prediction results of the output layer.

[0132] The calculation formula for a multilayer perceptron is as follows:

[0133] MLP(x) = σ(xW1+b1)W2+b2;

[0134] In the formula: σ(·) represents the activation function GeLU; and These represent the fully connected layers of a multilayer perceptron; and Representing fully connected layers and The bias.

[0135] In practice, Graph Convolutional Network (GCN) is a type of neural network that directly processes graph structures. Its inter-layer feature propagation formula is as follows:

[0136]

[0137] in It is the adjacency matrix of an undirected graph, I N It is the identity matrix; W (l) H represents the linear projection layer between layers; σ(·) represents the ReLU activation function; (l) H is the feature matrix of each layer's input; (0) Features of the initial input; and Let i and j represent the elements in the i-th and j-th columns of the degree matrix and adjacency matrix, respectively.

[0138] Unlike Transformer, GCN has a strong ability to extract local features because it introduces a prior graph structure into the network. GCN inherently possesses a very strong inductive bias. In other computer vision tasks, it achieves excellent recognition results with only simple training. This is extremely well-suited for human pose estimation because the connections between human keypoints form a deterministic graph structure, and the range of motion of each joint in the human body is limited. In limited 3D human pose datasets, GCN is better suited than Transformer for extracting spatial pose features.

[0139] This invention improves upon the GCN block structure using MetaFormer, enabling the GCN block to extract features more efficiently. For example... Figure 2 As shown in (c), the GCN block includes a first normalized layer, a GCN network layer, a second normalized layer, and a multilayer perceptron connected sequentially from end to end;

[0140] The input of the first normalized layer is the input of the GCN block; the input of the first normalized layer and the output of the GCN network layer are residually connected and used as the input of the second normalized layer; the input of the second normalized layer and the output of the multilayer perceptron are residually connected and used as the output of the GCN block.

[0141] Specifically, the feature extraction formula for GCN blocks is as follows:

[0142] x′=GCN(LayerNorm(x))+x;

[0143] O=MLP(LayerNorm(x′))+x′;

[0144] The feature extraction formula for the GCN network layer is as follows:

[0145]

[0146] In the formula: This indicates a linear projection layer between layers; x is the inter-layer input; O is the inter-layer output; BatchNorm indicates normalization processing; Degree matrix; This represents the adjacency matrix.

[0147] In this embodiment, BatchNorm (Batch Normalization) is a technique used to accelerate the training process of neural networks, and it can also be applied in GCN. BatchNorm normalizes the input data distribution of each feature by subtracting the mean of each feature and dividing by the standard deviation. This helps to alleviate the vanishing and exploding gradient problems, accelerates the convergence process of the network, and improves the generalization ability of the model.

[0148] The degree matrix is ​​a diagonal matrix composed of the degree of each node in the graph (i.e., the number of edges connecting each node). In GCN, the degree matrix is ​​often normalized by raising it to the power of -1 / 2 to normalize the graph convolution operation. This normalization is called symmetric normalization, which ensures that the result of graph convolution is not affected by the degree of nodes in the graph, thereby improving the stability and generalization ability of the model. The adjacency matrix represents the structural information of the graph. The adjacency matrix records the connection relationships between nodes in the graph. In GCN, the adjacency matrix is ​​usually multiplied by the feature matrix to propagate node feature information. The adjacency matrix plays a crucial role in graph convolution operations; it represents the relationships between nodes in the graph, enabling GCN to perform effective information transfer and feature learning on graph structure data.

[0149] In this invention, the structure of the GCN block not only improves model accuracy but also makes the modules have similar structures, thus improving portability.

[0150] In practice, to compensate for the shortcomings of Transformer in extracting fine-grained spatial pose features, we introduced GCN into the model. In fact, networks based solely on GCN do not perform well in human pose estimation tasks. Figure 3 As shown in (a), GCN contains an adjacency matrix of a graph structure, which allows it to quickly learn the underlying skeletal information of human pose. However, precisely because of this, information between joints can only propagate along the graph structure. Joints that are far apart on the graph structure need to go through multiple layers of information propagation to obtain each other's information, which makes the depth of the GCN model directly affect the final estimation result.

[0151] This invention proposes a multi-scale Transformer (with features as input) We maintain consistency in the dimensions of input and output features across different feature layers, such as... Figure 3 As shown in (b), it divides a complete frame sequence into multiple short sequences for processing, thereby enhancing the model's ability to capture short-term dependencies between frames while reducing computational cost.

[0152]

[0153] In the formula: K represents the high-dimensional pose feature; S represents the scale, which gradually decreases to 1 as the depth increases.

[0154] GCN can capture the spatial local relationships between joints. Multi-scale Transformer can capture the temporal local relationships between the same joint at different time points, and its function is very similar to that of GCN. Therefore, we propose a feature layer based on GCN and Transformer, such as... Figure 2 As shown in (b), we extract the local spatiotemporal features of human pose using GCN blocks and multi-scale Transformers, and extract the global spatiotemporal features of human pose using traditional Transformers.

[0155] like Figure 2 As shown in (b), the multi-scale Transformer includes a Reshape function, a first normalization layer, an attention network layer, a second normalization layer, a multilayer perceptron, and a Restore function connected sequentially from end to end.

[0156] The input to the Reshape function is the input to the multi-scale Transformer; the input to the first normalized layer and the output of the attention network layer are residually connected and then used as the input to the second normalized layer; the input to the second normalized layer and the output of the multilayer perceptron are residually connected and then used as the input to the Restore function; the output of the Restore function is used as the output of the multi-scale Transformer.

[0157] Specifically, in PyTorch, the `reshape` function is used to change the shape of a tensor while keeping the number of elements unchanged. Its purpose is to reorganize the dimensions of a tensor to adapt to different computational needs. Both the `reshape` and `restore` functions use the `reshape` function to change the data dimensions; the difference lies in that `reshape` is used to change the dimensions, while `restore` is used to restore the original dimensions.

[0158] In its specific implementation, this invention uses a dynamic weighting method to fuse local and global features.

[0159] Specifically, the calculation formula for dynamically weighting local pose features and global pose features is as follows:

[0160] K i =K l ·W l +K g ·W g ;

[0161] W l W g =softmax(concat(K) l ,K g )W);

[0162] In the formula: This indicates that the input is 2d. m A linear layer with an output of 2; W l W g K represents the weights of local and global features, respectively. i This represents the output of the i-th feature layer.

[0163] In practical implementation, most existing 3D human pose estimation methods adopt end-to-end models, where the final prediction result depends only on the output of the last layer of the network. MHFormer (from W. Li, H. Liu, H. Tang, P. Wang, and L. Van Gool, "Mhformer: Multi-hypothesis transformer for 3d human poseestimation," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 13147-13156) uses the outputs of multiple feature encoding layers as different pose hypotheses, but this method is not efficient. To improve the reusability of data in the model, we propose a hierarchical feature fusion method. We select l features from the outputs of the arranged feature layers and input them into the feature fusion module to obtain the final features. We will discuss different fusion methods below.

[0164] Specifically, the hierarchical feature fusion layer performs feature fusion in one or more of the following ways:

[0165] 1) Weighted average pooling

[0166] The purpose of feature fusion is to combine low-dimensional raw features with high-dimensional abstract features to improve the accuracy of pose estimation with a small increase in model parameters. Currently, one of the most commonly used data fusion methods is weighted average pooling.

[0167] The formula is expressed as:

[0168]

[0169] In the formula: K represents the overall characteristic; w i K represents the weight of the i-th layer; i The i-th layer represents the output feature; L represents the set of selected feature layers; w i These are trainable parameters; the sum of the weights is constrained to 1 by normalization.

[0170] 2) Softmax

[0171] Unlike weighted average pooling, softmax is a method that approximates the maximum or minimum value.

[0172] The formula is expressed as:

[0173]

[0174] In the formula: w i ′ represents the weighted value after processing by the softmax() function.

[0175] When the weight differences are significant, the module's output is approximately equal to the feature value of the layer with the largest weight. When the weight differences are small, the module's output is approximately the mean of the features from each layer. This method introduces a small number of features from other layers while preserving key features, thus correcting the final result.

[0176] 3) Transformer

[0177] Multi-head attention is one of the key mechanisms for feature extraction in Transformers. Similarly, it can also be used for feature fusion.

[0178] The formula is expressed as:

[0179]

[0180] In the formula: MSA represents the multi-head self-attention mechanism in Transformer.

[0181] We concatenate the features from l feature layers along the channel dimension and calculate self-attention. The resulting... The number of feature channels is l times that of the original feature Ki. Finally, we use a multilayer perceptron to reduce the dimensionality of the feature channels, obtaining the final output:

[0182]

[0183] This method requires more computation, has greater uncertainty, and needs more data for training compared to the previous two methods.

[0184] 4) Average pooling

[0185] We can directly control the weights of each feature using manually set parameters, or simply take their average.

[0186] The formula is expressed as:

[0187]

[0188] This method does not introduce additional trainable parameters, but the fixed parameters have a very large impact on the final result.

[0189] The additional computational cost of using the four methods described above is negligible compared to the computational cost of the model itself. This feature fusion method establishes a direct link between shallow, raw pose features and deep, abstract pose features. It can be directly used as a plug-in for other pose estimation models, exhibiting strong portability.

[0190] In practical implementation, the attitude estimation model of this invention adopts an end-to-end training method. The loss function for training the attitude estimation model is as follows:

[0191]

[0192] In the formula: This represents the Mean Per-Joint Position Error (MPJPE). This represents the normalized average joint position error (N-MPJPE, N-MPJPE). This represents the Mean Per-Joint Velocity Error (MPJVE).

[0193] In this embodiment, the average joint position error Normalized mean joint position error and average joint velocity error All calculations were performed using existing, mature methods.

[0194] In this invention, MPJPE and N-MPJPE directly calculate the error of each joint, which is the most direct standard for measuring model performance. MPJVE calculates the velocity error of each joint, making the posture estimation during motion more accurate.

[0195] To better illustrate the advantages of the technical solution of the present invention, the following experiment is disclosed in this embodiment.

[0196] 1. Dataset and Evaluation Metrics

[0197] We evaluate our work on two well-known 3D human pose datasets: Human3.6M (from C. Ionescu, F. Li, and C. Sminchisescu, "Latent structured models for human poseestimation," in 2011 International Conference on Computer Vision, 2011: IEEE, pp. 2220-2227) and MPI-INF-3DHP (from D. Mehta et al., "Monocular 3d human poseestimation in the wild using improved CNN supervision," in 2017 International Conference on 3D Vision (3DV), 2017: IEEE, pp. 506-516).

[0198] Human3.6M is a large, publicly available dataset for 3D human pose estimation research. It is the most widely used indoor dataset in 3D HPE research, containing 3.6 million 3D human poses and their corresponding images. The dataset includes performances by 11 professional actors (6 men and 5 women) in 17 scenes, such as discussions, smoking, and taking photos. High-resolution 50Hz video was captured using four calibrated cameras to achieve accurate capture and synchronization. We selected five subsets (S1, S5, S6, S7, S8) for training and two subsets (S9, S11) for testing. We used a skeleton with 17 joints for estimation. In this dataset, MPJPE and P-MPJPE were used as evaluation metrics. P-MPJPE is a rigidly aligned version of MPJPE, used for post-processing alignment between the estimated 3D pose and the original values.

[0199] MPI-INF-3DHP is another popular 3D human pose estimation dataset, involving complex indoor and outdoor scenes. It contains videos of 8 actors performing 8 activities from 14 camera perspectives, totaling over 1.3 million frames. We used 8 subsets (S1, S2, S3, S4, S5, S6, S7, S8) as the training set and 6 subsets (TS1, TS2, TS3, TS4, TS5, TS6) as the test set. To compare with existing work, we used a skeleton with 14 joints (head, neck, shoulder, elbow, wrist, hip, knee, and ankle) for estimation. In addition to MPJPE, we also used PCK (percentage of correct keypoints within a 150mm range) and AUC (area under the curve) to evaluate the performance of our model on this dataset.

[0200] 2 Deployment Details

[0201] Our pose evaluation model is implemented using the PyTorch framework. The model parameters are set as follows: 16 feature layers, 128 feature dimensions, and 243 (Human 3.6M) and 81 (MPI-INF-3DHP) frames per input. For the multi-scale Transformer, we set four scales: 27, 9, 3, and 1, with four layers per scale. Correspondingly, we use features from these four scales for feature fusion as the final output. In Human3.6M, we use a stacked hourglass model (from A. Newell, K. Yang, and J. Deng, "Stacked hourglass networks for human pose estimation," in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, 2016: Springer, pp. 483-499.) as a 2D pose estimator, combining its output 2D pose and confidence scores into a 3D feature, which serves as the input to our model. Furthermore, we conducted experiments on two datasets using 2D pose labels as input.

[0202] For all training parameters, we used an RTX 4090 GPU for both training and testing. We used AdamW (from I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," arXiv preprint arXiv:1711.05101, 2017) as the optimizer, with an initial learning rate of 0.0005, a decay factor of 0.99 per epoch, and a batch size of 6. The training epochs were 60 (Human3.6M) and 90 (MPI-INF-3DHP) on the two datasets. The hyperparameters of the loss function... Set them to 1, 0.5, and 20 respectively.

[0203] 3 Performance Comparison

[0204] 3.1 Human 3.6M

[0205] We compared our method with existing methods on Human 3.6M. Table 1 lists the parameter count, computational cost (MACs), and error for each method. For fair comparison, all listed methods use either CPN or a stacked hourglass model as the 2D pose detector. Furthermore, the number of input frames per iteration and whether only frames at the center of the sequence are estimated for each input do not affect the fairness of the experimental results; listing these facilitates a more intuitive comparison. Using a 2D pose detector, our method achieves the highest performance of 37.7mm under the MPJPE standard and 31.7mm under the P-MPJPE standard. It outperforms the existing best-performing method (MotionAGFormer) by 0.7mm (a 1.8% improvement in MPJPE) and 0.9mm (a 2.8% improvement in P-MPJPE) under both standards. Using 2D pose labels as input, our method achieves the best performance of 17.3mm under the MPJPE standard, representing an improvement of 0.5mm and 2.8% compared to MotionBERT. Our model achieves state-of-the-art performance under both noisy and undisturbed conditions with a 2D pose detector. This validates the effectiveness of our method and demonstrates its robustness to diverse data sources. In terms of parameter and computational cost, our method is slightly more expensive than MotionAGFormer. Among models of similar size, our model is more lightweight than most while offering superior performance.

[0206] Table 1. Quantitative comparison of experiments on Human 3.6M.

[0207]

[0208]

[0209] The input to the model in Table 1 is assumed to be a 2D pose detector. F: Number of frames estimated by the model in one pass. CE: Whether the model only estimates the center frame. P1: MPJPE, in mm, lower is better. P2: P-MPJPE, in mm, lower is better. P1(GT): MPJPE using 2D pose labels as input. Bold text represents the best performing method, underlined text represents the second best.

[0210] 3.2MPI-INF-3DHP

[0211] Table 2 presents the quantitative comparison of our experiments on MPI-INF-3DHP. All methods used 2D pose labels as input. Our model achieved best performance in all three evaluation metrics: PCK, AUC, and MPJPE. Specifically, in the PCK metric, our model was on par with the state-of-the-art (SOTA) performance; in the AUC metric, our model significantly outperformed the SOTA model by 3.7% with a score of 87.9%; and in the MPJPE metric, our model surpassed the SOTA model by 1.3mm with a score of 16.9mm, representing an improvement of approximately 7.1%.

[0212] Table 2 Quantitative comparison of experiments on MPI-INF-3DHP

[0213]

[0214]

[0215] In Table 2, F: Number of frames estimated by the model in one iteration. PCK, AUC: Higher is better. MPJPE: Unit: mm, Lower is better. Bold text represents the best performing method, and underlined text represents the second best.

[0216] 4 Ablation Experiment

[0217] We conducted a series of ablation experiments on Human 3.6M to verify the effectiveness of each module in our method. All the following experiments used a stacked hourglass model as a 2D pose detector.

[0218] Effectiveness of each module: Based on the existing model, we removed the feature fusion module to verify its effectiveness. Furthermore, we removed the GCN module and adopted a pure Transformer structure. Table 3 shows the experimental results. With the feature fusion module removed, the model error increased by 0.3 mm (from 37.7 mm to 38.0 mm). With the pure Transformer structure, the model error increased by 2.1 mm (from 38.0 mm to 40.1 mm). This verifies the effectiveness of the two modules in our method.

[0219] Table 3 Ablation experiments of each module of the network

[0220]

[0221] In Table 3, MPJPE: unit mm, the lower the better.

[0222] Model parameter settings: We conducted ablation experiments on the key parameters of the model, as shown in Table 4. Increasing or decreasing the number of model layers and feature channels did not improve performance. Due to the limited amount of trainable 3D human pose data, excessive parameters actually led to performance degradation. The model achieved optimal performance when the number of model layers was set to 16 and the number of feature channels was set to 128.

[0223] Table 4 Ablation Experiment of Model Parameters

[0224]

[0225]

[0226] In Table 4, MPJPE is in mm, and the lower the better.

[0227] Feature Fusion Module: We analyzed the feature fusion methods, as shown in Table 5. Among the four methods, softmax achieved the best performance. However, the other three methods introduced more errors. Average pooling, while not introducing additional parameters or computation, led to performance degradation because shallow features can acquire the same weights as deep features. Compared to traditional end-to-end models, our model seems to lack such highly weighted shallow features. Furthermore, the Transformer-based fusion method introduced a significant amount of additional parameters and computation, yet its performance was even worse than direct average pooling. Due to insufficient training data, Transformer struggles to reconstruct true features from features of different depths, making shallow features more like noise interference for the final output layer. Finally, weighted average pooling and softmax showed drastically different performance for the same number of parameters and computation. For comparison, we recorded the changes in the weight ratio of each layer in the feature fusion module of these two methods with the number of training epochs, such as... Figure 4 As shown (where a larger weight number corresponds to a deeper feature layer, and Layer_weight3 represents the weight corresponding to the output of the last layer of the module. All weights have been normalized). During the training process of the weighted average pooling method, the weights of deeper features gradually decrease, eventually approaching 0.5. The sum of the weights of shallower features gradually increases, with the weight of the shallowest feature, Layer_weight0, having the largest weight, approximately 0.3. During the training process of the Softmax method, the weight of the deep feature, Layer_weight3, remains high, eventually reaching 0.96. The weights of intermediate layer features are almost zero, with only the shallowest feature having a small amount of weight.

[0228] Based on the experimental results of the feature fusion module above, we can conclude that shallow data can help the model optimize experimental results, but too much shallow data will affect the original performance. We only need a small amount of shallow data to correct the final experimental results.

[0229] Table 5 Ablation experiments using feature fusion methods

[0230]

[0231] In Table 5, MPJPE is in mm, and the lower the better.

[0232] 5. Qualitative Analysis

[0233] We conducted a qualitative comparison using visualization on Human3.6M.

[0234] Local features and global features: In the feature layer, we use a dynamic fusion method to process local features and global features. Figure 5 The weights of local and global features at each layer are shown. We averaged all weight data obtained during testing and normalized the local and global features for each layer. In the first and last layers, local features play a dominant role, while global features are negligible. In the middle layers, they each account for a certain proportion. If we measure the contribution of the two types of features to inference by their mean, the mean for local features is 0.598, and the mean for global features is 0.402. Experimental results validate the effectiveness of our proposed method.

[0235] 3D Human Pose: During the testing process, we randomly selected four groups of estimated poses and visualized them, such as... Figure 6 As shown (all poses have been processed centered on the root node. The gray skeleton represents the ground truth label. Green circles indicate areas where the estimation results of the three methods differ significantly). For comparison, we cite experimental data from MotionAGFormer and MotionBERT. Figure 6 In the diagram, joints on the left side of the human body are represented in black, and joints on the right side are represented in red. The gray skeleton represents the ground truth label. All poses in the diagram have been processed around the root node. The results show that our model provides more accurate estimations for the head, limbs, and other parts than other models.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A 3D human pose estimation method based on feature fusion of GCN and Transformer, characterized in that, include: S1: Obtain 2D pose keypoints with confidence; S2: Input the 2D pose key points with confidence into the trained pose estimation model, and output the corresponding predicted 3D human pose key points. The processing steps for the attitude estimation model are as follows: S201: Mapping 2D pose keypoints to a high dimension through an embedding layer to obtain high-dimensional features; S202: High-dimensional features are extracted sequentially through multiple layers of feature layers to obtain the corresponding spatiotemporal features; The feature layer includes global feature branches and local feature branches. The spatiotemporal features of the feature layer are generated by dynamically weighting the global pose features extracted by the global feature branches and the local pose features extracted by the local feature branches. The global feature branch extracts global pose features by sequentially connecting a spatial Transformer and a temporal Transformer. The local feature branch extracts local pose features through sequentially connected GCN blocks and multi-scale Transformers; S203: The spatiotemporal features corresponding to each hierarchical feature layer are fused through the hierarchical feature fusion layer to generate the corresponding overall features; S204: The overall features are processed by the regression head to generate corresponding predicted 3D human pose key points; S3: Use the output predicted 3D human pose keypoints as the corresponding 3D human pose estimation results.

2. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 1, characterized in that: In step S202, both the spatial Transformer and the temporal Transformer include a first normalized layer, an attention network layer, a second normalized layer, and a multilayer perceptron, which are connected end to end in sequence. The input to the first normalized layer is the input to either the spatial or temporal Transformer; the input to the first normalized layer and the output of the attention network layer are residually connected and then used as the input to the second normalized layer; the input to the second normalized layer and the output of the multilayer perceptron are residually connected and then used as the output to either the spatial or temporal Transformer.

3. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 2, characterized in that: 1) The attention network layer uses a multi-head self-attention mechanism, expressed by the following formula: MSA=Concat(head1,...,head n )W O ; head i =Attention(Q i ,K i ,V i ); In the formula: Concat represents a chain operation; head i Indicates the i-th attention head; Each attention head uses a scaled dot product attention mechanism to calculate the attention, as shown in the following formula: In the formula: W O d m A linear projection layer of dimensionality; n represents the number of key points in a spatial Transformer and the number of time frames in a temporal Transformer; d m This represents the number of feature channels, i.e., the dimension. 2) The calculation formula for a multilayer perceptron is as follows: MLP(x) = σ(xW1+b1)W2+b2; In the formula: σ(·) represents the activation function GeLU; and These represent the fully connected layers of a multilayer perceptron; and Representing fully connected layers and The bias; MLP(x) represents the multilayer perceptron.

4. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 1, characterized in that: In step S202, the GCN block includes a first layer normalization layer, a GCN network layer, a second layer normalization layer, and a multilayer perceptron connected sequentially from end to end; The input to the first normalization layer is the input to the GCN block; The input of the first normalized layer and the output of the GCN network layer are residually connected and then used as the input of the second normalized layer. The input of the second normalized layer and the output of the multilayer perceptron are residually connected and used as the output of the GCN block.

5. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 4, characterized in that: The feature extraction formula for GCN blocks is as follows: x′=GCN(LayerNorm(x))+x; O=MLP(LayerNorm(x′))+x′; The feature extraction formula for the GCN network layer is as follows: In the formula: BatchNorm represents the standardization process; Degree matrix; Represents the adjacency matrix; The inter-layer feature propagation formula for the GCN network is as follows: In the formula: GCN represents the GCN network layer; LayerNorm represents the layer normalization layer; MLP represents the multilayer perceptron; W1 (l) W2 (l) This represents the linear projection layer between layers; x is the inter-layer input; O is the inter-layer output; It is the adjacency matrix of an undirected graph, I N It is the identity matrix; W (l) H represents the linear projection layer between layers; σ(·) represents the ReLU activation function; (l) H is the feature matrix of each layer's input; (0) Features of the initial input; and Let i and j represent the elements in the i-th and j-th columns of the degree matrix and adjacency matrix, respectively.

6. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 1, characterized in that: In step S202, the multi-scale Transformer includes a Reshape function, a first normalization layer, an attention network layer, a second normalization layer, a multilayer perceptron, and a Restore function, which are connected end to end in sequence. The input to the Reshape function is the input to the multi-scale Transformer; the input to the first normalized layer is residually connected to the output of the attention network layer and then used as the input to the second normalized layer. The input of the second normalized layer and the output of the multilayer perceptron are residually connected and used as the input of the Restore function; the output of the Restore function is used as the output of the multiscale Transformer.

7. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 6, characterized in that: The formula for calculating the Reshape function is as follows: In the formula: K represents the high-dimensional attitude feature; S represents the scale, which gradually decreases to 1 as the depth increases.

8. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 1, characterized in that: In step S202, the calculation formula for dynamically weighting the local pose features and global pose features is as follows: K i =K l ·W l +K g ·W g ; W l ,W g =softmax(concat(K l ,K g )W); In the formula: This indicates that the input is 2d. m A linear layer with an output of 2; W l W g K represents the weights of local and global features, respectively. i This represents the output of the i-th feature layer.

9. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 1, characterized in that: In step S203, the hierarchical feature fusion layer performs feature fusion in one or more of the following ways: 1) Weighted average pooling The formula is expressed as: In the formula: K represents the overall characteristics; w i K represents the weight of the i-th layer; i The i-th layer represents the output feature; L represents the set of selected feature layers; w i These are trainable parameters; 2) Softmax The formula is expressed as: In the formula: w′ i This represents the weighted value after processing by the softmax() function; 3) Transformer The formula is expressed as: In the formula: MSA represents the multi-head self-attention mechanism in Transformer; 4) Average pooling The formula is expressed as:

10. The feature fusion 3D human pose estimation method based on GCN and Transformer as described in claim 1, characterized in that: The loss function for training the pose estimation model is as follows: In the formula: This represents the average joint position error; This represents the normalized average joint position error. This represents the average joint velocity error.

Images