Three-dimensional human pose estimation method fusing double-layer semantic information

Abstract

The application provides a three-dimensional human body posture estimation method fusing double-layer semantic information, and relates to the technical field of three-dimensional posture estimation.The application generates an initial feasible three-dimensional posture containing diversified information by using a hierarchical feature extraction module, and provides different levels of semantic information; important information is strengthened through effective and fine processing of the features, so that the feasible three-dimensional posture features used for fusion operation finally have stronger expression ability; the correlation calculation module and the correlation feature fusion mechanism can effectively fuse the complementary information in the feasible three-dimensional posture features, and fully utilize the global information to promote the model to perform more accurate human body posture estimation work; compared with other methods, the application has lower average joint error, higher joint point accuracy, lower parameter quantity and calculation amount, better avoids the error caused by human self-occlusion and complex posture, and has higher accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional pose estimation technology, specifically a three-dimensional human pose estimation method that integrates two-layer semantic information. Background Technology

[0002] 3D human pose estimation is an important visual task, aiming to predict the 3D coordinates of human joints and the angles between them, and to construct a representation of the human structure for further analysis of human pose. It has wide applications, such as in human-computer interaction, action recognition, augmented reality, and virtual reality. Monocular 3D human pose estimation methods can be divided into two main categories: single-stage and two-stage methods. Single-stage methods estimate the 3D human pose directly from the input 2D image in one step, providing the 3D coordinates of all key points at once without intermediate 2D pose representations. Because there are no intermediate steps to refine or progressively correct the estimation results, the accuracy of pose estimation is prone to low in complex scenes (occlusion, complex poses).

[0003] Most methods employ a two-stage approach for 3D human pose estimation: first, keypoints are detected and located in the image using 2D human pose estimation; then, a network is used to perform 2D-to-3D estimation, leveraging the superior performance of 2D human pose estimation. The two-stage method can accurately and efficiently estimate 3D pose using the detected 2D keypoint information. For example, SimpleBaseline uses a fully connected residual network to directly map 2D pose to 3D space on a single video frame. However, the lack of temporal information leads to inaccurate pose estimation. Video-based methods incorporate temporal information; the contextual information provided by adjacent frames can significantly aid in estimating the pose of the current frame. For instance, VideoPose3D uses temporal convolution to process sequence information using a TCN (Temporal Convolutional Network) and outputs 3D pose, thus detecting human pose in videos. Furthermore, it incorporates semi-supervised training to address the issue of limited labeled real 3D pose data. Typically, the bone length of the same person does not change significantly within the same video segment. Consistency constraints on bone length can effectively reduce inaccurate 3D estimation results caused by uncertain bone length. For example, Anatomy3D decomposes the task into bone orientation and length estimation, achieving good prediction results while ensuring temporal consistency, but its efficiency in capturing global contextual information across frames is relatively low. Tan Jiawei et al. optimized the 3D joint coordinates of the human body using 2D pose estimation results, reducing high-frequency body jitter to maintain motion continuity. They also added bone connection constraints to establish a model that maintains smooth motion trajectories and optimizes the consistency of joint connection structures before and after movements, achieving accurate estimation of human posture.

[0004] With the development of artificial intelligence, Transformers, with their high-performance self-attention mechanism, are playing an increasingly important role in computer vision. ViT directly applies the standard Transformer architecture to sequence image patches, achieving image classification by constructing a simple yet highly scalable model. In human pose estimation, PoseFormer uses a Transformer network to capture joint correlations and temporal dependencies to estimate the 3D pose of humans in videos. However, due to human self-occlusion and the depth ambiguity of 2D data, the accuracy of 3D human pose estimation is not high. StridedTransformer uses a Transformer architecture with strided convolutions to elevate a long 2D pose sequence into a single 3D pose. ConvFormer proposes a novel dynamic multi-head convolutional self-attention mechanism to comprehensively simulate human joint relationships in both intra-frame and global motion sequences. DC-GCT proposes a bi-chain graph convolutional Transformer that constrains poses through a bi-chain design consisting of local-to-global and global-to-local chains to obtain a more complex representation suitable for the current human pose. However, these methods all ignore the multi-solution problem in the 2D pose to 3D pose mapping process, and a single solution often fails to achieve satisfactory results.

[0005] To better handle the multiple-solution problem, Jahangiri et al. generated multiple 3D candidate poses consistent with 2D keypoints by constraining pose through compositional modeling and anatomy. Wehrbein et al. proposed a normalized flow-based method that utilizes the determinism of 3D-to-2D mapping to address the ambiguity in the 2D-to-3D upscaling process, generating multiple feasible 3D poses. While these methods achieved good results, they all focused on one-to-many mappings and could not effectively integrate the different semantic information contained in multiple feasible 3D poses. MHForme, combined with a Transformer network, completed the fusion of multiple feasible 3D poses through two mappings: one-to-many and many-to-one. However, due to the extensive use of the Transformer structure, the deep learning model was large in scale, and the lack of sufficient learning of the dependencies between feasible 3D poses resulted in low accuracy of the fusion results. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to propose a three-dimensional human pose estimation method that integrates two-layer semantic information, including:

[0007] Step 1: Obtain N video frames, including the video frame to be estimated and N-1 video frames within a preset time range. Obtain the two-dimensional human joint pose sequence X∈R corresponding to the N video frames. N ×J×2The two-dimensional pose sequence X of human joints includes the coordinates of human joint points in each video frame, and J is the number of human joint points in each video frame.

[0008] Step 2: Set the total number of feasible pose features M, and process the human joint two-dimensional pose sequence X through the hierarchical feature extraction module HFE to obtain M initial feasible three-dimensional pose features.

[0009] Step 3: Process each initial feasible 3D pose feature through the feature refinement module (FRM) to obtain M feasible 3D pose features refined by the feature refinement module, and the convergent representation A of the feasible 3D pose features.

[0010] Step 4: The HFF module is used to process the M feasible 3D pose features refined by the feature refinement module and the convergent representation A of the feasible 3D pose features to obtain the final human pose estimation result of the video frame to be estimated.

[0011] Optionally, step 2 specifically includes:

[0012] Step 2.1: Set the total number M of feasible pose features, and stitch together the joint coordinates (x, y) of all video frames in the human joint 2D pose sequence X to obtain the stitched joint information. Add first sequence information E to each joint in the two-dimensional pose sequence X of human joints. SPos ∈R (J·2)×N Set the initial value of parameter m to 1;

[0013] Step 2.2: Transfer the first sequence information E SPos Joint information after splicing The fusion process yields a sequence of key points with fused spatial information. Specifically, this is achieved through the following formula:

[0014]

[0015] in, For the m-th intermediate joint sequence, X m-1 This represents the (m-1)th initial feasible 3D pose feature;

[0016] The joint point sequence obtained after fusing spatial information is input into the encoder of the hierarchical feature extraction module, and the first feasible 3D pose feature is obtained through MESA encoding, specifically through the following formula:

[0017]

[0018] in, For the m-th intermediate feasible 3D pose feature This is the sequence of joints obtained after fusing the m-th spatial information. This is the m-th first feasible 3D pose feature;

[0019] The intermediate joint sequence, the joint point sequence obtained after fusing spatial information, and the first feasible 3D pose features are added together to obtain the features output by the encoder, which is specifically achieved through the following formula:

[0020]

[0021] in, The feature output by the m-th encoder;

[0022] Perform a one-dimensional convolution on the features output by the encoder to obtain the initial feasible three-dimensional pose features X. m Specifically, this is achieved through the following formula:

[0023]

[0024] Among them, X m Let m be the m-th initial feasible 3D pose feature, GELU be the activation function, and Conv be the one-dimensional convolution;

[0025] Step 2.3: Determine if m is greater than M. If m is greater than M, obtain M initial feasible 3D pose features. If m is less than or equal to M, increment the value of m by one and return to step 2.2.

[0026] Optionally, in step 2.2 Specifically, it is calculated using the following formula:

[0027] right Perform a linear mapping to obtain query Q, key K, and value V. Based on query Q, key K, and value V, calculate... Specifically, this is achieved through the following formula:

[0028]

[0029] Attn′=Sigmoid(Sum(Attn));

[0030] Attention′(Q,K,V)=Softmax(Attn)×Attn′×V;

[0031] Where d represents the dimensions of query Q, key K, and value V, and Attention'(Q,K,V) is the output result. Attn is the scaled dot product of Q and K, and Attn′ is the result after applying Sum and Sigmoid activation to Attn.

[0032] Optionally, step 3 specifically includes:

[0033] Step 3.1: Process each initial feasible 3D pose feature using the Feature Refinement Module (FRM), and blend the feasible 3D pose features Z∈R. N×(C×N) ;

[0034] Step 3.2: Process the hybrid feasible 3D pose features Z to obtain M feasible 3D pose features refined by the feature refinement module, and the convergent representation A of the feasible 3D pose features.

[0035] Optionally, step 3.1 specifically includes:

[0036] For each initial feasible 3D pose feature, it is transformed through bitwise operations and linear transformations to obtain high-dimensional features, and the second order information E of each video frame in the sequence is obtained. TPos ∈R N×C By embedding high-dimensional features, feasible 3D pose features with embedded temporal and positional information are obtained. Specifically, this is achieved through the following formula:

[0037]

[0038] Among them, X m The m-th initial feasible 3D pose feature. Z represents the m-th feasible 3D pose feature embedded with temporal location information. m Z is the m-th high-dimensional feature. m ∈R N×C ;

[0039] The feasible 3D pose features embedded with time and position information are input into MESA. The output of MESA is added to the feasible 3D pose features embedded with time and position information to obtain the first intermediate feasible 3D pose features, which is achieved through the following formula:

[0040]

[0041] in, This represents the m-th first intermediate feasible 3D pose feature;

[0042] Then, M first intermediate feasible 3D pose features are calculated. All first intermediate feasible 3D pose features are concatenated to obtain the hybrid feasible 3D pose feature Z, which is achieved through the following formula:

[0043]

[0044] in, This is the first feasible 3D pose feature in the first intermediate stage. This is the second first intermediate feasible 3D pose feature. This is the Mth first intermediate feasible 3D pose feature.

[0045] Optionally, step 3.2 specifically includes:

[0046] The hybrid feasible 3D pose features Z are processed using a multilayer perceptron (MLP), and the processed result is added to the hybrid feasible 3D pose features Z to obtain a convergent representation after exchanging key information from different feasible 3D pose features. Specifically, this is achieved through the following formula:

[0047]

[0048] Convergent representation after exchanging key information of different feasible 3D pose features The features are split into M feasible 3D pose features, which are represented by the following formula:

[0049]

[0050] Split(·) is the split operation. This is the first feasible 3D pose feature after splitting. This is the second feasible 3D pose feature after splitting. The Mth feasible 3D pose feature after splitting;

[0051] The decomposed feasible 3D pose feature corresponding to each initial feasible 3D pose feature is added to the feasible 3D pose feature corresponding to that initial feasible 3D pose feature to obtain the feasible 3D pose feature after refinement by the feature refinement module, which is specifically expressed by the following formula:

[0052]

[0053] Among them, A m The m-th feasible 3D pose feature after refinement by the feature refinement module;

[0054] This yields M feasible 3D pose features refined by the feature refinement module. All these features are then concatenated to obtain a convergent representation A of the feasible 3D pose features, specifically expressed by the following formula:

[0055] A = Concat(A1, A2, ..., A M )∈R N×(C×m) ;

[0056] Where A1 is the first feasible 3D pose feature refined by the feature refinement module, and A2 is the second feasible 3D pose feature refined by the feature refinement module. M This is the Mth feasible 3D pose feature after refinement by the feature refinement module.

[0057] Optionally, step 4 specifically includes:

[0058] The convergent representation A of the feasible 3D pose features is processed by the Association Computation Module (ACM) to obtain the association features R, which are specifically expressed by the following formula:

[0059] R = ACM(A);

[0060] The cross-attention algorithm is used to process the associated features R and the feasible 3D pose features refined by the feature refinement module, resulting in feasible 3D pose features after relational fusion, which is expressed by the following formula:

[0061]

[0062] Among them, A m+1 F is the (m+1)th feasible 3D pose feature refined by the feature refinement module. m Let d represent the feasible 3D pose feature after the fusion of the m-th relation, where d represents the dimension.

[0063] Then, M feasible 3D pose features after relation fusion are obtained. Through splicing and multilayer perceptron, all feasible 3D pose features after relation fusion are processed to obtain the final convergent representation F of feasible 3D pose features, which is specifically achieved through the following formula:

[0064] F′=Concat(F1,F2,…,F M );

[0065] F = F′ + MLP(LN(F′));

[0066] Where F' is the concatenated representation of feasible 3D pose features after the fusion of M relations, F'∈R N×(C×N) F1 represents the feasible 3D pose features after the first relation fusion, and F2 represents the feasible 3D pose features after the second relation fusion. M The feasible 3D pose feature after the Mth relation is fused;

[0067] A linear transformation layer is used to perform linear regression on F to obtain the keypoint sequence of the three-dimensional pose. from Select the 3D pose of the video frame to be estimated. This serves as the final result of human pose estimation.

[0068] The beneficial effects of adopting the above technical solution are as follows:

[0069] This invention utilizes a hierarchical feature extraction module to generate initial feasible 3D poses containing diverse information, providing semantic information at different levels. Through effective and refined feature processing, important information is enhanced, ensuring that the feasible 3D pose features used for fusion operations have stronger expressive power. The association calculation module and association feature fusion mechanism can effectively fuse complementary information in feasible 3D pose features, fully utilizing global information to drive the model to perform more accurate human pose estimation. Compared with other methods, this invention achieves lower average joint error, higher joint accuracy, lower parameter and computational cost, better avoids errors caused by self-occlusion and complex poses, and achieves higher accuracy and robustness. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of a three-dimensional human pose estimation method that integrates two-layer semantic information in an embodiment of the present invention;

[0071] Figure 2 This is a schematic diagram of the hierarchical feature extraction module HFE in an embodiment of the present invention;

[0072] Figure 3 This is a schematic diagram of the feature refinement module (FRM) in an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of the structure of the hierarchical feature fusion module HFF in an embodiment of the present invention;

[0074] Figure 5 This is a schematic diagram illustrating the DC-GCT method, P-STMO method, MHFormer method, and the visualization results of the present invention on the Human3.6M dataset in the embodiments of the present invention.

[0075] Wherein, (a2), (a3), (a4), and (a5) are the human pose estimation results of the input image (a1) obtained based on the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and the present invention, respectively; (a6) is the true pose value of the input image (a1); (b2), (b3), (b4), and (b5) are the human pose estimation results of the input image (b1) obtained based on the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and the present invention, respectively; and (b6) is the true pose value of the input image (b1). (c2), (c3), (c4), and (c5) are the human pose estimation results of the input image (c1) based on the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and the present invention, respectively; (c6) is the true pose value of the input image (c1); (d2), (d3), (d4), and (d5) are the human pose estimation results of the input image (d1) based on the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and the present invention, respectively; (d6) is the true pose value of the input image (d1).

[0076] Figure 6 This is a schematic diagram illustrating the human pose estimation results using network video in an embodiment of the present invention. (a1) is the input image, (a2) is the human pose estimation result of (a1), (b1) is the input image, (b2) is the human pose estimation result of (b1), (c1) is the input image, (c2) is the human pose estimation result of (c1), (d1) is the input image, (d2) is the human pose estimation result of (d1), (e1) is the input image, (e2) is the human pose estimation result of (e1), and (f1) is the input image, (f2) is the human pose estimation result of (e1). (f1) is the human pose estimation result, (g1) is the input image, (g2) is the human pose estimation result of (g1), (h1) is the input image, (h2) is the human pose estimation result of (h1), (i1) is the input image, (i2) is the human pose estimation result of (i1), (j1) is the input image, (j2) is the human pose estimation result of (j1), (k1) is the input image, (k2) is the human pose estimation result of (k1), (l1) is the input image, (l2) is the human pose estimation result of (l1).

[0077] Figure 7This is a schematic diagram of the human pose estimation results with self-occlusion and complex movements in an embodiment of the present invention. In the diagram, (a1) is the input image with self-occlusion and complex movements, (a2) is the human pose estimation result of (a1), (b1) is the input image with self-occlusion and complex movements, (b2) is the human pose estimation result of (b1), (c1) is the input image with self-occlusion and complex movements, (c2) is the human pose estimation result of (c1), (d1) is the input image with self-occlusion and complex movements, and (d2) is the human pose estimation result of (d1). Detailed Implementation

[0078] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0079] To address the problems existing in current technologies and to solve the issue of insufficient learning of dependencies between feasible 3D poses during the fusion of multiple feasible 3D poses, this invention proposes a Semantic Information Fusion Network (SIF) to fuse semantic information at different levels. The main contributions are as follows:

[0080] 1) A semantic information fusion network structure is proposed, which can fully learn the dependencies between different feasible 3D poses, improve the accuracy of human pose estimation, and effectively deal with situations such as self-occlusion of people and the existence of complex poses.

[0081] 2) Design a hierarchical feature extraction module and a feature refinement module to integrate spatial and temporal information to extract semantic information at different levels and strengthen the positional correlation between key points.

[0082] 3) Design a hierarchical feature fusion module and an association calculation submodule to realize the fusion processing of multi-level semantic information and the generation of the final pose representation by utilizing the association features between feasible 3D poses.

[0083] Specifically, a hierarchical feature extraction module is proposed to model the intrinsic structural information of human joints and extract feasible 3D pose features containing semantic information at different levels, while improving the utilization rate of joint position information. Secondly, to further improve network performance, a feature refinement module is designed to perform self-information transmission of feasible 3D pose features, thereby enhancing the positional correlation between joints. Finally, a hierarchical feature fusion module and an association calculation submodule are proposed to learn the dependencies between multiple feasible 3D pose features and perform cross-feasible 3D pose information transmission between feasible 3D pose features based on the relationships, so as to fuse them into accurate and unified feasible 3D pose features, thereby making full use of the different levels of semantic information of different feasible 3D poses to obtain the final 3D human pose estimation result.

[0084] The Semantic Information Fusion Network (SIF) proposed in this invention consists of a Hierarchical Feature Extraction Module (HFE), a Feature Refinement Module (FRM), and a Hierarchical Feature Fusion Module (HFF). First, the Hierarchical Feature Extraction Module models the intrinsic structural information of human joints, extracting m initial feasible 3D pose features with different levels of semantic information, where N is the number of input video frames. Next, the extracted initial feasible 3D pose features are input into the Feature Refinement Module, which transmits internal information for each feasible 3D pose feature. Finally, the Hierarchical Feature Fusion Module obtains the final prediction result.

[0085] Based on this, the present invention provides a three-dimensional human pose estimation method that integrates two-layer semantic information, combined with Figure 1 This may include the following steps:

[0086] Step 1: Obtain N video frames, including the video frame to be estimated and N-1 video frames within a preset time range. Obtain the two-dimensional human joint pose sequence X∈R corresponding to the N video frames. N ×J×2 The two-dimensional pose sequence X of human joints includes the coordinates of human joint points in each video frame, and J is the number of human joint points in each video frame.

[0087] Among these, the positional relationships between the joints of the human body can better assist in 3D human pose estimation. To better discover and utilize the key information between the joints, this invention designs a hierarchical feature extraction module (HFE) to construct reasonable initial feasible 3D pose features to express semantic information at different levels. Since the Transformer architecture has good performance in processing sequential data (especially long dependency modeling), the method of this invention uses the Transformer architecture to build the network. The basic components of the Transformer include Multi-Head Self-Attention (MSA) and Multilayer Perceptron (MLP). In Multi-Head Self-Attention (MSA), the input x∈R... n×d Linear mapping to query Q∈R n×d Key K∈R n×d Sum of values ​​V∈R n×d Where n is the length of the input sequence and d is the dimension, the scaled attention can be calculated as follows:

[0088]

[0089] MSA splits the query Q, key K, and value V into h parts, then performs attention calculations in parallel, and finally concatenates the outputs of the h attention heads linearly (h is the number of attention heads in the multi-head attention). A Multilayer Perceptron (MLP) consists of two linear layers for non-linear feature transformation.

[0090] Based on this, the present invention constructs a hierarchical feature extraction module (HFE) based on the Transformer network structure, such as... Figure 2 Then, the initial feasible 3D pose features are extracted through the hierarchical feature extraction module HFE, which is specifically achieved through step 2.

[0091] Step 2: Set the total number of feasible pose features M, and process the human joint two-dimensional pose sequence X through the hierarchical feature extraction module HFE to obtain M initial feasible three-dimensional pose features.

[0092] The value of M is set based on the user's needs; in this invention, the value of M is 2.

[0093] Step 2.1: Set the total number M of feasible pose features, and concatenate the joint coordinates (x, y) of all video frames in the two-dimensional pose sequence X of the human joints together, i.e. Figure 2 The concatenated coordinates in the data are used to obtain the spliced ​​joint information. Add first sequence information E to each joint in the two-dimensional pose sequence X of human joints. SPos ∈R (J·2)×N ,Right now Figure 2The spatial location is embedded in the data to save the spatial information of human joints, and the initial value of parameter m is set to 1.

[0094] Step 2.2: Transfer the first sequence information E SPos Joint information after splicing The fusion process yields a sequence of key points with fused spatial information. Specifically, this is achieved through the following formula:

[0095]

[0096] in, For the m-th intermediate joint sequence, X m-1 This represents the (m-1)th initial feasible 3D pose feature;

[0097] The joint sequence obtained after fusing spatial information is input into the encoder of the hierarchical feature extraction module, and the first feasible 3D pose feature is obtained through MESA encoding, which is specifically achieved through the following formula:

[0098]

[0099] in, For the m-th intermediate feasible 3D pose feature This is the sequence of joints obtained after fusing the m-th spatial information. This is the m-th first feasible 3D pose feature;

[0100] MESA (Multi-Head Enhanced Self-Attention) is a multi-head enhanced self-attention mechanism, and LN(·) is a LayerNorm layer. While MESA and MSA share similarities, they employ different computational strategies. Like MSA, the input x∈R n×d Linear mapping to query Q∈R n×d Key K∈R n×d Sum of values ​​V∈R n×d The algorithm performs h segmentations and uses parallel processing to calculate attention, then linearly concatenates the outputs of the h attention heads. Unlike MSA, this invention introduces an additional Sum operation and Sigmoid activation function after performing the dot product operation between Q and K. The aim is to construct more accurate feasible 3D pose features. This invention focuses on learning the relationships between different feasible solutions in the human pose estimation process. The SUM and Sigmoid operations added in MESA can effectively improve the correlation between joint positions in a single feasible solution, helping to calculate the dependencies between feasible solutions in subsequent calculations, thereby improving the computational performance of the model.

[0101] Based on this Specifically, it is calculated using the following formula:

[0102] right Perform a linear mapping to obtain query Q, key K, and value V. Based on query Q, key K, and value V, calculate... Specifically, this is achieved through the following formula:

[0103]

[0104] Attn′=Sigmoid(Sum(Attn));

[0105] Attention′(Q,K,V)=Softmax(Attn)×Attn′×V;

[0106] Where d represents the dimensions of query Q, key K, and value V, and Attention'(Q,K,V) is the output result. Attn is the scaled dot product of Q and K, and Attn′ is the result after applying Sum and Sigmoid activation to Attn.

[0107] To improve gradient propagation and network performance, a residual connection is added between the encoder's input and output features. Specifically, the intermediate joint sequence, the joint point sequence obtained after fusing spatial information, and the first feasible 3D pose features are added to obtain the encoder's output features, which is achieved through the following formula:

[0108]

[0109] in, The feature output by the m-th encoder;

[0110] Furthermore, one-dimensional convolution is used to learn spatial information. Specifically, one-dimensional convolution is performed on the features output by the encoder to obtain the initial feasible three-dimensional pose features X. m Specifically, this is achieved through the following formula:

[0111]

[0112] Among them, X m Let m be the m-th initial feasible 3D pose feature, GELU be the activation function, and Conv be the one-dimensional convolution;

[0113] Step 2.3: Determine if m is greater than M. If m is greater than M, obtain M initial feasible 3D pose features, i.e. Figure 2 If the feasible 3D pose features are in the range, and m is less than or equal to M, increment the value of m by one and return to step 2.2.

[0114] In other words, in the specific implementation of this invention, the initial value of m is 1, and the value of M is 2. When m = 1, the first initial feasible three-dimensional pose feature X1 is obtained, and then m is incremented by 1, at which point the value of m is 2. Then, we return to step 2.2. Then, the second initial feasible three-dimensional pose feature X2 is obtained by sequentially executing the process.

[0115] The initial feasible 3D pose features generated in the hierarchical feature extraction module contain multi-level semantic information. To fully utilize the important information such as positional correlation contained in the feasible 3D pose features and effectively capture temporal dependencies in the sequence time domain, a feature refinement module (FRM) is designed to enhance the important information within the feasible 3D pose features and mine temporal information. Based on this, this invention designs the feature refinement module FRM as follows: Figure 3 For specific implementation steps, see step 3.

[0116] Step 3: Process each initial feasible 3D pose feature through the feature refinement module (FRM) to obtain M feasible 3D pose features refined by the feature refinement module, and the convergent representation A of the feasible 3D pose features.

[0117] Step 3.1: Process each initial feasible 3D pose feature using the Feature Refinement Module (FRM), and blend the feasible 3D pose features Z∈R. N×(C×N) ;

[0118] For each initial feasible 3D pose feature, it is transformed through bitwise operations and linear transformations to obtain high-dimensional features, and the second order information E of each video frame in the sequence is obtained. TPos ∈R N×C By embedding high-dimensional features, feasible 3D pose features with embedded temporal and positional information are obtained. To ensure that each frame of data carries the corresponding timestamp information, thereby accurately preserving and reflecting the time clues in the sequence, and ultimately achieving the goal of optimizing feasible 3D pose features and improving network performance, the following formula is used:

[0119]

[0120] Among them, X m The m-th initial feasible 3D pose feature. Z represents the m-th feasible 3D pose feature embedded with temporal location information. m Z is the m-th high-dimensional feature. m ∈R N×C ;

[0121] The refinement process of feasible 3D pose features includes two parts. The first part involves self-information transfer for individual feasible 3D pose features, using a multi-head self-attention mechanism to enhance the features, and adding residual connections between the initial input and output to combine multiple feasible 3D pose features embedded with temporal information. The input is fed into a parallel MESA block. The first part specifically includes: inputting the feasible 3D pose features embedded with time position information into the MESA, and adding the output of the MESA with the feasible 3D pose features embedded with time position information to obtain the first intermediate feasible 3D pose features, which is specifically achieved through the following formula:

[0122]

[0123] in, This represents the m-th first intermediate feasible 3D pose feature;

[0124] Then, M first intermediate feasible three-dimensional pose features are calculated;

[0125] The second part employs a multilayer perceptron that mixes feasible 3D pose features. By exchanging keypoint information between different feasible 3D pose features, M feasible 3D pose features are mixed into a convergent representation. Specifically, the second part includes: concatenating all first intermediate feasible 3D pose features to obtain the mixed feasible 3D pose feature Z, which is achieved through the following formula:

[0126]

[0127] in, This is the first feasible 3D pose feature in the first intermediate stage. This is the second first intermediate feasible 3D pose feature. This is the Mth first intermediate feasible 3D pose feature.

[0128] Step 3.2: Process the hybrid feasible 3D pose features Z to obtain M feasible 3D pose features refined by the feature refinement module, and the convergent representation A of the feasible 3D pose features.

[0129] The hybrid feasible 3D pose features Z are processed using a multilayer perceptron (MLP), and the processed result is added to the hybrid feasible 3D pose features Z to obtain a convergent representation after exchanging key information from different feasible 3D pose features. Specifically, this is achieved through the following formula:

[0130]

[0131] Convergent representation after exchanging key information of different feasible 3D pose features The features are split into M feasible 3D pose features, which are represented by the following formula:

[0132]

[0133] Split(·) is the split operation. This is the first feasible 3D pose feature after splitting. This is the second feasible 3D pose feature after splitting. The Mth feasible 3D pose feature after splitting;

[0134] In this invention, the splitting is performed by re-splitting evenly according to the channel dimension.

[0135] To better propagate gradient information, a residual connection is added between the overall input and the results obtained from the splitting process in the feature refinement process. Specifically, this involves adding the split feasible 3D pose feature corresponding to each initial feasible 3D pose feature to the feasible 3D pose feature corresponding to that initial feasible 3D pose feature, to obtain the feasible 3D pose feature after refinement by the feature refinement module, as expressed by the following formula:

[0136]

[0137] Among them, A m The m-th feasible 3D pose feature after refinement by the feature refinement module;

[0138] This yields M feasible 3D pose features refined by the feature refinement module. All these features are then concatenated to obtain a convergent representation A of the feasible 3D pose features, specifically expressed by the following formula:

[0139] A = Concat(A1, A2, ..., A M )∈R N×(C×m) ;

[0140] Where A1 is the first feasible 3D pose feature refined by the feature refinement module, and A2 is the second feasible 3D pose feature refined by the feature refinement module. M This is the Mth feasible 3D pose feature after refinement by the feature refinement module.

[0141] The feasible 3D pose features, refined by the feature refinement module, show significant enhancements in spatial position correlation between human joints and temporal continuity and dependency in the temporal domain. To further improve the accuracy of human pose estimation, a hierarchical feature fusion module is proposed. This module fuses information across feasible 3D poses by constructing an information exchange channel and fully learning the dependencies between feasible 3D poses. It integrates complementary positional and temporal correlation details from feasible 3D pose features. Based on this, the hierarchical feature fusion module HFF is designed, and its structure is shown in the figure below. Figure 4 The specific implementation is achieved through step 4.

[0142] Step 4: The HFF module is used to process the M feasible 3D pose features refined by the feature refinement module and the convergent representation A of the feasible 3D pose features to obtain the final human pose estimation result of the video frame to be estimated.

[0143] Specifically, in step 4, the associated computing module (ACM) is first used to learn the dependencies between feasible 3D pose features. The associated features R are obtained through two consecutive one-dimensional convolutions and a BatchNorm operation. Specifically, the associated computing module (ACM) processes the convergent representation A of the feasible 3D pose features to obtain the associated features R, which is expressed by the following formula:

[0144] R = ACM(A);

[0145] To achieve information transfer between feasible 3D poses, feasible 3D pose features A are analyzed based on the associated features R. m Feature fusion is essentially an application of the cross-attention mechanism. The traditional calculation process for cross-attention using three feature vectors is as follows: Input x∈R... n×d y∈R n×d z∈R n×d Linearly mapped to query Q∈R respectively n×d Key K∈R n×d Sum of values ​​V∈R n×d Where n is the length of the input sequence and d is the dimension, the scaled cross-attention is calculated as follows:

[0146]

[0147] The dot product operation is performed by querying Q and key K, and the attention weights are obtained by using the softmax function. The attention weights are multiplied by the value V to obtain the result of cross-attention.

[0148] Based on this, the present invention can obtain associated features by learning the dependencies between feasible 3D pose features, which can effectively adjust attention weights and assist in multi-head cross-attention calculation. Specifically, it includes: processing the associated features R and the feasible 3D pose features refined by the feature refinement module through a cross-attention algorithm to obtain feasible 3D pose features after relationship fusion, as expressed by the following formula:

[0149]

[0150] Among them, A m+1 F is the (m+1)th feasible 3D pose feature refined by the feature refinement module. m Let d represent the feasible 3D pose feature after the fusion of the m-th relation, where d represents the dimension.

[0151] Wherein, when the value of m reaches M, m+1 is set to 1. In the specific implementation process, when the value of m is 1, A... m+1 Let A2 be the value of m. When the value of m is 2, A m+1 It is A1.

[0152] This leads to the feasible 3D pose features obtained by fusing M relationships;

[0153] Similar to the feature refinement module, a multilayer perceptron with hybrid feasible 3D pose features is used to fully exchange joint correlation information between different feasible 3D pose features, merging the feasible 3D pose features into a convergent representation. Specifically, this involves processing the feasible 3D pose features after fusion of all relationships through concatenation and a multilayer perceptron to obtain the final convergent representation F of the feasible 3D pose features, which is achieved through the following formula:

[0154] F′=Concat(F1,F2,…,F M );

[0155] F = F′ + MLP(LN(F′));

[0156] Where F' is the concatenated representation of feasible 3D pose features after the fusion of M relations, F'∈R N×(C×N) F1 represents the feasible 3D pose features after the first relation fusion, and F2 represents the feasible 3D pose features after the second relation fusion. M The feasible 3D pose feature after the Mth relation is fused;

[0157] A linear transformation layer is used to perform linear regression on F to obtain the keypoint sequence of the three-dimensional pose. from Select the 3D pose of the video frame to be estimated. This serves as the final result of human pose estimation.

[0158] Based on the above scheme, the present invention conducted the following experiments:

[0159] 1. Dataset and Evaluation Metrics

[0160] The experiment used the three-dimensional human pose estimation datasets Human3.6M, MPI-INF-3DHP, and HumanEva-I.

[0161] Human3.6M is currently the most representative 3D human assessment dataset (3D HPE, 3D Human Pose Estimation). This dataset consists of 3.6 million images captured by four simultaneous cameras at a frequency of 50 Hz, with 11 subjects performing 15 daily activities in an indoor environment. This invention trains the model on five scenarios (S1, S5, S6, S7, S8) and tests it on two scenarios (S9, S11). A commonly used evaluation metric, Mean Per Joint Position Error (MPJPE), is used to measure the average Euclidean distance (in millimeters) between the ground truth and estimated joints.

[0162] MPI-INF-3DHP is a large indoor and outdoor three-dimensional human pose dataset consisting of 1.3 million frames of images, containing more motion images compared to Human3.6M. To better evaluate the present invention, the same experimental parameter settings as PoseFormer

[10] and MHFormer were used, and common evaluation metrics were used for the experiment: 1) Mean Joint Error MPJPE; 2) Percentage of Correct Keypoints PCK (with a threshold of 150 mm); 3) Area Under Curve (AUC).

[0163] HumanEva-I is a small human pose dataset consisting of six daily activities performed by four subjects, such as walking, running, boxing, and greeting. The dataset contains 20,610 labeled images. This invention uses the same experimental parameter settings as ConvForme to experiment on the mean joint error (MPJPE) value.

[0164] 2. Experimental Environment and Parameters

[0165] The network proposed in this invention is implemented in PyTorch on a GeForce RTX 3090 GPU, trained end-to-end using the Lion optimizer, and the loss is calculated using Mean Squared Error (MSE) to minimize the error between the estimated pose and the true pose. The specific loss function is as follows:

[0166]

[0167] In the formula, and Let N and X represent the estimated and true values ​​of the 3D pose of joint i in the nth frame, respectively. Let the number of input joint video frames N be 351, the initial learning rate be 0.0001, the batch size be 128, the epoch be 10, the downsampling factor after each epoch be 0.95, and the downsampling factor after the 5th epoch be 0.5. Two experiments were conducted on the Human3.6M dataset: (1) using the two-dimensional labeled data (2d_gt) in the Human3.6M dataset as the model input; (2) using the two-dimensional pose estimation network CPN (Cascadedpyramid network). The two-dimensional joint coordinates (2d_cpn) obtained from the video image estimation in the Human3.6M dataset were used as the model input. On the MPI-INF-3DHP dataset, the two-dimensional labeled data given in the dataset were used as the model input.

[0168] 3. Experimental Results

[0169] 1) Experimental results on the Human 3.6M dataset

[0170] On the Human3.6M dataset, the experimental results of this invention were compared with those of various methods, using 2d_gt and 2d_cpn as model inputs respectively. The MPJPE evaluation values ​​are shown in Table 1. As can be seen from the table, this invention achieved better results in all cases. In the experiment using 2d_gt as model input, the MPJPE value obtained by this invention was 27.2 mm, a decrease of 10.8% compared to the MHFormer method. Compared to the ConvFormer method, the error was reduced by 2.6 mm. In the experiment using 2d_cpn as input, the MPJPE value obtained by this invention was 42.6 mm, a decrease of 0.4 mm compared to the MHFormer method and a decrease of 0.6 mm compared to the ConvFormer method. The experimental results show that this invention improves the accuracy of human pose estimation under various conditions, and the accuracy is even higher when using two-dimensional labeled data as model input.

[0171] Table 1 Comparison of MPJPE metrics for different methods on the Human 3.6M dataset

[0172]

[0173]

[0174] In this invention, data with a feasible number of three-dimensional poses of 2 are used. The bold text indicates the optimal result, "_" indicates the suboptimal result, and "-" indicates that the corresponding literature did not conduct the corresponding experiment.

[0175] This invention was experimentally compared with several other methods that generate multiple feasible 3D poses on the Human3.6M dataset using 2d_cpn as model input. The results are shown in Table 2. Since most methods use a one-to-many mapping strategy, this invention was fairly compared with these methods using the MPJPE error evaluation metric for the best feasible 3D pose. It can be seen that this invention uses fewer feasible 3D poses, only two. While reducing the number of feasible 3D poses generated, this invention also outperforms previous similar works, achieving the lowest error result. Furthermore, this invention uses a many-to-one mapping after the one-to-many mapping, ultimately determining a unique feasible 3D pose, which is more practical in real-world applications and greatly enhances its practical value.

[0176] Table 2 Comparison of MPJPE indices for different multiple feasible 3D pose methods on Human 3.6M

[0177]

[0178]

[0179] The bold font is the optimal result.

[0180] The training data sample size increases with the number of input video frames. Experiments were conducted on the Human3.6M dataset using frames other than 351 frames. The impact of the small sample dataset on model performance was verified by calculating the MPJPE value. The results are shown in Table 3. As can be seen from the table, as the training data sample size gradually decreases, the estimation error of this invention increases slowly, but it is still improved relative to MHFormer, proving that this invention is less affected by the amount of training data.

[0181] Table 3. Experimental results for different input video frame numbers

[0182]

[0183] In this invention, the number of feasible three-dimensional poses generated is 2, and the data unit is mm.

[0184] 2) Experimental results on the MPI-INF-3DHP dataset

[0185] To evaluate the generalization ability of the semantic information fusion network, experiments were conducted on the MPI-INF-3DHP dataset using a 9-frame two-dimensional pose sequence as model input, and compared with various methods. The results are shown in Table 4. This invention achieved optimal results across all evaluation metrics, with PCK, AUC, and MPJPE values ​​of 96.0, 69.8, and 43.0, respectively. Compared to the MHFormer method, PCK and AUC values ​​improved by 2.2 and 13.4, respectively, while the MPJPE value decreased by 15.0. Compared to the ConvFormer method, there was no improvement in PCK value, but the MPJPE value decreased by 10.6. The experimental results effectively demonstrate the generalization ability of this invention.

[0186] Table 4 Comparison of results of different methods on the MPI-INF-3DHP dataset

[0187]

[0188] 3) Experimental results on the HumanEva-I dataset

[0189] To further verify the impact of small sample datasets on network performance, three actions—walking, running, and hitting (boxing)—were selected as model inputs on the HumanEva-I dataset. The results are shown in Table 5. This invention achieved optimal results for most actions and suboptimal results for a small number of actions. The experimental results verify that this invention can also achieve good results on small sample datasets.

[0190] Table 5 Comparison of results for different methods on the HumanEva-I dataset

[0191]

[0192] In this invention, the number of feasible 3D poses generated is 2. The bold text represents the optimal result, "_" represents the suboptimal result, and "-" represents the corresponding literature that did not conduct the corresponding experiment.

[0193] 4) Visualization results comparison

[0194] Visualization experiments were conducted on the Human 3.6M dataset to compare the DC-GCT method, P-STMO method, MHFormer method, and the present invention. The results are as follows: Figure 5As shown, the arrows point to the deviations in the visualization results. (a2), (a3), (a4), and (a5) represent the human pose estimation results of the input image (a1) obtained using the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and this invention, respectively; (a6) represents the true pose of the input image (a1). (b2), (b3), (b4), and (b5) represent the human pose estimation results of the input image (b1) obtained using the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and this invention, respectively; (b6) represents the true pose of the input image (b1). Values; (c2), (c3), (c4), and (c5) are the human pose estimation results of the input image (c1) obtained based on the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and this invention, respectively; (c6) is the true pose value of the input image (c1); (d2), (d3), (d4), and (d5) are the human pose estimation results of the input image (d1) obtained based on the DC-GCT algorithm, P-STOM algorithm, MHFormer algorithm, and this invention, respectively; (d6) is the true pose value of the input image (d1). Figure 5 As can be seen from (a2), (a3), (a4), (a5), (a6), and (b2), (b3), (b4), (b5), (b6), the present invention is closer to the true pose in the case of self-occlusion of the person. Figure 5 (c1) Predictions are more accurate in complex human posture scenarios, due to... Figure 5 As can be seen from (d2), (d3), (d4), (d5), and (d6), the present invention provides a more detailed estimation of human joints.

[0195] The results of human pose estimation using network video are as follows Figure 6As shown, (a1) is the input image, (a2) is the human pose estimation result of (a1), (b1) is the input image, (b2) is the human pose estimation result of (b1), (c1) is the input image, (c2) is the human pose estimation result of (c1), (d1) is the input image, (d2) is the human pose estimation result of (d1), (e1) is the input image, (e2) is the human pose estimation result of (e1), (f1) is the input image, (f2) is the human pose estimation result of (f1), and (g1) is the input image. (g2) is the human pose estimation result of (g1), (h1) is the input image, (h2) is the human pose estimation result of (h1), (i1) is the input image, (i2) is the human pose estimation result of (i1), (j1) is the input image, (j2) is the human pose estimation result of (j1), (k1) is the input image, (k2) is the human pose estimation result of (k1), (l1) is the input image, (l2) is the human pose estimation result of (l1). It can be seen that none of the actions appeared in the Human3.6M dataset.

[0196] To further verify the model's performance under conditions of human self-occlusion and complex poses, the model was trained using the Human3.6M dataset, and experiments were conducted using data from the MPI-INF-3DHP dataset that included human self-occlusion and complex movements. The results are as follows: Figure 7 As shown, (a1) is an input image with self-occlusion and complex movements, (a2) is the human pose estimation result of (a1), (b1) is an input image with self-occlusion and complex movements, (b2) is the human pose estimation result of (b1), (c1) is an input image with self-occlusion and complex movements, (c2) is the human pose estimation result of (c1), and (d1) is an input image with self-occlusion and complex movements, (d2) is the human pose estimation result of (d1). It can be seen that the present invention can accurately estimate the three-dimensional human pose in this case.

[0197] In summary, based on the results of 3D human pose estimation from dataset videos and online videos, this invention can estimate human 3D pose relatively accurately and can effectively address situations where people self-occlude and have complex poses.

[0198] 4. Computational complexity analysis

[0199] This invention is based on the Transformer architecture, but extensive use of the Transformer structure can lead to problems such as excessive computational cost. To obtain the computational cost and parameter count of the model, 27 frames of 2D pose ground truth data were used as model input for computational cost analysis. Experiments were conducted on the parameter count (Params), model computational cost (FLOPs), and mean joint error (MPJPE) under three conditions: generating 4, 3, and 2 feasible 3D poses. The experimental results are shown in Table 6. As can be seen from Table 6, the model generating 2 feasible 3D poses exhibits the best performance and achieves the lowest MPJPE value. When the number of feasible 3D poses is 2, the correlation between the features of two feasible 3D poses can be learned to the greatest extent to assist semantic information fusion and enable the network to achieve the highest accuracy. At the same time, when the number of feasible 3D poses is 2, the number of parameters and computational cost are less, which has certain advantages in terms of lightweight model. In addition, as can be seen from Table 7, compared with the MHFormer method, this invention reduces the error value by 3.6 while significantly reducing the number of model parameters and the computational cost.

[0200] Table 6 Experimental results on the impact of the number of feasible 3D poses on network performance

[0201]

[0202]

[0203] Table 7 Comparison of computational and parameter counts with the MHFormer method.

[0204]

[0205] 5 Ablation Experiment

[0206] To verify the impact of each design in the proposed model, ablation experiments were conducted on the Human3.6M dataset.

[0207] 1) The impact of each module on network performance

[0208] To verify the effectiveness of each module in the network and better evaluate the impact of different modules on the accuracy of pose estimation, experiments were conducted using the ground truth of 2D pose values ​​from the Human3.6M dataset as model input under different conditions: (1) removal of the hierarchical feature extraction module; (2) removal of the feature refinement module; (3) removal of the hierarchical feature fusion module; (4) the complete network model. The experimental results are shown in Table 8. As can be seen from the table, the experimental results of removing the feature refinement module have the largest gap with the experimental results of the complete network model, proving that this module can make full use of the effective information in the temporal and spatial domains captured in the feasible 3D pose features. The refinement of feasible 3D pose features plays a significant role in improving the accuracy of human pose estimation. The experimental results of removing the hierarchical feature fusion module are closest to the experimental results of the complete network model. However, this module can capture and integrate complementary positional and temporal information from two feasible 3D pose features, effectively carrying out the dual-level semantic information transmission and fusion process, which plays a key role in the human pose estimation work and the improvement of prediction accuracy. The hierarchical feature extraction module generates two initial feasible 3D pose features containing different semantic information, which forms the basis for human pose estimation in this invention and thus plays a crucial role. The experimental results show that, with the combined effect of all modules, the semantic information fusion network achieves optimal overall performance, validating the effectiveness of each module.

[0209] Table 8 shows the experimental results regarding the impact of modules on network performance.

[0210]

[0211]

[0212] In this invention, 351 frames of data are used to generate 2 feasible 3D poses.

[0213] 2) Comparison of MESA and MSA performance in the network

[0214] To verify the effectiveness of MESA in the network, replacement experiments were conducted on MESA in the Hierarchical Feature Extraction (HFE) module and the Feature Refinement (FRM) module, respectively. The results are shown in Table 9. As can be seen from the table, compared to model M1, which uses MSA in both modules, models M2 and M3, which replace a single module with MESA, show improved performance, with errors decreasing by 0.4 and 0.9 respectively. This indicates that MESA is effective in the network, and its role is more crucial in the feature extraction module. Model M4, which replaces both modules with MESA, shows a 1.7 reduction in error, demonstrating that MESA can improve model computation performance and enhance the accuracy of human pose estimation.

[0215] Table 9 Comparison of MESA and MSA performance

[0216]

[0217] 3. Conclusion

[0218] This invention proposes a semantic information fusion network for monocular 3D human pose estimation. The aim is to maximize the fusion of feasible 3D pose features containing semantic information at different levels by learning the dependencies between feasible 3D pose features, thus reducing the low accuracy problem caused by insufficient learning during multi-solution fusion of human pose estimation. A hierarchical feature extraction module generates initial feasible 3D poses containing diverse information, providing semantic information at different levels. Through effective refinement of features, important information is strengthened, ensuring that the feasible 3D pose features used for the final fusion operation have stronger expressive power. An association calculation module and an association feature fusion mechanism can effectively fuse complementary information in feasible 3D pose features, fully utilizing global information to drive the model to perform more accurate human pose estimation. Experiments on the Human3.6M, MPI-INF-3DHP, and HumanEva datasets show that the optimal results are achieved when the number of feasible 3D poses is 2. Compared with other methods, this invention achieves lower average joint error, higher joint accuracy, lower parameter and computational cost, better avoids errors caused by self-occlusion and complex poses, and achieves higher accuracy and robustness.

[0219] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A three-dimensional human pose estimation method that integrates two-layer semantic information, characterized in that, include: Step 1: Obtain N video frames, including the video frame to be estimated and N-1 video frames within a preset time range of the video frame to be estimated, and obtain the two-dimensional pose sequence of human joints corresponding to the N video frames. The two-dimensional pose sequence X of human joints includes the coordinates of human joints in each video frame, and J is the number of human joints in each video frame. Step 2: Set the total number of feasible pose features M, and process the two-dimensional pose sequence X of human joints through the hierarchical feature extraction module HFE to obtain M initial feasible three-dimensional pose features. Step 2.1: Concatenate the joint coordinates (x, y) of all video frames in the two-dimensional pose sequence X of the human joints to obtain the concatenated joint information. First sequence information is added to each joint in the two-dimensional pose sequence X of human joints. Set the initial value of parameter m to 1; Step 2.2: Transfer the first sequence information Joint information after splicing The fusion process is performed to obtain a sequence of joints with fused spatial information. Specifically, this is achieved through the following formula: ； ； in, For the m-th intermediate joint sequence, X m-1 Let LN(˙) be the m-1th initial feasible 3D pose feature, and LN(˙) be the LayerNorm layer. The joint point sequence obtained after fusing spatial information is input into the encoder of the hierarchical feature extraction module, and the first feasible 3D pose feature is obtained through MESA encoding, specifically through the following formula: ； ； in, Let m be the m-th intermediate feasible 3D pose feature. This is the sequence of joints obtained after fusing the m-th spatial information. Let m be the first feasible 3D pose feature, MESA be multi-head augmented self-attention, and MLP be a multilayer perceptron; The encoder output features are obtained by adding the intermediate joint sequence, the joint point sequence obtained after fusing spatial information, and the first feasible 3D pose features, which is specifically achieved through the following formula: ； in, The feature output by the m-th encoder; Perform a one-dimensional convolution on the features output by the encoder to obtain the initial feasible three-dimensional pose features X. m Specifically, this is achieved through the following formula: ； Among them, X m Let m be the m-th initial feasible 3D pose feature, GELU be the activation function, and Conv be the one-dimensional convolution; Step 2.3: Determine if m is greater than M. If m is greater than M, obtain M initial feasible 3D pose features. If m is less than or equal to M, increment the value of m by one and return to step 2.

2. Step 3: Process each initial feasible 3D pose feature through the feature refinement module (FRM) to obtain M feasible 3D pose features refined by the feature refinement module, and the convergent representation A of the feasible 3D pose features. Step 3.1: For each initial feasible 3D pose feature, transform it through bitwise operations and linear transformation operations to obtain high-dimensional features, and extract the second order information of each video frame in the sequence. By embedding high-dimensional features, feasible 3D pose features with embedded temporal and positional information are obtained. The feasible 3D pose features after embedding time and location information The input is fed into MESA, and the output of MESA is added to the feasible 3D pose features after embedding time and position information to obtain the first intermediate feasible 3D pose features. Then, M first intermediate feasible 3D pose features are calculated. All first intermediate feasible 3D pose features are concatenated to obtain the hybrid feasible 3D pose features Z. Step 3.2: Process the hybrid feasible 3D pose features Z using a multilayer perceptron (MLP), and add the processing result to the hybrid feasible 3D pose features Z to obtain a convergent representation after exchanging key information of different feasible 3D pose features. Specifically, this is achieved through the following formula: ； right The features are split into M feasible 3D pose features, which are represented by the following formula: ； in, For splitting operations, This is the first feasible 3D pose feature after splitting. This is the second feasible 3D pose feature after splitting. The Mth feasible 3D pose feature after splitting; The decomposed feasible 3D pose feature corresponding to each initial feasible 3D pose feature is added to the feasible 3D pose feature corresponding to that initial feasible 3D pose feature to obtain the feasible 3D pose feature after refinement by the feature refinement module, which is specifically expressed by the following formula: ； Among them, A m The m-th feasible 3D pose feature after refinement by the feature refinement module; This yields M feasible 3D pose features refined by the feature refinement module. All these features are then concatenated to obtain a convergent representation A of the feasible 3D pose features, specifically expressed by the following formula: ； Where A1 is the first feasible 3D pose feature refined by the feature refinement module, and A2 is the second feasible 3D pose feature refined by the feature refinement module. M The Mth feasible 3D pose feature after refinement by the feature refinement module; Step 4: The HFF module is used to process the M feasible 3D pose features refined by the feature refinement module and the convergent representation A of the feasible 3D pose features to obtain the final human pose estimation result of the video frame to be estimated.

2. The three-dimensional human pose estimation method fusing dual-layer semantic information according to claim 1, characterized in that, In step 2.2 Specifically, it is calculated using the following formula: right Perform a linear mapping to obtain query Q, key K, and value V. Based on query Q, key K, and value V, calculate... Specifically, this is achieved through the following formula: ； ； ； Where d represents the dimension of query Q, key K, and value V, and Attention ＇ (Q,K,V) represents the output result. This is the scaled dot product of Q and K. To The result after performing Sum summation and Sigmoid activation.

3. The three-dimensional human pose estimation method fusing dual-layer semantic information according to claim 1, characterized in that, Step 3.1 specifically includes: The second order information of each video frame in the sequence By embedding high-dimensional features, feasible 3D pose features with embedded temporal and positional information are obtained. Specifically, this is achieved through the following formula: ； in, Z represents the m-th feasible 3D pose feature embedded with temporal location information. m For the m-th high-dimensional feature, ; The feasible 3D pose features embedded with time and position information are input into MESA. The output of MESA is added to the feasible 3D pose features embedded with time and position information to obtain the first intermediate feasible 3D pose features, which is achieved through the following formula: ； in, This represents the m-th first intermediate feasible 3D pose feature; Then, M first intermediate feasible 3D pose features are calculated. All first intermediate feasible 3D pose features are concatenated to obtain the hybrid feasible 3D pose feature Z, which is achieved through the following formula: ； in, This is the first feasible 3D pose feature in the first intermediate stage. This is the second first intermediate feasible 3D pose feature. This is the Mth first intermediate feasible 3D pose feature.

4. The three-dimensional human pose estimation method fusing dual-layer semantic information according to claim 1, characterized in that, Step 4 specifically includes: The Association Computation Module (ACM) processes the convergent representation A of feasible 3D pose features to obtain the association features R. Specifically, the ACM learns the dependencies between feasible 3D pose features, and obtains the association features R through two consecutive one-dimensional convolutions and a BatchNorm operation, as expressed by the following formula: ； The cross-attention algorithm is used to process the associated features R and the feasible 3D pose features refined by the feature refinement module, resulting in feasible 3D pose features after relational fusion, which is expressed by the following formula: ； Among them, A m+1 F is the (m+1)th feasible 3D pose feature refined by the feature refinement module. m The m-th relation is the feasible 3D pose feature after fusion. Then, M feasible 3D pose features after relation fusion are obtained. Through splicing and multilayer perceptron, all feasible 3D pose features after relation fusion are processed to obtain the final convergent representation F of feasible 3D pose features, which is specifically achieved through the following formula: ； ； Where F' is the representation of the feasible 3D pose features after the fusion of M relations. F1 represents the feasible 3D pose features after the first relation fusion, and F2 represents the feasible 3D pose features after the second relation fusion. M The feasible 3D pose feature after the Mth relation is fused; A linear transformation layer is used to perform linear regression on F to obtain the keypoint sequence of the three-dimensional pose. ,from Select the 3D pose of the video frame to be estimated. This serves as the final result of human pose estimation.

Images