Three-dimensional attitude estimation method combining global modeling and local refinement

Abstract

The invention discloses a three-dimensional attitude estimation method combining global modeling and local refinement, which comprises the following steps of: firstly, extracting a two-dimensional attitude sequence by using a human body video data set; secondly, inputting the two-dimensional attitude sequence into a structural modeling main branch, modeling a spatial topological relation and a time sequence dynamic state between joints, and outputting a global three-dimensional attitude sequence; and inputting the two-dimensional attitude sequence into a local refining branch, modeling dynamic change and detail information of a local area, and outputting a local three-dimensional attitude sequence. And finally, fusing the global three-dimensional attitude sequence and the local three-dimensional attitude sequence, generating a three-dimensional attitude sequence output, and completing three-dimensional attitude estimation. According to the method, the problem of insufficient cross-frame information transmission in a traditional method is relieved, and the accuracy and robustness of attitude estimation in a dynamic complex scene are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer graphics and deep learning, and particularly relates to a three-dimensional pose estimation method combining global modeling and local refinement. BACKGROUND

[0002] Three-dimensional human pose estimation is an important research direction in the field of computer vision, and is widely used in virtual reality, motion analysis, behavior recognition and other scenarios. The task aims to predict the positions of human key points in three-dimensional space from a single image or video sequence, and its accuracy directly affects the effectiveness of downstream applications.

[0003] At present, there are mainly the following methods in this field: one is a direct modeling method based on image regression, which uses a convolutional neural network (CNN) to directly regress three-dimensional joint coordinates from the input image; the other is a method based on two-dimensional key point regression, which first extracts two-dimensional key points in the image, and then predicts three-dimensional coordinates through a regression network. These methods have made certain progress on static images, but still have great limitations in time series modeling and structure perception.

[0004] In order to further enhance the human structure modeling capability, some methods introduce graph neural networks (GNN), which build the human skeleton as a graph structure, model the connection relationship between human joints through graph convolution (GCN) or graph attention mechanism (GAT), and thus improve the network's understanding ability of human topological structure. Representative works such as GAST-Net (graph attention spatio-temporal network) and ST-GCN (spatio-temporal graph convolutional network) can effectively model the local dependency of joints. However, this kind of method still faces the problem of insufficient modeling ability when dealing with global dependency across joints or complex motion changes.

[0005] In recent years, modeling methods based on Transformer have been gradually applied to the task of three-dimensional pose estimation. Related methods usually adopt a dual-flow structure that decouples space and time to capture long-term temporal dependencies and spatial feature information. For example, methods such as MHFormer (multi-hypothesis Transformer network) enhance the time modeling capability through temporal self-attention and achieve good performance in video pose estimation. However, most of these methods do not explicitly use the graph information of human structure, resulting in insufficient modeling of local constraints.

[0006] Existing three-dimensional human pose estimation methods still have certain limitations in structure modeling, timing modeling and pose prediction mechanism. Although the graph convolution method represented by ST-GCN and GAST-Net can model the topological structure between human joints, due to its dependence on fixed connection graph convolution operator, it lacks flexible attention mechanism and is difficult to adapt to the change of dynamic dependence relationship between joints, which limits the global pose structure modeling capability. On the other hand, methods such as MHFormer introduce a global modeling structure based on Transformer, which can enhance the timing modeling capability to a certain extent, but such methods do not combine the prior graph structure of human skeleton, resulting in insufficient ability to model spatial structure. In addition, existing methods mostly use single-stage direct pose regression strategy, lack of multi-stage optimization mechanism with controllable process, sensitive to occlusion, blur or noise interference in the input, and have weak robustness. Although diffusion models have shown their potential in action generation and other fields in recent years, existing solutions have not effectively applied them to the three-dimensional pose estimation task, and lack a step-by-step optimization framework that adapts to human structure modeling needs. SUMMARY

[0007] To solve the above problems, the present application proposes a three-dimensional pose estimation method combining global modeling and local refinement, wherein the main branch is composed of a graph attention network and a double-branch Transformer, which respectively model the spatial structure dependence and temporal dynamic evolution process of the human skeleton, and extract structure-aware and time-consistent feature representations from two-dimensional pose sequences. At the same time, the local refinement branch takes the same two-dimensional pose sequence as input, focuses on the dynamic details and noise-sensitive joints in the local area, and supplements the deficiencies of the main branch in detail restoration through independent modeling. Finally, the output results of the two branches are fused in the feature or prediction layer, effectively integrating global structure and local detail information, thereby realizing more stable and fine three-dimensional pose estimation.

[0008] The technical scheme adopted by the present application to achieve the above-mentioned purposes is a three-dimensional pose estimation method combining global modeling and local refinement. The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0009] The three-dimensional pose estimation method combining global modeling and local refinement includes the following steps:

[0010] Step 1: Use a human video dataset to extract two-dimensional pose sequences as the common input basis for the structure modeling main branch and the local refinement branch.

[0011] A public human video dataset is selected, which contains a large number of human motion sequences in a front view, and has frame-level annotation information. For each video sequence, a two-dimensional pose estimator is used to recognize the pose of each frame image. The cascaded pyramid network (CPN) is used as the two-dimensional pose estimator, and the multi-scale feature fusion and cascaded optimization mechanism are used to analyze the two-dimensional pose structure of the human body. The two-dimensional poses recognized in each frame are organized in chronological order to form a two-dimensional pose sequence with continuous time sequence information.

[0012] Step 2, the above two-dimensional pose sequence is input into the structure modeling main branch of the application, which is composed of a graph attention network and a double-branch Transformer structure, and models the spatial topological relationship and time sequence dynamics between joints, and outputs a global three-dimensional pose sequence.

[0013] The structure modeling main branch: the spatial dependence between joints is modeled by using the graph attention network, and the spatial and temporal dimension features are extracted by combining the double-branch Transformer.

[0014] The structure modeling main branch aims to extract the spatial structure features and time dynamic features of the human body from the two-dimensional pose sequence. The branch mainly consists of two key modules: a graph attention network module and a double-branch Transformer module.

[0015] Firstly, the graph attention network constructs the two-dimensional pose in each frame into a human skeleton graph, and each key point in the human skeleton graph is taken as a graph node, and the anatomical connection relationship between adjacent nodes constitutes the edge of the static graph. On this basis, the graph attention mechanism is introduced, and the edge weight is dynamically calculated according to the correlation between the node features, so as to enhance the modeling ability of the structural changes and non-local dependence between joints. By weighting and aggregating the features of adjacent nodes in the human skeleton graph, frame-level features with structure perception ability are obtained. Subsequently, the frame-level features are input into the double-branch Transformer module for further spatio-temporal modeling: the output of the spatial attention branch is the spatial attention The output of the time attention branch is the time attention The element-wise summed fusion is The spatial-temporal information is cooperatively integrated. Subsequently, the fused features are input into a residual feedforward network (FFN), which is composed of a standardization layer LayerNorm, a layer of lightweight convolution and an activation function in cascade. The residual feedforward network outputs high-dimensional features with spatio-temporal collaborative expression ability. Finally, the high-dimensional features are sent to a multilayer perceptron (MLP) for nonlinear transformation and regression operation, and a global three-dimensional pose sequence is output.

[0016] Step 3: Local Refinement Branch: Taking the original 2D pose sequence as input, focus on modeling the dynamic changes and details of local areas to compensate for the potential deficiencies of the main branch in fine-grained modeling, and output local 3D pose sequence.

[0017] The local refinement branch takes the original two-dimensional pose sequence as input, and each frame contains the (x,y) coordinates of J key points. The whole is composed of a local structure-aware encoder and a multi-scale temporal modeling module connected in series.

[0018] First, the local structure-aware encoder takes the 2D pose of each frame as input, which contains a set of sequentially arranged joint coordinates. Each joint coordinate is constructed into a unit patch (keypoint block) and mapped to a high-dimensional feature representation through a Patch Embedding layer. Specifically, a learnable linear projection layer is used to map each 2D coordinate (x, y) to an embedding vector of dimension C. Simultaneously, spatial position information is introduced. A learnable position vector is assigned to each joint through Spatial Position Embedding, also of dimension C, and is element-wise added to the high-dimensional feature representation obtained through Patch Embedding to inject explicit position priors. After this embedding process, the initial encoded features for each joint are obtained.

[0019] The initial encoded features of these joints are then sequentially input into the Transformer Encoder. To enhance the structural modeling capability between keypoints, a structure-aware local self-attention mechanism is introduced. The input 2D pose is extracted by a CPN network, where the order and topology of the keypoints are explicitly given. During encoding, a local receptive range is defined for each keypoint based on the skeleton connectivity, allowing attention interactions only with its directly connected adjacent keypoints in the anatomical structure. Through this local attention mechanism, local features are obtained.

[0020] Subsequently, this local feature is fed into a multi-scale temporal modeling module, which contains three parallel one-dimensional dilated convolution branches with progressively increasing kernel size and dilation rate. The outputs of the three convolution branches are concatenated along the channel dimension to form a fused temporal feature, which is then input into a channel recalibration module for dynamic enhancement. The channel recalibration module employs a squeeze-and-excitation mechanism to generate channel attention weights s∈[0,1]. CThe resulting channel attention weights are then multiplied channel-by-channel (F_scale) with the fused features formed by concatenating the three temporal branches. This explicitly weights the temporal features of each channel, highlighting key information and suppressing redundant interference. This weighted result is then passed through a 1×1 convolutional layer to compress the number of channels and residually connected to the input local features, resulting in a structurally enhanced temporal feature representation. Finally, this representation is mapped to a local 3D pose sequence through two fully connected regression heads, serving as the final output of this branch.

[0021] Step 4: Fuse the global 3D attitude sequence and the local 3D attitude sequence to generate a stable and accurate 3D attitude sequence output, thus completing the 3D attitude estimation.

[0022] The fusion stage directly performs element-wise weighted averaging on the 3D pose sequences output from the two branches, thus balancing global coherence and local accuracy. To improve the accuracy and stability of 3D pose estimation, this method introduces three key loss terms for joint optimization: weighted average joint position error L... W Time consistency loss L T and the average velocity error per joint L M .

[0023] Beneficial effects of this invention:

[0024] This invention combines spatial and temporal dual-path Transformer modules to form a structure-temporal collaborative modeling path. It designs a refined branch for local region modeling, focusing on handling dynamic details and occluded areas that the main branch may ignore. It exhibits stable and consistent pose estimation capabilities in different lighting conditions, background interference, and scenes with large variations in pose complexity. It can accurately reconstruct human poses under different actions, demonstrating high accuracy and realism. Attached Figure Description

[0025] Figure 1 This is a schematic flowchart of the method of the present invention;

[0026] Figure 2 This is an overall block diagram of a 3D pose estimation method that combines global modeling and local refinement;

[0027] Figure 3 A schematic diagram of the spatiotemporal modeling process;

[0028] Figure 4 This is a visualization of the prediction results of the present invention on multiple publicly available 3D human pose estimation datasets;

[0029] Figure 5 This is a diagram showing the results of testing the present invention on real-world human images. Detailed Implementation

[0030] Existing 3D human pose estimation methods often suffer from insufficient accuracy when dealing with rapid movements, occlusion, and fine-grained structural representations. This invention proposes a bi-branch 3D pose modeling method, modeling human pose from both global structure and local dynamics perspectives. The main branch models global topology and temporal dependencies based on graph attention and a space-time Transformer, while the local branch introduces local attention and multi-scale temporal modeling mechanisms, significantly improving the accuracy and robustness of pose estimation in dynamic and complex scenes. This invention also proposes a pose optimization framework that dynamically optimizes joint positions and poses through deep learning, eliminating mesh distortion and positioning errors caused by complex movements and rapid pose changes. This framework can accurately optimize poses when handling dynamic movements, improving the accuracy and detail of mesh reconstruction.

[0031] This invention innovatively introduces a graph attention mechanism to model the spatial structural relationships between key points, combining it with a dual-path Transformer module (spatial and temporal) to form a structure-temporal collaborative modeling path. This structure not only captures the anatomical connections between skeletons but also enhances the temporal continuity of key points, significantly alleviating the problem of insufficient cross-frame information transmission in traditional methods.

[0032] This invention designs a refinement branch for local region modeling, focusing on handling detailed dynamics and occluded regions that the main branch may overlook. This branch employs a local spatial attention mechanism and a lightweight multi-scale temporal convolution module to supplement the detailed poses of key action frames from the original 2D pose sequence, thereby improving the stability and accuracy of overall pose estimation in complex action sequences.

[0033] To achieve the above objectives, the present invention employs a 3D pose estimation method that combines global modeling and local refinement. The invention will be described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] like Figure 1 As shown, the 3D pose estimation method combining global modeling and local refinement includes the following steps:

[0035] Step 1: Use the human video dataset to extract two-dimensional pose sequences, which serve as the common input basis for the main branch of structural modeling and the local refinement branch.

[0036] To support 3D human pose estimation modeling, this invention selects the publicly available human video datasets Human3.6M and MPI-INF-3DHP as input data sources. These datasets contain a large number of human motion sequences from a frontal viewpoint, with frame-level annotation information, making them suitable for extracting continuous human motion trajectories from videos for subsequent network training and evaluation.

[0037] For each video sequence, a two-dimensional pose estimator is used to perform pose recognition frame by frame. This invention uses a Cascaded Pyramid Network (CPN) as the two-dimensional pose estimator. Leveraging its multi-scale feature fusion and cascaded optimization mechanism, it can robustly resolve the two-dimensional pose structure of the human body even under complex occlusion and pose changes. The two-dimensional poses identified in each frame are organized chronologically to form a two-dimensional pose sequence with continuous temporal information.

[0038] Step 2: The aforementioned two-dimensional pose sequence is input into the structural modeling main branch of this invention, which consists of a graph attention network and a two-branch Transformer structure. This branch is primarily used to model the spatial topological relationships and temporal dynamics between joints, outputting a global three-dimensional pose sequence. Through a unified two-dimensional pose input, the two branches maintain differentiated modeling strategies while achieving feature sharing and complementarity, effectively improving the overall accuracy and robustness of three-dimensional pose estimation.

[0039] like Figure 2 The diagram shows the overall block diagram of the 3D pose estimation method combining global modeling and local refinement in this invention. The block diagram includes the following inputs: a 2D pose sequence; graph attention (GAT); a spatial-temporal transformer (STFormer); spatial attention; a multi-scale temporal convolution module; and a multilayer perceptron (MLP). Steps 2 to 4 represent the overall process of 3D pose estimation combining global modeling and local refinement.

[0040] Step 2.1: Main branch of structural modeling: Use graph attention network to model the spatial dependency between joints, and combine it with a two-branch Transformer to extract spatial and temporal features respectively.

[0041] The structural modeling main branch aims to extract the spatial structural features and temporal dynamic features of the human body from two-dimensional pose sequences. This branch mainly consists of two key modules: a graph attention network module and a two-branch Transformer module.

[0042] First, the graph attention network constructs a human skeleton graph from the 2D pose of each frame. Each keypoint in the human skeleton graph serves as a graph node, and the anatomical connections between adjacent nodes form the edges of the static graph. Based on this, a graph attention mechanism is introduced to adaptively and dynamically calculate edge weights according to the correlation between node features, enhancing the modeling ability for structural changes and non-local dependencies between joints. By weighted aggregation of the features of adjacent nodes in the human skeleton graph, frame-level features with structural awareness are obtained for each frame. Subsequently, these frame-level features are input into the two-branch Transformer module for further spatiotemporal modeling: specifically, the spatial attention branch (AttentionJ) first rearranges the frame-level features into a shape [B,T,J,C], where B refers to the batch, T refers to the number of time steps, J refers to the number of keypoints, and C refers to the number of channels. Under this representation, the feature vectors of each keypoint in each frame constitute a spatial sequence. A multi-head self-attention (MHSA) mechanism is used to calculate the query (Q), key (K), and value (V) vectors respectively, performing interactive modeling on all relevant nodes. This process captures the spatial structural dependencies and nonlocal responses between different joints. The temporal attention branch (AttentionT) rearranges frame-level features into a shape [B,J,T,C], where B represents the batch size, J represents the number of keypoints, T represents the number of time steps, and C represents the number of channels. Under this representation, the feature sequence of each keypoint at different time steps is treated as a temporal series input. A multi-head self-attention (MHSA) mechanism is used to model the trajectory of each joint in the temporal dimension, capturing motion change patterns and local temporal context dependencies through query (Q), key (K), and value (V) vectors. The output of the spatial attention branch is the spatial attention... The output of the temporal attention branch is temporal attention. Through element-by-element addition and fusion, This achieves the coordinated integration of spatial and temporal information. Subsequently, the fused features are input into a residual feed-forward network (FFN), which consists of a normalized layer (LayerNorm), a lightweight convolutional layer, and a cascaded activation function. This not only enhances the nonlinear expressive power of the features but also facilitates cross-dimensional feature transformation and reshaping. The residual feed-forward network outputs high-dimensional features with coordinated spatiotemporal expressive power. For example... Figure 3 As shown, the two branches model spatial structure dependence and temporal dynamic features respectively, and are finally integrated in the fusion module into a high-dimensional feature representation with spatiotemporal co-expressive capabilities. Finally, the high-dimensional features are fed into a multilayer perceptron (MLP) for nonlinear transformation and regression operations, outputting a global 3D pose sequence.

[0043] The formula for time attention:

[0044]

[0045] Q T Query matrix: Represents the query representation over time. K T The key matrix represents the keys along the time dimension. V T The Value matrix represents the values ​​over time. T : Represents the feature dimension in the time dimension, used to scale the dot product (to avoid excessive gradient).

[0046] The formula for spatial attention:

[0047]

[0048] Q J The query matrix represents the query representation along the keypoint dimension (Joint). K J The key matrix represents the keys along the keypoint dimension. V J The Value matrix represents the values ​​along the joint dimension. J : Represents the dimension of the keypoint features, used as a scaling factor.

[0049] Step 3: Local Refinement Branch: Taking the original 2D pose sequence as input, focus on modeling the dynamic changes and details of local areas to compensate for the potential deficiencies of the main branch in fine-grained modeling, and output local 3D pose sequence.

[0050] The local refinement branch takes the original two-dimensional pose sequence as input, and each frame contains the (x,y) coordinates of J key points. The whole is composed of a local structure-aware encoder and a multi-scale temporal modeling module connected in series.

[0051] First, the local structure-aware encoder (see...) Figure 2The input is based on the 2D pose of each frame, which contains a set of sequentially arranged joint coordinates. Each joint coordinate is constructed into a unit patch (keypoint patch) and mapped to a high-dimensional feature representation through a PatchEmbedding layer. Specifically, we use a learnable linear projection layer to map each 2D coordinate (x, y) into an embedding vector of dimension C. Spatial position information is also introduced; a learnable position vector is assigned to each joint through Spatial Position Embedding, also of dimension C, and this vector is element-wise added to the high-dimensional feature representation obtained through Patch Embedding to inject explicit position priors. After this embedding process, the initial encoded features for each joint are obtained.

[0052] The initial encoded features of these joints are then sequentially input into the Transformer Encoder. To enhance the structural modeling capability between keypoints, we introduce a structure-aware local self-attention mechanism. The input 2D pose is extracted by a CPN network, where the keypoint order and topology are explicitly given (e.g., the 17 keypoints and their connections defined in the Human3.6M dataset), thus eliminating the need to reconstruct the skeleton map. During encoding, a local awareness range is defined for each keypoint based on the skeleton connections, allowing attention interactions only with its directly connected adjacent keypoints in the anatomical structure. Specifically, in the self-attention computation, a fixed adjacency matrix A∈0,1 is first defined based on the skeleton topology. J×J A ij =1 indicates that the i-th joint is connected to the j-th joint. Subsequently, after calculating the similarity score between the Query and the Key, only A... ij Positions with a value of 1 retain attention computation, while other positions are set to negative infinity to block interaction paths of unstructured connections. This operation can be achieved using a structure mask, i.e., if A ij If the value is 1, then the attention score is calculated; otherwise, the position is set to -∞. This structure-aware mechanism, by introducing skeleton connection constraints, ensures that the attention mechanism only propagates within the local neighborhood allowed by the joint structure, effectively improving local modeling capabilities and suppressing global redundancy interference. Local features are obtained through the aforementioned local attention mechanism.

[0053] Q=XW_Q, K=XW_K, V=XW_V

[0054] X: Input 2D pose embedding features, with dimensions J×C, where J is the number of keypoints and C is the number of channels; W_Q, W_K, W_V: Represent the learnable linear transformation matrices, respectively; Q, K, V: Represent Query, Key, and Value, respectively, each with dimensions J×d, where d is the attention dimension.

[0055]

[0056] S: Attention score matrix, representing the similarity between each keypoint; d: Attention embedding dimension, used for scaling to prevent excessive gradients.

[0057]

[0058] A ij =1, indicating that the i-th keypoint is connected to the j-th keypoint, and attention is retained. S ij q: Attention score of the i-th keypoint to the j-th keypoint i : The query vector of the i-th key point, k j : Key vector of the j-th key point, d: Dimension of the embedding vector in the attention mechanism, used for scaling.

[0059] Attention(Q,K,V)=softmax(S)·V

[0060] Softmax(S): The degree of attention each location gives to other locations; V: Represents the value.

[0061] Subsequently, this local feature is fed into the multi-scale temporal modeling module (see...). Figure 2 This module contains three parallel one-dimensional dilated convolution branches with kernel sizes of 3, 5, and 9, respectively, and dilation rates that increase sequentially to cover different time scales and capture short-term mutations and medium-term dynamic trends. The outputs of the three convolution branches are concatenated along the channel dimension to form a fused temporal feature, which is then input into the channel recalibration module for dynamic enhancement. The channel recalibration module employs a squeeze-and-excitation mechanism, first performing spatial compression on each channel through a global average pooling (F_sq) operation to generate a channel descriptor z∈R. C This reflects the importance statistics of each channel. Next, the channel descriptor is fed into a two-layer fully connected network (F_ex): the first layer compresses the dimension from C to C / r (where r is the compression ratio, usually set to 16), performs ReLU activation, and then restores it to C dimensions. Finally, a sigmoid function is used to generate normalized channel attention weights s∈[0,1]. CThe resulting channel attention weights are then multiplied channel-by-channel (F_scale) with the fused features formed by concatenating the three temporal branches. This explicitly weights the temporal features of each channel, highlighting key information and suppressing redundant interference. This weighted result is then passed through a 1×1 convolutional layer to compress the number of channels and residually connected to the input local features, resulting in a structurally enhanced temporal feature representation. Finally, this representation is mapped to a local 3D pose sequence through two fully connected regression heads, serving as the final output of this branch.

[0062] Step 4: Fuse the global 3D attitude sequence and the local 3D attitude sequence to generate a stable and accurate 3D attitude sequence output, thus completing the 3D attitude estimation.

[0063] To generate stable and accurate 3D pose estimation results, this method employs a result-level fusion strategy after the main branch and local refinement branch each complete their 3D pose predictions. The main branch focuses on modeling the global structure and long-term temporal dependencies of the human skeleton, providing a coherent global 3D pose sequence. Meanwhile, the local refinement branch focuses on short-term local dynamic changes, enhancing the model's ability to capture detailed movements and rapid local transformations. The fusion stage directly performs an element-wise weighted average of the 3D pose sequences output by the two branches, thus balancing global coherence and local accuracy. This fusion strategy, while maintaining the global modeling advantages of the backbone network, further improves the model's performance in local detail restoration and dynamic continuity, effectively enhancing the robustness and accuracy of the final 3D pose sequence under complex movements, rapid motion, and occlusion conditions.

[0064] To improve the accuracy and stability of 3D pose estimation, this method introduces three key loss terms for joint optimization: weighted average joint position error L. W Time consistency loss L T and the average velocity error per joint L M Among them, L W A weighting mechanism is introduced based on traditional joint position error to differentiate the importance of different joints, thereby improving the prediction accuracy of key joints; L T Used to constrain the amplitude of pose changes between adjacent frames, ensuring the continuity of the prediction sequence in the time dimension and effectively mitigating inter-frame jitter; L M This strengthens the modeling ability for motion rhythm and dynamic consistency by constraining the error between the predicted and actual posture at the velocity level. These three factors work together in the posture estimation process, enhancing the model's robustness and expressive power from three dimensions: spatial structure, temporal smoothness, and motion rationality. The loss formula is as follows:

[0065]

[0066] Where N represents the N joints of the human body, T represents the number of frames in the sequence, and p i,j This represents the predicted 3D pose of the i-th joint in the j-th frame, gt i,j It represents a realistic 3D pose.

[0067]

[0068] Y represents the predicted joint position at T time steps, N represents the N joints of the human body, and T represents the T time steps.

[0069]

[0070] N: Number of joints, T: Number of frames in time, Δp i,t : represents the prediction velocity of the i-th sample in frame t, Δgt i,t : represents the actual velocity of the i-th sample in frame t.

[0071] Figure 4 This presentation showcases the visualized prediction results of our proposed method on multiple publicly available 3D human pose estimation datasets, covering both standard indoor environments (such as Human3.6M) and outdoor data in natural scenes (such as 3DPW). Each column corresponds to the original image input and its 3D pose reconstruction result. As can be seen from the figures, our method exhibits stable and consistent pose estimation capabilities under different lighting conditions, background interference, and scenarios with significant variations in pose complexity.

[0072] also, Figure 5 The results of testing this invention on real-world human images are shown. Given a set of human images collected online, the invention generated corresponding 3D poses. Analysis of the results shows that our method can accurately reconstruct human poses under different actions, demonstrating high accuracy and realism.

Claims

1. A 3D pose estimation method combining global modeling and local refinement, characterized in that, Includes the following steps: Step 1: Extract two-dimensional pose sequences using a human video dataset; Step 2: Input the two-dimensional attitude sequence into the main branch of the structure modeling, model the spatial topological relationship and temporal dynamics between joints, and output the global three-dimensional attitude sequence; Step 3: Input the 2D pose sequence into the local refinement branch, model the dynamic changes and detailed information of the local region, and output the local 3D pose sequence; Step 4: Fuse the global 3D attitude sequence and the local 3D attitude sequence to generate a 3D attitude sequence output, thus completing the 3D attitude estimation.

2. The 3D pose estimation method combining global modeling and local refinement according to claim 1, characterized in that, The specific implementation process of extracting the two-dimensional pose sequence is as follows: obtain a human video dataset containing human action sequences from a frontal view, with frame-level annotation information; for each video sequence, use a two-dimensional pose estimator to perform pose recognition on each frame of the image: use a cascaded pyramid network (CPN) as a two-dimensional pose estimator, and parse out the two-dimensional pose of the human body through multi-scale feature fusion and cascaded optimization mechanism. The two-dimensional poses identified in each frame are organized in chronological order to form a two-dimensional pose sequence with continuous temporal information.

3. The 3D pose estimation method combining global modeling and local refinement according to claim 1, characterized in that, The main branch of the structural modeling is implemented as follows: The main branch of structural modeling consists of an attention network module and a two-branch Transformer module. It uses a graph attention network to model the spatial dependency between joints and combines the two-branch Transformer to extract spatial and temporal features respectively. First, the graph attention network constructs a human skeleton graph from the 2D pose of each frame. Each keypoint in the human skeleton graph serves as a graph node, and the anatomical connections between adjacent nodes form the edges of the static graph. A graph attention mechanism is introduced to adaptively and dynamically calculate edge weights based on the correlation between node features, enhancing the modeling ability for structural changes and nonlocal dependencies between joints. Through weighted aggregation of adjacent nodes in the human skeleton graph, frame-level features with structural awareness are obtained for each frame. Subsequently, these frame-level features are input into a two-branch Transformer module for spatiotemporal modeling, yielding feature X. att Subsequently, feature X att The input is fed into a residual feedforward network, which consists of a normalized layer LayerNorm, a lightweight convolution, and a cascaded activation function. The residual feedforward network outputs high-dimensional features with spatiotemporal co-expressive capabilities. Finally, the high-dimensional features are fed into a multilayer perceptron for nonlinear transformation and regression operations, outputting a global three-dimensional pose sequence.

4. The 3D pose estimation method combining global modeling and local refinement according to claim 3, characterized in that, The spatiotemporal modeling is specifically implemented as follows: The spatial attention branch, AttentionJ, first rearranges the frame-level features into a shape [B,T,J,C], where B refers to the batch, T refers to the number of time steps, J refers to the number of keypoints, and C refers to the number of channels. Under this representation, the feature vectors of each keypoint in each frame constitute a spatial sequence. The multi-head self-attention mechanism (MHSA) is used to calculate the query Q, key K, and value V vectors respectively, and to perform interactive modeling on all key nodes. The temporal attention branch, AttentionT, rearranges the frame-level features into a shape [B,J,T,C]. Under this representation, the feature sequence of each keypoint at different time steps is regarded as a temporal series input. The multi-head self-attention mechanism (MHSA) is used to model the trajectory of each joint in the temporal dimension. The motion change pattern and local temporal context dependency are calculated by using the query Q, key K, and value V vectors. The output of the spatial attention branch is the spatial attention. The output of the temporal attention branch is temporal attention. Characterized by element-wise addition and fusion 5. The 3D pose estimation method combining global modeling and local refinement according to claim 1, characterized in that, The specific implementation of the local refinement branch is as follows: The local refinement branch takes a two-dimensional pose sequence as input, and each frame contains the (x,y) coordinates of J key points. The whole is composed of a local structure-aware encoder and a multi-scale temporal modeling module connected in series. First, the local structure-aware encoder takes the 2D pose of each frame as input, which contains a set of sequentially arranged joint coordinates. Each joint coordinate is constructed into a unit keypoint patch and mapped to a feature representation through a keypoint local patch embedding layer: a learnable linear projection layer is used to map each 2D coordinate (x, y) into an embedding vector of dimension C. At the same time, spatial position information is introduced, and a learnable position vector is assigned to each joint through spatial position embedding. The embedding dimension is also C, and it is added element-wise to the feature representation obtained by keypoint local patch embedding to obtain the initial encoded features of each joint. The initial encoded features of the joints are then sequentially input into the Transformer Encoder, where a structure-aware local self-attention mechanism is introduced to obtain local features; Subsequently, this local feature is fed into a multi-scale temporal modeling module, which contains three parallel one-dimensional dilated convolution branches with sequentially increasing kernel size and dilation rate. The outputs of the three convolution branches are concatenated along the channel dimension to form a fused temporal feature, which is then input into a channel recalibration module for dynamic enhancement, generating channel attention weights s∈[0,1]. C The obtained channel attention weights are then multiplied channel by channel with the fusion features formed by splicing the three temporal branches, that is, the temporal features of each channel are explicitly weighted. The weighted result is then passed through a 1×1 convolutional layer to compress the number of channels and residually connected with the input local features to output a temporal feature representation with enhanced structure. Ultimately, this representation is mapped to a local 3D pose sequence through two fully connected regression heads.

6. The 3D pose estimation method combining global modeling and local refinement according to claim 5, characterized in that, The structure-aware local self-attention mechanism is specifically implemented as follows: the input two-dimensional pose is extracted by the CPN network, where the order of key points and their topology are clearly given; during the Transformer Encoder encoding process, a local perception range is defined for each key point according to the skeleton connection relationship, and attention interaction is only allowed with adjacent key points directly connected to the anatomical structure. The attention interaction specifically involves: in the self-attention computation, firstly, a fixed adjacency matrix A∈0,1 is defined based on the skeleton topology. J×J , where A ij =1 indicates that the i-th joint is connected to the j-th joint; subsequently, after calculating the similarity score between query Q and key K, only A... ij Positions with a value of 1 retain attention calculations, while other positions are set to negative infinity. Local features are obtained through attention interactions.

7. The 3D pose estimation method combining global modeling and local refinement according to claim 6, characterized in that, The channel recalibration module employs a compression and excitation mechanism. First, it performs spatial compression on each channel through a global average pooling operation to generate a channel descriptor z∈R. C Next, the channel descriptor is fed into a two-layer fully connected network: the first layer compresses the dimension from C to C / r, where r is the compression ratio, then restores it to C dimensions after ReLU activation, and uses the Sigmoid function to generate normalized channel attention weights s∈[0,1]. C .

8. The three-dimensional pose estimation method combining global modeling and local refinement according to any one of claims 1 to 7, characterized in that, The specific implementation process of step 4 is as follows: During the fusion phase, the global 3D pose sequence and the local 3D pose sequence output from the two branches are weighted and averaged element by element. Three loss terms are introduced for joint optimization: weighted average joint position error L W Time consistency loss L T and the average velocity error per joint L M Among them, L W A weighting mechanism is introduced based on traditional joint position errors to differentiate the importance of different joints; L T Used to constrain the magnitude of attitude changes between adjacent frames; L M By constraining the error between the predicted and actual posture at the velocity level, the ability to model motion rhythm and dynamic consistency is enhanced.

Images