A three-dimensional human pose estimation method based on token fusion U-shaped architecture

Abstract

This invention proposes a 3D human pose estimation method based on a token-fused U-shaped architecture. The method includes: extracting and mapping features from the original video frames to obtain a high-dimensional feature sequence; performing hierarchical feature extraction and temporal compression on the high-dimensional feature sequence using an encoder to obtain a compressed feature sequence; performing hierarchical feature reconstruction and temporal recovery on the compressed feature sequence using a decoder to obtain a feature sequence that restores full resolution; mapping the high-dimensional features of each joint in the restored full-resolution feature sequence to 3D spatial coordinates using a linear regression layer to generate a 3D human pose sequence; and using the 3D human pose sequence as the 3D human pose estimation result. This invention combines a token-fused temporal encoder and a token-recovery temporal decoder to compress temporal redundancy without causing temporal disorder, thereby reducing computational complexity while preserving long-distance dependencies.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation technology, and in particular to a three-dimensional human pose estimation method based on a token fusion U-shaped architecture. Background Technology

[0002] 3D human pose estimation (3D HPE) is a fundamental task in computer vision, aiming to predict human joints from monocular images or videos. It is widely used in action recognition, motion tracking, and virtual reality. Currently, the mainstream paradigm for video-based 3D HPE follows a "2D-to-3D lifting" workflow, which involves first detecting 2D keypoints in each frame and then mapping them to 3D space. In recent years, the Video Pose Transformer (VPT) has achieved state-of-the-art (SOTA) performance by effectively modeling long-range dependencies, demonstrating superiority over traditional graph convolutional networks (GCNs) in capturing temporal correlations across video frames.

[0003] However, the superior performance of VPT comes at the cost of significant computational overhead. To effectively handle occlusion and complex actions, these models typically require long input sequences (e.g., 81 to 351 frames), and the quadratic computational complexity of standard self-attention mechanisms severely degrades inference speed. This high computational cost hinders their deployment on resource-constrained edge devices. Therefore, achieving a significant reduction in complexity without sacrificing long-range modeling capabilities remains a key challenge.

[0004] While numerous efficiency optimization methods for visual Transformers have been proposed, their direct application to video pose estimation still faces significant limitations. Traditional token pruning discards tokens with low information content. Although effective for image classification, this approach is unsuitable for dense prediction tasks requiring frame-by-frame output (such as pose estimation). Token pausing, while preserving tokens, often prevents subsequent network layers from utilizing their contained information, leading to decreased accuracy. Existing token merging methods, while maintaining information integrity, frequently result in temporal disorder due to unstructured clustering and incur significant performance losses when recovering the full-frame output. Summary of the Invention

[0005] In view of the above situation, the main objective of this invention is to propose a three-dimensional human pose estimation method based on a token fusion U-shaped architecture to solve the above-mentioned technical problems.

[0006] This invention proposes a three-dimensional human pose estimation method based on a token fusion U-shaped architecture, the method comprising the following steps: Step 1: Extract and map the features of the original video frames to obtain a high-dimensional feature sequence; Step 2: Perform encoder hierarchical feature extraction and temporal compression on the high-dimensional feature sequence to obtain a compressed feature sequence; Step 3: Perform decoder hierarchical feature reconstruction and temporal restoration on the compressed feature sequence to obtain the feature sequence with full resolution restored. Step 4: Through a linear regression layer, the high-dimensional features of each joint in the feature sequence that recovers full resolution are mapped to three-dimensional spatial coordinates to generate a three-dimensional human pose sequence; the three-dimensional human pose sequence is used as the three-dimensional human pose estimation result.

[0007] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention combines a Token Merging Temporal Encoder and a Token Recovery Temporal Decoder to compress temporal redundancy without causing temporal disorder, thereby reducing computational complexity while preserving long-distance dependencies.

[0008] 2. This invention integrates the local topology awareness of GCN with the global modeling of the attention mechanism. By introducing a feature modulation module, it solves the problem of semantic misalignment and enhances the expressive power of features.

[0009] 3. Comprehensive experiments on the Human3.6M and MPI-INF-3DHP benchmark datasets demonstrate that U-VPT (the present invention) significantly reduces computational costs and accelerates inference speed while maintaining state-of-the-art (SOTA) accuracy. Attached Figure Description

[0010] Figure 1 This is a flowchart of the three-dimensional human pose estimation method based on a token fusion U-shaped architecture proposed in this invention; Figure 2 This is a diagram illustrating the overall framework of the three-dimensional human pose estimation method based on a token fusion U-shaped architecture proposed in this invention. Figure 3 This is a schematic diagram of the dual-stream spatial modulation module of the three-dimensional human pose estimation method based on the token fusion U-shaped architecture proposed in this invention; Figure 4 This is a schematic diagram of topologically aware spatial attention in the 3D human pose estimation method based on a token fusion U-shaped architecture proposed in this invention. Figure 5 This is a schematic diagram of the dual semantic alignment module of the three-dimensional human pose estimation method based on the token fusion U-shaped architecture proposed in this invention; Figure 6 This is a schematic diagram illustrating node saliency mining and channel dependency modeling in the 3D human pose estimation method based on a token fusion U-shaped architecture proposed in this invention. Figure 7 This invention relates to a token merging temporal encoder for a three-dimensional human pose estimation method based on a token fusion U-shaped architecture. Figure 8 This invention provides a token recovery time-series decoder for a three-dimensional human pose estimation method based on a token fusion U-shaped architecture. Detailed Implementation

[0011] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0012] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0013] Please see Figure 1 This invention proposes a three-dimensional human pose estimation method based on a token fusion U-shaped architecture, which includes the following steps: Step 1: Extract and map the features of the original video frames to obtain a high-dimensional feature sequence; In step 1, the features of the original video frames are extracted and mapped to obtain a high-dimensional feature sequence. The specific steps are as follows: Two-dimensional human skeleton nodes are detected in each frame of the original video, and the two-dimensional coordinates of the human skeleton nodes are extracted to obtain the human skeleton node sequence. The sequence of human skeletal nodes is mapped to a high-dimensional feature space through a linear projection layer to obtain a high-dimensional feature sequence.

[0014] Specifically, please see Figure 2This invention employs a U-shaped hierarchical architecture to estimate 3D pose sequences from 2D input. The network first projects the input into a high-dimensional space and feeds it into an encoder consisting of three stacked stages. Each stage integrates a dual-stream spatial modulation module (DSMB) and a token-merging temporal encoder (TMTE) to extract multi-scale features while progressively compressing the temporal dimension. Subsequently, the features pass through a bottleneck layer into a symmetric decoder, which uses a token recovery temporal decoder (TRTD) and DSMB to reconstruct the full-resolution sequence. To preserve fine-grained details lost during downsampling, the network introduces skip connections to fuse features from the encoder and decoder, and then generates the final 3D pose output through a regression head.

[0015] Step 2: Perform encoder hierarchical feature extraction and temporal compression on the high-dimensional feature sequence to obtain a compressed feature sequence; In step 2, the high-dimensional feature sequence is subjected to encoder hierarchical feature extraction and temporal compression to obtain a compressed feature sequence. The specific steps are as follows: Step 201: Perform parallel global flow branching and local flow branching on the high-dimensional feature sequence in the spatial dimension to obtain global flow features and local flow features respectively; Global flow features and local flow features are input into two parallel and topologically consistent paths. The processing procedure for one of these paths is as follows: Step 202: Take one of the global flow features or the local flow features as the reference feature and the other as the target feature; after nonlinear mapping and normalization, multiply the reference feature element-wise to obtain the updated reference feature; the present invention first performs node saliency mining in the time dimension of the reference feature. This step aims to filter and highlight key nodes containing rich information in the reference flow (e.g., active moving limb extremities). Step 203: Compress the updated reference features to aggregate local channel information, and then project them back to the original dimension to obtain reference features with aggregated channel information; Based on the reference features with aggregated channel information, process them through an activation function to generate channel modulation weights; Multiply the channel modulation weights element-wise with the target features to level the semantic gap between the global stream and the local stream, so as to obtain the updated target features. Step 204: Divide the updated target feature sequence into non-overlapping local windows according to the time dimension. After the features of the same joint in the local window are spliced ​​in a channel-first manner, they are then processed by layer normalization and linear projection layer in sequence to complete the downsampling compression in the time dimension and obtain the merged features. Step 205: Input the merged features into the standard Transformer layer, and process them through a multi-head self-attention mechanism to obtain the encoder-side temporal attention output features; Step 206: Perform layer normalization and nonlinear transformation on the temporal attention output features of the encoder, and then obtain the output features of the token merging temporal encoder through residual connection. Step 207: Obtain the output feature sequence of the Token Merging Temporal Encoder through the output features of the Token Merging Temporal Encoder; use the output features of the Token Merging Temporal Encoder in the current stage as the high-dimensional feature sequence of the next stage, and repeat steps 201 to 206 to gradually compress the temporal resolution until the preset compression depth is reached to obtain the compressed feature sequence. The specific steps for global flow branch processing are as follows: By utilizing high-dimensional feature sequences, the query matrix, key matrix, and value matrix are obtained through calculation. Attention is calculated based on the topological distance matrix, query matrix, key matrix, and value matrix between joints to obtain the topology-aware spatial attention weights; the topology-aware spatial attention weights are multiplied by the value matrix to obtain the global flow features; The specific steps for handling local flow branches are as follows: By utilizing a learnable graph convolutional network and a predefined adjacency matrix of human skeleton connection relationships, the features of each joint in the high-dimensional feature sequence are aggregated to update the features of adjacent joints and obtain local flow features.

[0016] Specifically, in the process of performing nonlinear mapping and normalization on the reference features, and then multiplying them element-wise to obtain the updated reference features, the following relationship exists: ; in, Indicates the updated reference feature, Indicates a reference feature, This indicates element-wise multiplication. This indicates the processing of node saliency mapping functions; The process involves compressing the updated reference features to aggregate local channel information, then projecting them back to the original dimension to obtain reference features with aggregated channel information. Based on these aggregated reference features, an activation function is used to generate channel modulation weights. Finally, the channel modulation weights are multiplied element-wise with the target features to obtain the updated target features. The following relationship applies: ; in, This represents the updated target features. Indicate target features, This indicates the processing of the Sigmoid function. This indicates the processing of channel modeling functions; After stitching together the features of the same joint within a local window using a channel-first approach, and then processing them sequentially through layer normalization and linear projection layers to achieve downsampling compression in the time dimension and obtain the merged features, the following relationship exists: ; in, Indicates the characteristics after merging. Indicates linear layer processing. This indicates normalization. Indicates channel splicing. Indicates the local window number Frame characteristics Indicates the local window number Frame characteristics Indicates the size of a local window; When the merged features are input into a standard Transformer layer and processed by a multi-head self-attention mechanism to obtain the temporal attention output features at the encoder end, the following relationship exists: ; in, This represents the temporal attention output features at the encoder end. This indicates the processing of multi-head self-attention mechanisms; In the process of performing layer normalization and nonlinear transformation on the temporal attention output features of the encoder, and then obtaining the output features of the token-merging temporal encoder through residual connection, the following relationship exists: ; in, This represents the output characteristics of the token merging time-series encoder. This indicates feedforward network processing; Attention is calculated based on the topological distance matrix, query matrix, key matrix, and value matrix between joints to obtain the topology-aware spatial attention weights. The process of multiplying these topology-aware spatial attention weights by the value matrix to obtain the global flow features corresponds to the following relationship: ; in, This represents topologically aware spatial attention. Represents the query matrix. This represents the transpose of the key matrix. Represents a value matrix, This represents the dimension of the key vector. This indicates the positional deviation. This represents the topological distance matrix between joints calculated using the shortest path algorithm. This indicates that attention weights are dynamically adjusted based on skeletal topological distance.

[0017] It should be noted that in this step 204, the features of the same joint within the local window are spliced ​​using a channel-first approach (e.g., channel 1 of the knee joint in frame 1, channel 1 of the knee joint in frame 2, etc.) instead of a frame-first approach (channel 1 of the knee joint in frame 1, channel 2 of the knee joint in frame 1).

[0018] Specifically, please see Figure 3 In order to collaboratively capture long-range semantic dependencies and local topological features in the spatial dimension, this invention proposes a dual-stream spatial modulation module (DSMB). The dual-stream spatial modulation module adopts a parallel dual-stream architecture: the global stream uses Topology Aware Spatial Attention (TASA) to extract global context, while the local stream uses a Learnable Graph Convolutional Network (LCN) to aggregate neighborhood information; in order to solve the semantic misalignment problem between these heterogeneous features, this invention designs a Bidirectional Semantic Alignment Module (BSAM) to perform feature interaction. Please see Figure 4 While absolute position embeddings can effectively encode the semantics of specific joints, they cannot explicitly capture the structural constraints of the skeleton (e.g., limb connections); to address this issue, this invention introduces topology-aware relative position embeddings (TARE) in TASA. Please see Figure 5The bidirectional semantic alignment module of this invention aims to dynamically calibrate the target stream using information from the reference stream; in a specific implementation, the process includes two consecutive steps: node saliency mining and channel dependency modeling. Please see Figure 6 In Node Saliency Mining (NSM), this invention learns the spatial importance weight of each joint through nonlinear mapping and spatially reweights the reference features; this step aims to filter and highlight key nodes (e.g., active moving extremities) that contain rich information in the reference stream. In channel dependency modeling (CDM), this invention employs a channel interaction network with a bottleneck structure to capture the correlation between channels; CDM first compresses features into a low-dimensional space to aggregate global channel information, and then projects them back to the original dimension to generate modulation weights; Based on this mechanism, DSMB performs bidirectional interaction: it uses GCN features to gate TASA features (locally guided global refinement) and uses TASA features to enhance GCN features (globally guided local enhancement). Finally, these two calibrated feature streams are fused through weighted summation, achieving an organic unity between global semantics and local topology.

[0019] For further details, please refer to Figure 7 To alleviate the computational redundancy caused by long video sequences and to construct feature representations with multi-scale characteristics, this invention proposes the Token Merging Temporal Encoder (TMTE). The Token Merging Temporal Encoder adopts a "compress-then-model" strategy: it first achieves downsampling in the time dimension by aggregating information within the local neighborhood, and then performs efficient long-distance feature refinement in the compressed latent space using the Transformer structure. Specifically, given an input feature sequence, TMTE first performs a Temporal Patch Merging (TPM) operation. This invention divides the temporal dimension into non-overlapping local windows, where each window contains s adjacent frames. The features of corresponding joints in these adjacent frames are first concatenated in the channel dimension, and then mapped to the target dimension through Layer Normalization (LN) and a linear projection layer. This process not only introduces an inductive bias that focuses on the local temporal context, but also compresses the sequence length to 1 / s of the original size. Subsequently, in order to capture the global temporal dependencies within the compressed sequence, this invention feeds the merged features into a standard Transformer layer; TPM concatenates the same joint of adjacent s-frames within a window using a channel-first approach rather than a frame-first approach, allowing the model to better capture redundant features in adjacent s-frames. Thanks to the TPM mechanism, the computational complexity of Multi-Head Self-Attention (MHSA) is reduced from... Significantly reduced to The location of the TPM module in TMTE and its implementation strategy will affect model performance.

[0020] Step 3: Perform decoder hierarchical feature reconstruction and temporal restoration on the compressed feature sequence to obtain the feature sequence with full resolution restored. In step 3, the compressed feature sequence undergoes decoder hierarchical feature reconstruction and temporal restoration processing to obtain a feature sequence that recovers full resolution. The specific steps are as follows: Step 301: Input the compressed feature sequence into the standard Transformer layer in the time dimension, and refine the temporal context through a multi-head self-attention mechanism and a feedforward network to obtain refined features; Step 302: Perform layer normalization, linear projection layer processing, and rearrangement operation on the refined features in sequence to obtain the expanded features. Step 303: Obtain the expanded feature sequence through the expanded features; use the expanded feature sequence as a high-dimensional feature sequence and perform gated fusion with the updated target feature sequence of the corresponding stage (the feature sequence of the corresponding encoder stage) to obtain the fused feature sequence. Step 304: Repeat steps 201 to 203 using the fused feature sequence as a high-dimensional feature sequence to obtain a new resolution feature sequence. Step 305: Repeat steps 301 to 304 using the new resolution feature sequence as the compressed feature sequence until the original number of input frames is reached, in order to obtain the feature sequence that restores the full resolution.

[0021] Specifically, in the time dimension, the compressed feature sequence is input into a standard Transformer layer, and then refined through a multi-head self-attention mechanism and a feedforward network to obtain the refined features. The corresponding relationship is as follows: ; in, This represents the output features of the decoder's temporal attention. Represents a compressed feature sequence. Indicates refined features; In the process of sequentially performing layer normalization, linear projection layer processing, and rearrangement operations on the refined features to obtain the expanded features, the following relationship exists: ; in, Indicates the features after expansion; This indicates a rearrangement operation, changing the shape of the tensor from... Convert to This is to achieve temporal upsampling, ensuring that the model output has the same number of 3D pose sequences as the input video frames, while avoiding the information ambiguity that usually results from direct interpolation. This represents the dimension of the expanded feature. Indicates batch size, Represents the number of frames in the time dimension. Indicates the channel size.

[0022] It should be noted that you should refer to [link / reference]. Figure 8 To reconstruct the original full-resolution temporal sequence from the latent features compressed by the encoder and achieve dense frame-level pose prediction, this invention proposes the Token Recovery Temporal Decoder (TRTD). The Token Recovery Temporal Decoder is structurally symmetrical to the Token Merging Temporal Encoder. Its core mechanism includes: first, refining the features in the latent space with low computational cost, and then restoring the temporal dimension through Temporal Patch Expanding (TPE). Specifically, given the output features of the token merging temporal encoder from the previous layer, TRTD employs a "refine-then-expand" strategy. First, this invention uses a standard Transformer layer to process compressed features, which is used to integrate contextual information within a compact temporal space; Subsequently, to restore the original temporal resolution, this invention performs a TPE operation as the inverse process of Temporal Block Merging (TPM). This operation first expands the feature channels to s times the original size through layer normalization and linear layers, and then "folds" the increased channel dimensions back to the temporal dimension through a Reshape operation. TPE transforms the feature representation of a joint in a single compressed time step into the corresponding feature sequence of that joint in s adjacent frames, thereby restoring fine-grained temporal motion details.

[0023] Step 4: Through a linear regression layer, the high-dimensional features of each joint in the feature sequence that recovers full resolution are mapped to three-dimensional spatial coordinates to generate a three-dimensional human pose sequence; the three-dimensional human pose sequence is used as the three-dimensional human pose estimation result.

[0024] Furthermore, the present invention has conducted several experiments to compare the performance of existing models with that of the present invention, as detailed below: Datasets and Evaluation Metrics: Human3.6M is the largest 3D HPE indoor dataset; it contains 3.6 million images and 11 professional actors, and uses Mean Per Joint Position Error (MPJPE) and Protodyakonov Alignment MPJPE (P-MPJPE) as evaluation metrics; MPI-INF-3DHP contains 1.3 million images from indoor and outdoor scenes; its test set contains three different scenes: a studio with a green screen, a studio without a green screen, and an outdoor scene, and uses MPJPE, Correct Keypoint Percentage (PCK), and Area Under the Curve (AUC) as evaluation metrics; Implementation details: Experiments were conducted on a single NVIDIA RTX 4080 SUPER GPU with horizontally flipped data augmentation. The model was trained for 150 epochs using the AdamW optimizer (batch size 4, weight decay 0.02). The initial learning rate was... And it is adjusted by the ReduceLROnPlateau scheduler (decay factor 0.8, patience 5).

[0025] This invention evaluates U-VPT against existing methods on the Human 3.6M dataset; Table 1 compares the performance of 15 action categories in terms of MPJPE, P-MPJPE, and computational cost; (Input...) In the case of frames, the method of the present invention achieves optimal results, with MPJPE of 37.7 mm and P-MPJPE of 31.7 mm; it is worth noting that even in At frame rate, U-VPT remains competitive (MPJPE at 40.5mm), outperforming most methods that rely on larger inputs; for example, it surpasses PoseformerV2 (…). (45.2mm) and MHFormer ( (43.0mm); In addition, the model of the present invention achieves the best balance between performance and efficiency; compared with TCPFormer, U-VPT provides equivalent performance while significantly reducing overhead: it requires fewer parameters (14.5M vs. 35.1M) and fewer multiply-accumulate operations per frame (MACs) (75M vs. 449M).

[0026] Table 1. Quantitative comparison on the Human 3.6M dataset

[0027] This invention evaluates the generalization ability of the model on the complex MPI-INF-3DHP dataset. Using ground-truth 2D poses as input and a sequence length of T=81, the method of this invention achieves the best performance at T=81 (see Table 2); its correct keypoint percentage (PCK) reaches 98.8%, its area under the curve (AUC) reaches 85.7%, and its MPJPE is 20.4 mm.

[0028] Table 2: Quantitative comparisons on the MPI-INF-3DHP dataset

[0029] To validate the TMTE design, ablation experiments were conducted on the downsampling strategy and the location of the TPM module (see Table 3). Regarding the downsampling strategy (a) in Table 3, the TPM mechanism of this invention (37.7 mm) outperforms the Fold operation (38.2 mm), one-dimensional convolutions (38.4 mm), and heuristic sampling methods (strided / uniform sampling); this confirms that aggregating local information through linear projection preserves motion semantics better than discarding frames or using local convolution kernels. Regarding the module location (b) in Table 3, placing the TPM before the Transformer block (“Pre-MHSA”) yields a significant performance advantage (37.7 mm) compared to Post-MHSA (38.5 mm) and Post-FFN (38.7 mm).

[0030] Table 3: Ablation Experiment Analysis of TPM Module

[0031] To verify the TRTD design, ablation experiments were conducted on the upsampling type and the location of the TPE module (see Table 4). Regarding the upsampling type (a in Table 4), the TPE of this invention (37.7 mm) outperformed the Unfold operation alone (38.2 mm), transposed convolution (38.9 mm), and the heuristic interpolation method (nearest / bilinear interpolation). Regarding the module location (b in Table 4), the "Post-FFN" configuration achieved the best results (37.7 mm), surpassing Pre-MHSA (38.3 mm) and Post-MHSA (38.2 mm).

[0032] Table 4: Ablation Experiment Analysis of TPE Module

[0033] To verify the effectiveness of the dual-stream spatial modulation module (DSMB), ablation experiments were conducted on its components, as shown in Table 5. First, the performance of using either the local stream (GCN, 39.7 mm) or the global stream (TASA, 39.5 mm) alone was inferior to naive dual-stream fusion (39.1 mm). Second, to address the alignment problem of heterogeneous features, BSAM was introduced. Integrating node saliency mining (NSM) or channel dependency modeling (CDM) alone reduced the error to 38.6 mm and 38.4 mm, respectively. Finally, the complete DSMB module achieved the best performance (37.7 mm).

[0034] Table 5: Ablation Experimental Analysis of Internal Components of DSMB

[0035] The five experiments above lead to the conclusion that the U-VPT architecture proposed in this invention achieves efficient 3D human pose estimation by utilizing temporal redundancy. To address the computational complexity issue, this invention designs a token merging and recovery module for compressing temporal sequences. Furthermore, a dual-stream spatial module implemented through feature modulation integrates local and global cues, thereby constructing a robust spatiotemporal feature representation. Experiments show that U-VPT achieves state-of-the-art (SOTA) performance while significantly reducing computational costs, providing new insights into lightweight video pose estimation.

[0036] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0037] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0038] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A three-dimensional human pose estimation method based on a token fusion U-shaped architecture, characterized in that, The method includes the following steps: Step 1: Extract and map the features of the original video frames to obtain a high-dimensional feature sequence; Step 2: Perform encoder hierarchical feature extraction and temporal compression on the high-dimensional feature sequence to obtain a compressed feature sequence. The specific steps are as follows: Step 201: Perform parallel global flow branching and local flow branching on the high-dimensional feature sequence in the spatial dimension to obtain global flow features and local flow features respectively; Global flow features and local flow features are input into two parallel and topologically consistent paths. The processing procedure for one of these paths is as follows: Step 202: Take one of the global flow features or the local flow features as the reference feature and the other as the target feature; perform nonlinear mapping and normalization on the reference feature and then multiply it element-wise with the target feature to obtain the updated reference feature; Step 203: Compress the updated reference features to aggregate local channel information, and then project them back to the original dimension to obtain reference features that aggregate channel information. Based on reference features that aggregate channel information, channel modulation weights are generated through activation function processing; the channel modulation weights are then multiplied element-wise with the target features to obtain the updated target features. Step 204: Divide the updated target feature sequence into non-overlapping local windows according to the time dimension. After the features of the same joint in the local window are spliced ​​in a channel-first manner, they are then processed by layer normalization and linear projection layer in sequence to complete the downsampling compression in the time dimension and obtain the merged features. Step 205: Input the merged features into the standard Transformer layer, and process them through a multi-head self-attention mechanism to obtain the encoder-side temporal attention output features; Step 206: Perform layer normalization and nonlinear transformation on the temporal attention output features of the encoder, and then obtain the output features of the token merging temporal encoder through residual connection. Step 207: Obtain the output feature sequence of the Token Merging Temporal Encoder by using the output features of the Token Merging Temporal Encoder; The output features of the token merging temporal encoder in the current stage are used as the high-dimensional feature sequence in the next stage. Steps 201 to 206 are repeated to gradually compress the temporal resolution until the preset compression depth is reached to obtain the compressed feature sequence. Step 3: Perform decoder hierarchical feature reconstruction and temporal restoration on the compressed feature sequence to obtain the feature sequence with full resolution restored. Step 4: Through a linear regression layer, the high-dimensional features of each joint in the feature sequence that recovers full resolution are mapped to three-dimensional spatial coordinates to generate a three-dimensional human pose sequence; the three-dimensional human pose sequence is used as the three-dimensional human pose estimation result.

2. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 1, characterized in that, In step 1, the features of the original video frames are extracted and mapped to obtain a high-dimensional feature sequence. The specific steps are as follows: Two-dimensional human skeleton nodes are detected in each frame of the original video, and the two-dimensional coordinates of the human skeleton nodes are extracted to obtain the human skeleton node sequence. The sequence of human skeletal nodes is mapped to a high-dimensional feature space through a linear projection layer to obtain a high-dimensional feature sequence.

3. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 1, characterized in that, In the process of applying nonlinear mapping and normalization to the reference features and then multiplying them element-wise to obtain the updated reference features, the following relationship exists: ； in, Indicates the updated reference feature, Indicates a reference feature, This indicates element-wise multiplication. This indicates the processing of node saliency mapping functions; The process involves compressing the updated reference features to aggregate local channel information, then projecting them back to the original dimension to obtain reference features with aggregated channel information. Based on these aggregated reference features, an activation function is used to generate channel modulation weights. Finally, the channel modulation weights are multiplied element-wise with the target features to obtain the updated target features. The following relationship applies: ； in, This represents the updated target features. Indicate target features, This indicates the processing of the Sigmoid function. This indicates the processing of channel modeling functions.

4. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 3, characterized in that, After stitching together the features of the same joint within a local window using a channel-first approach, and then processing them sequentially through layer normalization and linear projection layers to achieve downsampling compression in the time dimension and obtain the merged features, the following relationship exists: ； in, Indicates the characteristics after merging. Indicates linear layer processing. This indicates normalization. Indicates channel splicing. Indicates the local window number Frame characteristics Indicates the local window number Frame characteristics Indicates the size of a local window; When the merged features are input into a standard Transformer layer and processed by a multi-head self-attention mechanism to obtain the temporal attention output features at the encoder end, the following relationship exists: ； in, This represents the temporal attention output features at the encoder end. This indicates the processing of multi-head self-attention mechanisms; In the process of performing layer normalization and nonlinear transformation on the temporal attention output features of the encoder, and then obtaining the output features of the token-merging temporal encoder through residual connection, the following relationship exists: ； in, This represents the output characteristics of the token merging time-series encoder. This indicates feedforward network processing.

5. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 4, characterized in that, Parallel global flow branching and local flow branching are performed on the high-dimensional feature sequence in the spatial dimension to obtain global flow features and local flow features, respectively. The specific steps of global flow branching are as follows: Using high-dimensional feature sequences, the query matrix, key matrix, and value matrix are calculated. Attention is calculated based on the topological distance matrix, query matrix, key matrix, and value matrix between joints to obtain the topology-aware spatial attention weights; the topology-aware spatial attention weights are multiplied by the value matrix to obtain the global flow features; The specific steps for handling local flow branches are as follows: By utilizing a learnable graph convolutional network and a predefined adjacency matrix of human skeleton connection relationships, the features of each joint in the high-dimensional feature sequence are aggregated to update the features of adjacent joints and obtain local flow features.

6. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 5, characterized in that, Attention is calculated based on the topological distance matrix, query matrix, key matrix, and value matrix between joints to obtain the topology-aware spatial attention weights. The process of multiplying these topology-aware spatial attention weights by the value matrix to obtain the global flow features corresponds to the following relationship: ； in, This represents topologically aware spatial attention. Represents the query matrix. This represents the transpose of the key matrix. Represents a value matrix, This represents the dimension of the key vector. This indicates the positional deviation. This represents the topological distance matrix between joints. This indicates that attention weights are dynamically adjusted based on skeletal topological distance.

7. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 6, characterized in that, In step 3, the compressed feature sequence undergoes decoder hierarchical feature reconstruction and temporal restoration processing to obtain a feature sequence that restores full resolution. The specific steps are as follows: Step 301: Input the compressed feature sequence into the standard Transformer layer in the time dimension, and refine the temporal context through a multi-head self-attention mechanism and a feedforward network to obtain refined features; Step 302: Perform layer normalization, linear projection layer processing, and rearrangement operation on the refined features in sequence to obtain the expanded features. Step 303: Obtain the expanded feature sequence through the expanded features; use the expanded feature sequence as a high-dimensional feature sequence and perform gated fusion with the updated target feature sequence of the corresponding stage to obtain the fused feature sequence. Step 304: Repeat steps 201 to 203 using the fused feature sequence as a high-dimensional feature sequence to obtain a new resolution feature sequence. Step 305: Repeat steps 301 to 304 using the new resolution feature sequence as the compressed feature sequence until the original number of input frames is reached, in order to obtain the feature sequence that restores the full resolution.

8. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 7, characterized in that, In the time dimension, the compressed feature sequence is input into a standard Transformer layer, and then refined through a multi-head self-attention mechanism and a feedforward network to obtain the refined features. The corresponding relationship is as follows: ； in, This represents the output features of the decoder's temporal attention. Represents a compressed feature sequence. This indicates a refined feature.

9. The three-dimensional human pose estimation method based on a token fusion U-shaped architecture according to claim 8, characterized in that, In the process of sequentially performing layer normalization, linear projection layer processing, and rearrangement operations on the refined features to obtain the expanded features, the following relationship exists: ； in, Indicates the features after expansion. This indicates a rearrangement operation. This represents the dimension of the expanded feature. Indicates batch size, Represents the number of frames in the time dimension. Indicates the channel size.

Images