A three-dimensional human pose estimation method, system and device based on space-time channel attention and multi-hypothesis fusion

Abstract

A three-dimensional human pose estimation method, system and device based on space-time channel attention and multi-hypothesis fusion, two-dimensional key points are extracted from input video frames and high-dimensional embedding is performed; a network model is constructed, and space, time and channel attention modeling is performed in parallel to extract joint space dependence, time sequence dynamics and channel correlation features; the three features are weighted and fused by learnable weights and output through residual connection; a cascaded fusion module extracts multi-scale hypothesis features, multi-head cross attention realizes multi-hypothesis information interaction, and regression outputs three-dimensional pose; parameters are set for training, and mean position error (MPJPE) is used for evaluation. The method can fully capture the spatial dependence and time sequence dynamics between joints, adaptively fuse multi-dimensional features, and effectively improve the accuracy and robustness of three-dimensional human pose estimation.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional human pose estimation technology, specifically to a three-dimensional human pose estimation method, system, and device based on spatiotemporal channel attention and multiple hypothesis fusion. Background Technology

[0002] 3D human pose estimation is an important technique in computer vision, applicable to motion analysis, human-computer interaction, virtual reality, and medical rehabilitation. The main objective of this research is to estimate the corresponding 3D human pose information from 2D human images, which places high demands on computational resources, computational complexity, and model generalization ability. Existing 3D human pose estimation methods can either directly estimate the 3D human pose from the image or first estimate the 2D human pose from the image and then estimate the 3D human pose from the 2D human pose. Direct estimation methods have high equipment requirements and the models are relatively complex. Methods that upscale from 2D pose to 3D pose can leverage existing mature 2D pose estimation algorithms, reducing the requirements on computing equipment and fully utilizing high-precision 2D detection results, thus significantly improving estimation accuracy.

[0003] Based on different modeling principles, 3D human pose estimation methods are mainly divided into geometric analysis methods and data-driven methods based on deep learning. Geometric analysis methods estimate the 3D coordinates of human pose using the principle of triangulation, but the multi-view acquisition method limits its effectiveness in handling complex scenes. The rapid development of deep learning technology has driven the transformation of 3D human pose estimation from geometric analysis methods to data-driven methods. The latter, through architectures such as convolutional neural networks or Transformers, enables the modeling of complex pose feature relationships from large-scale pose data. However, the two types of frameworks differ significantly in their feature modeling capabilities: convolutional neural networks, based on the feature extraction mechanism of local receptive fields, can only capture local correlations within a limited range. When dealing with long sequence tasks such as continuous sequence pose prediction, they struggle to effectively model global dependencies across frames, resulting in performance limitations. In contrast, Transformers, with their global self-attention mechanism, can directly model long-range dependencies of any sequence, exhibiting a significant advantage in capturing long sequence dependencies. Therefore, they have gradually become the mainstream technical approach for long sequence human pose estimation. However, the Transformer method, which currently has advantages in processing long sequence data, faces problems such as insufficient potential feature extraction, large number of parameters, and high computational complexity, which pose a great challenge to achieving better estimation results.

[0004] Patent application CN121281137A discloses a three-dimensional human pose estimation method, system, electronic device, and readable storage medium. The three-dimensional human pose estimation method includes the following steps: inputting a video frame sequence; obtaining the two-dimensional coordinates of human key points in each frame using a two-dimensional pose extractor to generate a two-dimensional pose sequence; performing linear embedding, spatial embedding, and temporal embedding on the extracted two-dimensional poses using an LST module; further capturing spatial dependencies and temporal dynamic changes in the two-dimensional pose data through a dual attention mechanism embedded in the DAS-TF module to obtain an enhanced spatiotemporal feature representation and outputting the three-dimensional coordinates of the human key points; after completing the first round of parameter updates, performing a second round of parameter optimization using a BV-Loss module to output the optimized three-dimensional human pose estimation result. This patent application utilizes a spatiotemporal attention mechanism to model spatial dependencies and temporal dynamic relationships, and uses the BV-Loss module to optimize training parameters; however, the model has a large number of parameters and a high computational complexity. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to provide a 3D human pose estimation method, system, and device based on spatiotemporal channel attention and multi-hypothesis fusion. In the spatiotemporal channel module with a self-attention mechanism, input features are divided into spatial, temporal, and channel components, respectively modeling inter-joint, inter-frame, and inter-feature channel dependencies, and adaptively fused based on learnable weights. Multiple efficient hypothesis solutions are generated through multi-layer coding and multi-scale hypothesis extraction. Finally, the optimal estimation result is output through mutual learning and fusion of multiple hypotheses, resulting in high estimation accuracy and low computational complexity.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion includes the following steps: Step 1: Preprocess the input continuous video frames of human pose to obtain high-dimensional embedding features; specifically, this includes: extracting a two-dimensional human keypoint sequence; performing linear projection on the keypoint coordinates; and performing channel dimension upscaling on the projected features to generate the high-dimensional embedding features. Step 2: Construct and run a 3D human pose estimation network model, outputting 3D human pose coordinates; the network model executes the following sequentially: Spatiotemporal channel feature modeling: Spatial attention modeling, temporal attention modeling and channel attention modeling are performed in parallel on the high-dimensional embedded features to extract intra-frame joint spatial dependency features, cross-frame temporal dynamic features and inter-channel correlation features, respectively. Learnable feature fusion: The three-way features are weighted and fused using three sets of learnable weight coefficients, and combined with residual connections to output encoded features; Multi-layer joint coding and multi-scale feature extraction: Multiple fusion modules are cascaded and stacked to form a deep encoder, and features are extracted from different depths as multi-scale pose hypothesis features; Multi-hypothesis inference: After each hypothesis feature is independently embedded and encoded, information interaction and fusion between different hypotheses are realized through a multi-head cross-attention mechanism. The fused features are sent to the regression head and output three-dimensional human pose coordinates. The regression head is a fully connected layer that maps the fused features to the three-dimensional joint coordinate space. Step 3: Train the 3D human pose estimation network model: Using the preprocessed high-dimensional embedding features as input, set the network operating parameters, train the network model on the experimental platform, evaluate it using the average position error index, and output the final 3D human pose estimation result.

[0007] The specific method for step one is as follows: 1) Extraction of two-dimensional human body key point sequences The system acquires continuous human motion video frame images, extracts the two-dimensional planar coordinates of key human joints in a single frame using a target key point detection algorithm, and splices the continuous temporal frame data to form a two-dimensional key point original sequence with dimensions of frame number × joint number × channel number. 2) Coordinate linear projection A linear spatial transformation is performed on the original two-dimensional keypoint coordinate sequence to stitch and flatten the two-dimensional coordinates of all human joints in each frame. The coordinate values ​​are mapped and reorganized through the linear transformation matrix, and the joint coordinates are converted from the original space into a feature space that the model can process. 3) High-dimensional feature upsizing embedding A 1×1 convolution kernel is used to perform channel dimension upscaling on the linearly projected two-dimensional coordinate features. The original two-dimensional coordinate low-dimensional channel features are mapped to a preset high-dimensional channel space by point-by-point convolution. Without changing the joint structure and temporal length, the number of feature channels is increased to enhance the feature expression capability and generate high-dimensional embedded features.

[0008] The specific method for modeling the spatiotemporal channel features in step two is as follows: Spatiotemporal channel feature modeling includes three parallel branches: spatial attention modeling, temporal attention modeling, and channel attention modeling. All of them use the Transformer structure for feature extraction. The core part of the Transformer is the multi-head self-attention mechanism. In multi-head attention mechanisms, input First, a query matrix is ​​generated using a linear mapping. Key matrix Sum matrix ,in Indicates the sequence length. Indicate the dimension; subsequently, , V is divided into multiple subspaces, and scaling dot product attention operations are performed in each subspace. The final output result is obtained by concatenation. This process is shown in formula (1):

[0009] in, Indicates the number of heads of attention. This represents the linear projection weights in the multi-head self-attention mechanism.

[0010] The specific method for learnable feature fusion in step two is as follows: A learnable feature fusion mechanism is introduced to adaptively weight and combine the outputs of the three branches: Spatial branching, temporal branching, and channel branching employ parallel modeling, with all three branches using the same input features. By modeling features of different dimensions and mapping them back to the original dimensions, and then using learnable weights for multidimensional adaptive feature fusion, multidimensional attention information can be collaboratively expressed. In addition, to promote gradient propagation, a jump residual connection is introduced between the encoder input and the fusion output, the mathematical form of which is shown in formula (3). Finally, a complete spatiotemporal channel fusion module is constructed. (2) in, , , These represent the learnable fusion parameters of spatial branch, temporal branch, and channel branch features, respectively.

[0011] In the spatiotemporal channel feature modeling, the input features are first embedded into a high-dimensional space through linear projection, and then combined with layer normalization and multi-head self-attention mechanism for feature modeling. The process is as follows:

[0012] in, To embed the projection matrix, For embedded dimensions, For normalization layer, This is a multi-head self-attention mechanism; Spatial attention modeling: for those containing The two-dimensional pose of the first joint, the second Frame input is ,in The original coordinate dimension is used; firstly, the input features are mapped to a high-dimensional embedding space through linear mapping projection, resulting in... Subsequently, the high-dimensional embedded feature input spatial encoder establishes global dependencies among all joints through a multi-head self-attention mechanism, outputting spatial features as follows: ; Temporal attention modeling: for continuous Frame pose sequence , fixed number Each joint, its time series A linear mapping to a high-dimensional space yields... The resulting feature sequence The data is input into a time encoder to obtain a time feature representation. ; Channel attention modeling: for a given The frame pose feature channels are modeled using a multi-head self-attention mechanism to model the relationships between all feature channels. First, the channels are... Input Mapped to a higher dimension, where, Representing all joint coordinates, we obtain The resulting feature sequence is then input into the channel encoder, where a multi-head self-attention mechanism establishes dependencies between channels, outputting channel features. .

[0013] The specific method for learnable feature fusion in step two is as follows: Let the spatial branch output characteristic be Y. s The time branch output feature is Y t The channel branch output characteristic is Y. c ; through learnable weights , , The three features are weighted and added to the encoder input x to obtain the spatiotemporal channel fusion module: .

[0014] The specific methods for step two, multi-layer joint coding and multi-scale feature extraction, are as follows: Multiple spatiotemporal channel fusion modules are arranged in a serial cascaded manner to form a deep feature encoder. The first spatiotemporal channel fusion module receives the original input features of the network, and the input of each subsequent spatiotemporal channel fusion module is the spatiotemporal channel fusion feature output by the previous level spatiotemporal channel fusion module. Features are passed down level by level, and modeling is deepened layer by layer. Through this cascaded structure, the network can extract shallow, medium, and deep multi-scale features, and generate three sets of hypothetical solutions based on the multi-scale features. ,in N is the number of frames, J is the number of joints, and c is the number of channels; all hypothetical features are uniformly mapped to a fixed-dimensional feature space.

[0015] The specific method for the multiple hypothesis inference in step two is as follows: Multiple parallel computations are employed to achieve deep information interaction between different hypotheses; specifically, given multiple hypothetical features... For the first Hypothetical characteristics The multi-head cross-attention mechanism selects other hypotheses to provide supplementary information, thereby achieving information complementarity between features at different levels, as follows:

[0016] in, This indicates a multi-head cross-attention operation. The representation layer is normalized, and m1 and m2 are two hypothetical indices other than m; The three sets of hypothetical features are concatenated and merged into a single hypothesis through hierarchical normalization. This forms a unified temporal representation for 3D pose regression, as follows:

[0017] in, Indicates splicing, MLP ( ) represents a multilayer perceptron, and e is the embedding dimension.

[0018] In step three, the network model's operating parameters include the initial learning rate, the learning rate decay coefficient, and the Adam optimizer is used to update the network parameters.

[0019] In step three, the network model's operating parameters include the initial learning rate and the learning rate decay coefficient, and the Adam optimizer is used to update the network parameters; the evaluation metric is the mean position error (MPJPE), calculated using the following formula:

[0020] in, Indicates the number of key points. Indicates the predicted key point location. This indicates the actual location of the joint.

[0021] The detection system based on the above detection method includes: The data preprocessing module is used to obtain the sequence of two-dimensional human body key points and complete the standardization and structuring of the input data; The backbone network module, including a spatiotemporal channel feature modeling module and a learnable adaptive fusion module, is used to perform multi-dimensional feature modeling and adaptive weighted fusion of the input human pose temporal features. The multi-layer feature encoding module, which is composed of cascaded spatiotemporal channel feature modeling modules, is used to extract deep pose features layer by layer and output multi-scale pose hypothesis features at different network depths. The multi-hypothesis interaction module adopts a cross-attention structure for information interaction and deep fusion of multi-scale hypothesis features; The regression output module is used to perform 3D coordinate regression on the fused features and output the final 3D human pose result.

[0022] A 3D human pose estimation and detection device based on spatiotemporal channel attention and multi-hypothesis fusion, comprising: Image acquisition device, used to acquire images of human posture; A processor configured to perform the method of any one of claims 1 to 8 when executing a computer program; Memory for storing the computer program; A display used to show the results of three-dimensional human pose estimation.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. In view of the shortcomings of existing model feature extraction, this invention designs a spatiotemporal channel attention module. By constructing spatial branches, temporal branches and channel branches, it models the spatial dependency between joints within a frame, the dynamic changes of the same joint across frames and the correlation between embedded feature channels, thereby enhancing the ability to express pose features from a multidimensional perspective.

[0024] 2. This invention designs a learnable feature fusion mechanism, which uses a parallel structure to jointly model spatiotemporal channel features and achieves dynamic weighted fusion of multidimensional attention features through trainable weights, thereby improving representation capabilities while maintaining the compactness of the model structure.

[0025] 3. To address the inherent multiple-solution problem in 3D pose estimation, this invention designs a multi-hypothesis generation and inference mechanism. It extracts features at different depths as pose hypotheses through hierarchical joint encoding, and promotes information interaction between hypotheses by using a cross-attention mechanism. Finally, it fuses the hypotheses through splicing and layer normalization to obtain more accurate 3D pose estimation results.

[0026] In summary, this invention jointly models human pose across three dimensions—space, time, and feature channels—to more fully uncover the structural information and dynamic dependencies in human motion. In the proposed framework, the spatiotemporal attention module models the spatial relationships between joints within the same frame, the temporal variations of the same joint across different frames, and the potential correlations between feature channels through parallel branches, thereby obtaining a richer pose representation. Simultaneously, a learnable feature fusion mechanism is introduced to adaptively weight and combine features extracted from different branches to further enhance feature representation capabilities. Furthermore, multi-scale features are extracted through a multi-layer coding structure, and a cross-attention-based multi-hypothesis inference mechanism enables information interaction and complementarity between different hypotheses, thereby improving the overall performance of 3D pose estimation. The results demonstrate that the proposed model can achieve accurate estimation of 3D human pose while reducing computational complexity, providing support for 3D human pose estimation. Attached Figure Description

[0027] Figure 1 This invention relates to a 3D human pose estimation model based on spatiotemporal channel attention and multiple hypothesis fusion.

[0028] Figure 2 The spatiotemporal channel attention modeling dimension and encoder structure diagram of the present invention are shown.

[0029] Figure 3 This is a structural diagram of the multi-layer joint coding and multi-scale feature extraction of the present invention.

[0030] Figure 4 This is a structural diagram illustrating the difference between the cross-attention mechanism and the self-attention mechanism of the present invention.

[0031] Figure 5 This is a diagram of the multi-hypothesis cross-learning structure of the present invention. Detailed Implementation

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments: See Figure 1 A 3D human pose estimation method based on spatiotemporal channel attention and multi-hypothesis fusion specifically includes the following steps: Step 1: Preprocess the input continuous human pose video frames, including 2D human keypoint sequence extraction, coordinate linear projection, and high-dimensional feature upsizing and embedding. 1) Extraction of two-dimensional human body key point sequences Acquire continuous human motion video frame images, extract the two-dimensional planar coordinates of key human joints in a single frame using a target key point detection algorithm (CPN detector or dataset with built-in ground truth annotation), and stitch together continuous temporal frame data to form a two-dimensional key point original sequence with dimensions of frame number × joint number × channel number.

[0033] 2) Coordinate linear projection A linear spatial transformation is performed on the original two-dimensional keypoint coordinate sequence. The two-dimensional coordinates of all human joints in each frame are stitched together and flattened. The coordinate values ​​are mapped and reorganized through the linear transformation matrix, and the joint coordinates are transformed from the original space into a feature space that the model can process.

[0034] 3) High-dimensional feature upsizing embedding A 1×1 convolution kernel is used to perform channel dimension upscaling on the linearly projected two-dimensional coordinate features. The original two-dimensional coordinate low-dimensional channel features are mapped to a preset high-dimensional channel space by point-by-point convolution. Without changing the joint structure and temporal length, the number of feature channels is increased to enhance the feature expression capability and generate high-dimensional embedded features, which serve as input data for subsequent encoding modules.

[0035] Step 2: Design a 3D human pose estimation network model 1) Spatiotemporal channel feature modeling See Figure 2 The spatiotemporal channel feature modeling consists of parallel spatial, temporal, and channel branches. The spatial branch mainly models the spatial relationships between joints within the same frame, the temporal branch mainly models the changes of the same joint across different frames, and the channel branch mainly models the feature channel relationships along the coordinate dimension. All three branches employ a Transformer structure for feature extraction, with the core component of the Transformer being a multi-head self-attention mechanism.

[0036] In multi-head attention mechanisms, input First, a query matrix is ​​generated using a linear mapping. Key matrix Sum matrix ,in Indicates the sequence length. This indicates the dimension. Subsequently, , V is divided into multiple subspaces, and scaling dot product attention operations are performed in each subspace. Finally, the output result is obtained by concatenation. This process is shown in formula (1).

[0037] (2) in, Indicates the number of heads of attention. This represents the linear projection weights in the multi-head self-attention mechanism.

[0038] In the three branches, the input features are first embedded into a high-dimensional space through linear projection, and then modeled by combining layer normalization and multi-head self-attention mechanism, as shown in Equation (2): (3) in, To embed the projection matrix, For embedded dimensions, For normalization layer, This is a multi-head self-attention mechanism.

[0039] 1.1 Spatial Attention Modeling. Spatial branching aims to model the spatial dependencies between joints within a single frame. For joints containing... The two-dimensional pose of the first joint, the second Frame input is ,in This represents the original coordinate dimension. First, the input features are mapped to a high-dimensional embedding space using a linear mapping projection, resulting in... Subsequently, an embedded feature input spatial encoder is used to establish global dependencies among all joints through a multi-head self-attention mechanism, outputting spatial features as follows: This branch can capture the structural relationships between joints within a frame, thereby enhancing spatial modeling capabilities.

[0040] 1.2 Temporal Attention Modeling. Temporal branches are used to model dynamic dependencies across frames. For continuous... Frame pose sequence , fixed number Each joint, its time series A linear mapping to a higher-dimensional space yields... The resulting feature sequence The data is input into a time encoder to obtain a time feature representation. This branch can effectively depict the dynamic changes of the same joint between different time steps.

[0041] 1.3 Channel Attention Modeling. Most pose estimation methods focus primarily on the spatial and temporal dimensions, neglecting the latent features between feature channels in the coordinate dimension. Therefore, this invention proposes a channel attention mechanism based on temporal embedding to model the relationships between different feature channels. For a given... In this invention, for the feature channels of frame pose, each feature channel is modeled in relation to all feature channels using a multi-head self-attention mechanism. First, this invention models the channels... Input Mapped to a higher dimension, where Representing all joint coordinates, we obtain The resulting feature sequence is then input into the channel encoder, where a multi-head self-attention mechanism establishes dependencies between channels, outputting channel features. Channel branches can learn the correlations between different motion features, reweight key motion patterns, and supplement the information on potential motion patterns that are difficult to capture by spatial and temporal branches at the feature representation level.

[0042] 2) Learnable feature fusion Since the spatial, temporal, and channel branches focus on significantly different dependencies, their importance varies for different actions and time periods. Therefore, this paper introduces a learnable feature fusion mechanism to adaptively weight and combine the outputs of the three branches.

[0043] Spatial branching, temporal branching, and channel branching employ parallel modeling, with all three branches using the same input features. By modeling features of different dimensions and mapping them back to the original dimensions, and then using learnable weights for multidimensional adaptive feature fusion, multidimensional attention information can be collaboratively expressed. In addition, to promote gradient propagation, a skip residual connection is introduced between the encoder input and the fusion output, the mathematical form of which is shown in formula (3). Finally, a complete spatiotemporal channel fusion module is constructed.

[0044] (4) in, , , These represent the learnable fusion parameters of spatial branch, temporal branch, and channel branch features, respectively.

[0045] 3) Multi-layer joint coding and multi-scale feature extraction See Figure 3 Multiple spatiotemporal channel fusion modules are serially cascaded to form a deep feature encoder. The first spatiotemporal channel fusion module receives the original input features of the network, and the input of each subsequent spatiotemporal channel fusion module is the spatiotemporal channel fusion feature output from the previous level. Features are passed down level by level, and modeling is deepened layer by layer. Through this cascaded structure, the network can extract shallow, medium, and deep multi-scale features and generate three sets of hypothetical solutions (shallow, medium, and deep) based on these multi-scale features. ,in These hypotheses arise from modeling different features from shallow to deep, encompassing diverse modeling features, providing rich learning samples for subsequent hypothesis fusion, and finally uniformly mapping to a fixed-dimensional feature space.

[0046] 4) Multiple Hypothesis Inference See Figure 4To address the complementarity between multi-scale features, this invention primarily employs multiple multi-head cross-attention mechanisms for parallel computation to achieve deep information interaction between different hypotheses. Unlike multi-head self-attention, which uses the same feature sequence to construct queries, keys, and values, multi-head cross-attention introduces feature sequences from different branches as queries, keys, and values ​​respectively, thus explicitly modeling the dependencies between multiple features.

[0047] See Figure 5 Given multiple hypothetical features For the first Hypothetical characteristics The remaining hypotheses are selected to provide supplementary information in order to achieve information complementarity between features at different levels, as shown in formula (4).

[0048] (5) in, This represents a multi-head cross-attention operation. Ultimately, the three sets of hypothetical features are concatenated and merged into a single hypothesis through layer normalization. A unified temporal representation is formed for 3D pose regression, as shown in formula (5). Finally, the fused features are fed into the regression head to output the 3D human pose coordinates.

[0049] (6) in, Indicates splicing, Representation layer normalization.

[0050] Step 3: Train the 3D human pose estimation network model The data processed in step 3) of step 1 is used as the input data for the 3D human pose estimation network in step 2. The network model's operating parameters are set, and the network model is trained on the experimental platform. Then, the performance of the trained network model is evaluated using commonly used evaluation metrics in the field of 3D human pose estimation, and finally, the estimation results of the 3D human pose are output.

[0051] In step three, the network model's operating parameters include the initial learning rate, the learning rate decay coefficient, and the Adam optimizer is used to update the network parameters.

[0052] The operating platform for step three is Ubuntu 20.04 system, using PyTorch 2.1.0 deep learning framework, with an AMD EPYC 7542 32-Core Processor CPU, an NVIDIA GeForce RTX3090 graphics card (GPU) with 24GB of memory, and Python as the programming language.

[0053] In step three, the commonly used evaluation metric in the field of 3D human pose estimation is the Mean Per-Joint Position Error (MPJPE). This metric is used to measure the average Euclidean distance between the 3D joint positions predicted by the model and the true values, reflecting the accuracy of the model in absolute position estimation. The lower the value, the more accurate the absolute position prediction.

[0054] The formula (6) for calculating the average position error is: (6) in, Indicates the number of key points. Indicates the predicted key point location. This indicates the actual location of the joint.

[0055] A 3D human pose estimation and detection device based on spatiotemporal channel attention and multi-hypothesis fusion, comprising: The data preprocessing module is used to obtain the sequence of two-dimensional human body key points and complete the standardization and structuring of the input data; The backbone network module, including a spatiotemporal channel feature modeling module and a learnable adaptive fusion module, is used to perform multi-dimensional feature modeling and adaptive weighted fusion of the input human pose temporal features. The multi-layer feature encoding module, which is composed of cascaded spatiotemporal channel feature modeling modules, is used to extract deep pose features layer by layer and output multi-scale pose hypothesis features at different network depths. The multi-hypothesis interaction module adopts a cross-attention structure for information interaction and deep fusion of multi-scale hypothesis features; The regression output module is used to perform 3D coordinate regression on the fused features and output the final 3D human pose result.

[0056] A 3D human pose estimation and detection device based on spatiotemporal channel attention and multi-hypothesis fusion, comprising: Image acquisition device for capturing images of human posture; A processor is used to store computer programs and execute the computer programs to implement the three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion as described in steps one to three. A display used to show the results of three-dimensional human pose estimation.

[0057] This invention compares the proposed 3D human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion with a series of popular 3D human pose estimation methods. All methods use the Human3.6M dataset, and the input data is the 2D pose detected by the CPN network. The results show that the proposed method improves the average position error and significantly reduces the computational complexity compared with other methods, proving that the proposed model can effectively estimate 3D human pose.

[0058] The experimental results are shown in Table 1.

[0059] Table 1. Comparison of params, FLOPs, and MPJPE for various methods on the Human 3.6M dataset. Zheng et al. (ICCV'21) 9.69 4874 44.3 Li et al. (CVPR'22) 31.69 25424 44.6 Tang et al. (CVPR'23) 18.9 156 214 44.1 Xu et al. (JOECU'24) 5.62 352 47.5 Hong et al. (IET'25) 20.42 2534 45.1 Zhou et al. (ASC'25) 7.55 1861 44.9 Ours 7.56 1810 43.3

Claims

1. A three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion, characterized in that: Includes the following steps: Step 1: Preprocess the input continuous video frames of human pose to obtain high-dimensional embedding features; Step 2: Construct and run a 3D human pose estimation network model, outputting 3D human pose coordinates; the network model executes the following sequentially: Spatiotemporal channel feature modeling: Spatial attention modeling, temporal attention modeling and channel attention modeling are performed in parallel on the high-dimensional embedded features to extract intra-frame joint spatial dependency features, cross-frame temporal dynamic features and inter-channel correlation features, respectively. Learnable feature fusion: The three-way features are weighted and fused using three sets of learnable weight coefficients, and combined with residual connections to output encoded features; Multi-layer joint coding and multi-scale feature extraction: Multiple fusion modules are cascaded and stacked to form a deep encoder, and features are extracted from different depths as multi-scale pose hypothesis features; Multi-hypothesis inference: After each hypothesis feature is independently embedded and encoded, information interaction and fusion between different hypotheses are realized through a multi-head cross-attention mechanism. The fused features are sent to the regression head and output three-dimensional human pose coordinates. The regression head is a fully connected layer that maps the fused features to the three-dimensional joint coordinate space. Step 3: Train the 3D human pose estimation network model: Using the preprocessed high-dimensional embedding features as input, set the network operating parameters, train the network model on the experimental platform, evaluate it using the average position error index, and output the final 3D human pose estimation result.

2. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 1, characterized in that: Step one specifically includes: extracting a two-dimensional human key point sequence; performing linear projection on the key point coordinates; and performing channel dimension upscaling on the projected features to generate the high-dimensional embedded features. 1) Extraction of two-dimensional human body key point sequences The system acquires continuous human motion video frame images, extracts the two-dimensional planar coordinates of key human joints in a single frame using a target key point detection algorithm, and splices the continuous temporal frame data to form a two-dimensional key point original sequence with dimensions of frame number × joint number × channel number. 2) Coordinate linear projection A linear spatial transformation is performed on the original two-dimensional keypoint coordinate sequence to stitch and flatten the two-dimensional coordinates of all human joints in each frame. The coordinate values ​​are mapped and reorganized through the linear transformation matrix, and the joint coordinates are converted from the original space into a feature space that the model can process. 3) High-dimensional feature upsizing embedding A 1×1 convolution kernel is used to perform channel dimension upscaling on the linearly projected two-dimensional coordinate features. The original two-dimensional coordinate low-dimensional channel features are mapped to a preset high-dimensional channel space by point-by-point convolution. Without changing the joint structure and temporal length, the number of feature channels is increased to enhance the feature expression capability and generate high-dimensional embedded features.

3. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 1, characterized in that: The specific method for modeling the spatiotemporal channel features in step two is as follows: Spatiotemporal channel feature modeling includes three parallel branches: spatial attention modeling, temporal attention modeling, and channel attention modeling. All of them use the Transformer structure for feature extraction. The core part of the Transformer is the multi-head self-attention mechanism. In multi-head attention mechanisms, input First, a query matrix is ​​generated using a linear mapping. Key matrix Sum matrix ,in Indicates the sequence length. Indicates dimension; Subsequently, , V is divided into multiple subspaces, and scaling dot product attention operations are performed in each subspace. The final output result is obtained by concatenation. This process is shown in formula (1): in, Indicates the number of heads of attention. This represents the linear projection weights in the multi-head self-attention mechanism.

4. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 3, characterized in that: In the spatiotemporal channel feature modeling, the input features are first embedded into a high-dimensional space through linear projection, and then combined with layer normalization and multi-head self-attention mechanism for feature modeling. The process is as follows: in, To embed the projection matrix, For embedded dimensions, For normalization layer, This is a multi-head self-attention mechanism; Spatial attention modeling: for spatial attention models containing The two-dimensional pose of the first joint, the second Frame input is ,in The original coordinate dimension is used; firstly, the input features are mapped to a high-dimensional embedding space through linear mapping projection, resulting in... Subsequently, the high-dimensional embedded feature input spatial encoder establishes global dependencies among all joints through a multi-head self-attention mechanism, outputting spatial features as follows: ; Temporal attention modeling: for continuous Frame pose sequence , fixed number Each joint, its time series A linear mapping to a high-dimensional space yields... The resulting feature sequence The data is input into a time encoder to obtain a time feature representation. ; Channel attention modeling: for a given The frame pose feature channels are modeled using a multi-head self-attention mechanism to model the relationships between all feature channels. First, the channels are... Input Mapped to a higher dimension, where, Representing all joint coordinates, we obtain The resulting feature sequence is then input into the channel encoder, where a multi-head self-attention mechanism establishes dependencies between channels, outputting channel features. .

5. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 1, characterized in that: The specific method for learnable feature fusion in step two is as follows: Let the spatial branch output characteristic be Y. s The time branch output feature is Y t The channel branch output characteristic is Y. c ; through learnable weights , , The three features are weighted and added to the encoder input x to obtain the spatiotemporal channel fusion module: 。 6. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 1, characterized in that: The specific methods for step two, multi-layer joint coding and multi-scale feature extraction, are as follows: The spatiotemporal channel fusion module receives the original input features from the network. The input to each subsequent spatiotemporal channel fusion module is the spatiotemporal channel fusion feature output from the previous module, with features presented hierarchically. The transmission and modeling are deepened layer by layer; through this cascaded structure, the network can extract multi-scale features from shallow, medium, and deep layers, and generate three sets of hypothetical solutions based on these multi-scale features. ,in N is the number of frames, J is the number of joints, and c is the number of channels; all hypothetical features are uniformly mapped to a fixed-dimensional feature space.

7. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 1, characterized in that: The specific method for the multiple hypothesis inference in step two is as follows: Multiple parallel computations are employed to achieve deep information interaction between different hypotheses; specifically, given multiple hypothetical features... For the first Hypothetical characteristics The multi-head cross-attention mechanism selects other hypotheses to provide supplementary information, thereby achieving information complementarity between features at different levels, as follows: in, This indicates a multi-headed cross-attention operation. The representation layer is normalized, and m1 and m2 are two hypothetical indices other than m; The three sets of hypothetical features are concatenated and merged into a single hypothesis through hierarchical normalization. This forms a unified temporal representation for 3D pose regression, as follows: in, Indicates splicing, MLP ( ) represents a multilayer perceptron, and e is the embedding dimension.

8. The three-dimensional human pose estimation method based on spatiotemporal channel attention and multiple hypothesis fusion according to claim 1, characterized in that: The network model's operating parameters in step three include the initial learning rate, the learning rate decay coefficient, and the Adam optimizer used to update the network parameters. In step three, the network model's operating parameters include the initial learning rate and the learning rate decay coefficient, and the Adam optimizer is used to update the network parameters; the evaluation metric is the mean position error (MPJPE), calculated using the following formula: in, Indicates the number of key points. Indicates the predicted key point location. This indicates the actual location of the joint.

9. A detection system based on the detection method according to any one of claims 1 to 8, characterized in that: include: The data preprocessing module is used to obtain the sequence of two-dimensional human body key points and complete the standardization and structuring of the input data; The backbone network module, including a spatiotemporal channel feature modeling module and a learnable adaptive fusion module, is used to perform multi-dimensional feature modeling and adaptive weighted fusion of the input human pose temporal features. The multi-layer feature encoding module, which is composed of cascaded spatiotemporal channel feature modeling modules, is used to extract deep pose features layer by layer and output multi-scale pose hypothesis features at different network depths. The multi-hypothesis interaction module adopts a cross-attention structure for information interaction and deep fusion of multi-scale hypothesis features; The regression output module is used to perform 3D coordinate regression on the fused features and output the final 3D human pose result.

10. A three-dimensional human pose estimation and detection device based on spatiotemporal channel attention and multi-hypothesis fusion, characterized in that: include: Image acquisition device, used to acquire images of human posture; A processor configured to perform the method of any one of claims 1 to 8 when executing a computer program; Memory for storing the computer program; A display used to show the results of three-dimensional human pose estimation.

Images