Three-dimensional human body posture estimation method based on double-branch space-time perception

Abstract

The invention discloses a three-dimensional human body posture estimation method based on double-branch space-time perception. The three-dimensional human body posture estimation method is specifically implemented according to the following steps: step 1, data preprocessing; 2, constructing a feature enhancement branch based on a space-time receiving weighted key value; step 3, constructing a feature enhancement branch based on global-local space-time diagram convolution; and 4, fusing the features to regress three-dimensional joint coordinates. According to the method, through collaborative design of the space-time receiving weighted key value branch and the global-local space-time diagram convolution branch, the balance between the calculation efficiency and the feature representation capability is realized.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and specifically relates to a three-dimensional human posture estimation method based on dual-branch spatiotemporal perception. Background Art

[0002] 3D human pose estimation is a key technology in computer vision and artificial intelligence. Its core goal is to predict the 3D spatial position of human joints from monocular images or videos, providing fundamental support for applications such as virtual reality, motion analysis, and human-computer interaction. Leveraging the feature extraction capabilities of deep neural networks, this method can infer high-precision 3D human pose information from complex scenes.

[0003] In recent years, Transformer-based models have significantly improved the accuracy of 3D human pose estimation by effectively capturing global spatiotemporal dependencies through self-attention mechanisms. However, self-attention requires computing similarity matrices across all frames, resulting in computational complexity that grows quadratically with sequence length, limiting its real-time application in long videos. Traditional methods, such as multi-layer graph convolutional networks (GCNs), can model the local topology of the human body but lack the ability to model global motion dynamics, making it difficult to handle complex pose variations.

[0004] Furthermore, in videos with rapid motion or occlusion, joint features are easily masked by background noise, further weakening the model's robustness. This poses significant challenges to modeling long sequences of spatiotemporal dependencies and accurately analyzing multi-scale joint relationships. Balancing computational efficiency with feature representation capabilities is crucial for improving 3D human pose estimation performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a three-dimensional human pose estimation method based on dual-branch spatiotemporal perception, which solves the problem of difficult balance between computational efficiency and feature representation capability in existing three-dimensional human pose estimation tasks.

[0006] The technical solution adopted by the present invention is a three-dimensional human posture estimation method based on dual-branch spatiotemporal perception, which is specifically implemented according to the following steps:

[0007] Step 1: Data preprocessing;

[0008] Step 2: Construct a feature enhancement branch based on spatiotemporal reception weighted key values;

[0009] Step 3: Construct a feature enhancement branch based on global-local spatiotemporal graph convolution;

[0010] Step 4: Fusion features regress 3D joint coordinates.

[0011] The technical solution of the present invention is also characterized in that:

[0012] Step 1 is implemented as follows:

[0013] Step 1.1: Collect 2D skeleton sequences from monocular videos or images, extract joint keypoint coordinates through a 2D pose detector, and construct a training set with 3D annotations.

[0014] Step 1.2: Perform random spatiotemporal enhancement on the input sequence to generate diverse training samples.

[0015] Step 1.3: Input the 2D joint coordinate sequence into the linear projection layer, map it to the high-dimensional feature space, and generate the initial feature tensor. Among them, B, T, J, d in are batch size, sequence length, number of joints, and number of channels, respectively, and R represents the real number domain;

[0016] Step 1.4, add learnable spatial position encoding By adding and fusing the spatiotemporal features element by element, the enhanced input features are obtained.

[0017] The random spatiotemporal enhancement operations in step 1.2 include inter-frame displacement, rotation perturbation, and temporal interpolation.

[0018] Step 2 is implemented as follows:

[0019] Step 2.1, x in First reshaped into Constructing a spatial bidirectional attention map Computed using a custom CUDA function, we perform linear complexity aggregation of global joint interactions, and the spatial bidirectional attention map is represented as:

[0020] wkv=RUN CUDA (B, J, C, γ s , δ s , K s , V s ) (1)

[0021] Among them, K s Indicates a key, V s Represents the value, γ s represents the normalized spatial attenuation along the joint dimension, δ s represents the initialization factor;

[0022] In step 2.2, the Sigmoid activation function is used to modulate the receiver of the spatial bidirectional attention map. The mathematical relationship is defined as follows:

[0023] O s =(σ(R s )⊙wkv)W o (2)

[0024] Among them, R s Represents the receiving vector, W o represents the linear projection matrix and ⊙ represents element-wise multiplication.

[0025] In step 2.3, the layers are then normalized and fed into a multilayer perceptron, followed by a residual connection. The mathematical relationship is defined as follows:

[0026] S R =MLP(LN(S M ))+O s +X in (3)

[0027] Step 2.4, construct a temporal bidirectional attention map The linear complexity of the global time interaction between consecutive steps is aggregated, and the mathematical relationship is defined as follows:

[0028] wkv=RUN CUDA (B, J, C, γ t , δ t , K t , V t ) (4)

[0029] Among them, K t Indicates a key, V t Represents the value, γ t represents the normalized spatial attenuation along the time dimension, δ t represents the initialization factor;

[0030] In step 2.5, the Sigmoid activation function is used to modulate the receiver of the temporal bidirectional attention map. The mathematical relationship is defined as follows:

[0031] O t =(σ(R t )⊙wkv)W o (5)

[0032] Among them, R t Represents the receiving vector, W o represents the linear projection matrix and ⊙ represents element-wise multiplication.

[0033] Step 2.6, then input to the normalization layer and multi-layer perceptron, and then perform residual connection. The mathematical relationship is defined as follows:

[0034] F R =MLP(LN(T R ))+O t +S R (6).

[0035] Step 3 is implemented as follows:

[0036] Step 3.1, build a global-local spatial graph convolution module;

[0037] Step 3.2, construct the global-local temporal graph convolution module.

[0038] Step 3.1 is as follows: reshape the features output from step 1.4 into Then it is fed into the global-local spatial graph convolution module to construct the skeletal adjacency matrix and the global learnable matrix to obtain the spatial structure matrix. The mathematical relationship is defined as follows:

[0039]

[0040] Among them, A J represents the bone adjacency matrix, represents the global learnable matrix, * represents the matrix transpose;

[0041]

[0042] Where E is the identity matrix, To normalize the spatial matrix, in order to retain the original feature information while introducing nonlinearity, the input Y in We add , and then apply the ReLU activation, which is defined as:

[0043] Y o =ReLU(Y in +BN(Y SG +H (0) )) (9)

[0044] Among them, BN(.) is the batch normalization operation, and then the residual features are combined with Y o Fusion, then through layer normalization and multi-layer perceptron, adding additional residual connections after these operations, the mathematical relationship is defined as follows:

[0045] Y sg =MLP(LN(Y o +x in ))+Y o +x in (10).

[0046] Step 3.2 is as follows: Transpose the output of step 3.1 and reshape it into Then it is fed into the global-local time graph convolution module to construct the time adjacency matrix and the time dynamic matrix to obtain the time matrix. The mathematical relationship is defined as follows:

[0047]

[0048] Among them, CT represents the temporal adjacency matrix, represents the time dynamic matrix;

[0049]

[0050] in To normalize the time matrix, in order to retain the original feature information while introducing nonlinearity, the input Y s We add , and then apply the ReLU activation, which is defined as:

[0051] Y p =ReLU(Y s +BN(Y TG +K (0) )) (13)

[0052] Then, the residual features are compared with Y o Fusion is then performed through layer normalization and multi-layer perceptrons. Additional residual connections are added after these operations. The mathematical relationship is defined as follows:

[0053] Y t =MLP(LN(Y p +Y sg ))+Y p +Y sg (14).

[0054] Step 4 is implemented as follows:

[0055] The output features of the feature enhancement branch of the spatiotemporal receiver weighted key and the feature enhancement branch of the global-local spatiotemporal graph convolution are dynamically fused through adaptive weights. The mathematical relationship is defined as follows:

[0056]

[0057] Among them, F i represents the feature embedding of the i-th layer, represents element-wise multiplication, and They correspond to the outputs of the spatiotemporal feature enhancement module and the spatiotemporal graph convolution module of the previous layer, and Represents the adaptive fusion weight.

[0058] The beneficial effects of the present invention are:

[0059] The present invention is based on a three-dimensional human posture estimation method with dual-branch spatiotemporal perception. Through the collaborative design of the spatiotemporal reception weighted key value branch and the global-local spatiotemporal graph convolution branch, a balance between computational efficiency and feature representation capabilities is achieved. The spatiotemporal reception weighted key value branch uses a bidirectional attention mechanism with linear complexity to replace the secondary calculation of the traditional Transformer, significantly reducing the processing overhead of long sequences; the global-local graph convolution branch fuses the skeletal adjacency matrix with the learnable dynamic matrix to simultaneously model the local topological structure and global motion pattern of the human body. The dual-branch features are dynamically fused through adaptive weights, which enhances the robustness to occlusion and rapid motion while retaining the spatiotemporal dependency of the joints. Ultimately, while ensuring real-time performance, the prediction accuracy of three-dimensional joint coordinates in complex scenes is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a network architecture diagram of the present invention;

[0061] Figure 2 This is the network architecture diagram of the global-local spatial graph convolution module in the present invention;

[0062] Figure 3 This is the network architecture diagram of the global-local time graph convolution module in the present invention;

[0063] Figure 4 This is the visual comparison result of the present invention's 3D human posture recognition on field videos;

[0064] Figure 5 This is a visual comparison of the present invention with other mainstream methods on the Human3.6M dataset. DETAILED DESCRIPTION

[0065] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Example 1

[0067] The present invention is based on a three-dimensional human posture estimation method based on dual-branch spatiotemporal perception, such as Figure 1 As shown, please follow the steps below:

[0068] Step 1: Data preprocessing;

[0069] Step 2: Construct a feature enhancement branch based on spatiotemporal reception weighted key values;

[0070] Step 3: Construct a feature enhancement branch based on global-local spatiotemporal graph convolution;

[0071] Step 4: Fusion features regress 3D joint coordinates.

[0072] Example 2

[0073] The 3D human pose estimation method based on dual-branch spatiotemporal perception, wherein step 1 is specifically implemented according to the following steps:

[0074] Step 1.1: Collect 2D skeleton sequences from monocular videos or images, extract joint keypoint coordinates through a 2D pose detector, and construct a training set with 3D annotations.

[0075] Step 1.2: Perform random spatiotemporal enhancement on the input sequence, including inter-frame displacement, rotation perturbation, and time series interpolation, to generate diverse training samples to improve model robustness.

[0076] Step 1.3: Input the 2D joint coordinate sequence into the linear projection layer, map it to the high-dimensional feature space, and generate the initial feature tensor. Among them, B, T, J, d in are batch size, sequence length, number of joints, and number of channels, respectively, and R represents the real number domain;

[0077] Step 1.4, add learnable spatial position encoding By adding and fusing the spatiotemporal features element by element, the enhanced input features are obtained.

[0078] Example 3

[0079] The 3D human pose estimation method based on dual-branch spatiotemporal perception, wherein step 2 is specifically implemented according to the following steps:

[0080] Step 2.1, x in First reshaped into Constructing a spatial bidirectional attention map Computed using a custom CUDA function, we perform linear complexity aggregation of global joint interactions, and the spatial bidirectional attention map is represented as:

[0081] wkv=RUN CUDA (B, J, C, γ s , δ s , K s , V s ) (1)

[0082] Among them, K s Indicates a key, V s Represents the value, γ s represents the normalized spatial attenuation along the joint dimension, δ s represents the initialization factor;

[0083] In step 2.2, the Sigmoid activation function is used to modulate the receiver of the spatial bidirectional attention map. The mathematical relationship is defined as follows:

[0084] Os =(σ(R s )⊙wkv)W o (2)

[0085] Among them, R s Represents the receiving vector, W o represents the linear projection matrix and ⊙ represents element-wise multiplication.

[0086] In step 2.3, the layers are then normalized and fed into a multilayer perceptron, followed by a residual connection. The mathematical relationship is defined as follows:

[0087] S R =MLP(LN(S M ))+O s +X in (3)

[0088] Step 2.4, construct a temporal bidirectional attention map The linear complexity of the global time interaction between consecutive steps is aggregated, and the mathematical relationship is defined as follows:

[0089] wkv=RUN CUDA (B, J, C, γ t , δ t , K t , V t ) (4)

[0090] Among them, K t Indicates a key, V t Represents the value, γ t represents the normalized spatial attenuation along the time dimension, δ t represents the initialization factor;

[0091] In step 2.5, the Sigmoid activation function is used to modulate the receiver of the temporal bidirectional attention map. The mathematical relationship is defined as follows:

[0092] O t =(σ(R t )⊙wkv)W o (5)

[0093] Among them, R t Represents the receiving vector, W o represents the linear projection matrix and ⊙ represents element-wise multiplication.

[0094] Step 2.6, then input to the normalization layer and multi-layer perceptron, and then perform residual connection. The mathematical relationship is defined as follows:

[0095] F R =MLP(LN(T R ))+Ot +S R (6).

[0096] Example 4

[0097] The 3D human pose estimation method based on dual-branch spatiotemporal perception, wherein step 3 is specifically implemented according to the following steps:

[0098] Step 3.1, build a global-local spatial graph convolution module, such as Figure 2 As shown;

[0099] Step 3.1 is as follows: reshape the features output from step 1.4 into The tongue is fed into the global-local spatial graph convolution module to construct the skeletal adjacency matrix and the global learnable matrix to obtain the spatial structure matrix. The mathematical relationship is defined as follows:

[0100]

[0101] Among them, A J represents the bone adjacency matrix, represents the global learnable matrix, * represents the matrix transpose;

[0102]

[0103] Where E is the identity matrix, To normalize the spatial matrix, in order to retain the original feature information while introducing nonlinearity, the input Y in We add , and then apply the ReLU activation, which is defined as:

[0104] Y o =ReLU(Y in +BN(Y SG +H (0) )) (9)

[0105] Among them, BN(.) is the batch normalization operation, and then the residual features are combined with Y o Fusion, then through layer normalization and multi-layer perceptron, adding additional residual connections after these operations, the mathematical relationship is defined as follows:

[0106] Y sg =MLP(LN(Y o +x in ))+Y o +x in (10).

[0107] Step 3.2, construct the global-local time graph convolution module, such as Figure 3 As shown;

[0108] Step 3.2 is as follows: Transpose the output of step 3.1 and reshape it into Then it is fed into the global-local time graph convolution module to construct the time adjacency matrix and the time dynamic matrix to obtain the time matrix. The mathematical relationship is defined as follows:

[0109]

[0110] Among them, C T represents the temporal adjacency matrix, represents the time dynamic matrix;

[0111]

[0112] in To normalize the time matrix, in order to retain the original feature information while introducing nonlinearity, the input Y s We add , and then apply the ReLU activation, which is defined as:

[0113] Y p =ReLU(Y s +BN(Y TG +K (0) )) (13)

[0114] Then, the residual features are compared with Y o Fusion is then performed through layer normalization and multi-layer perceptrons. Additional residual connections are added after these operations. The mathematical relationship is defined as follows:

[0115] Y t =MLP(LN(Y p +Y sg ))+Y p +Y sg (14).

[0116] Example 5

[0117] A 3D human pose estimation method based on dual-branch spatiotemporal perception. In step 4, inspired by MotionBERT, this architecture dynamically fuses the output features of the feature enhancement branch of the spatiotemporal reception weighted key and the feature enhancement branch of the global-local spatiotemporal graph convolution through adaptive weights. The mathematical relationship is defined as follows:

[0118]

[0119] Among them, F i represents the feature embedding of the i-th layer, represents element-wise multiplication, and They correspond to the outputs of the spatiotemporal feature enhancement module and the spatiotemporal graph convolution module of the previous layer, and Represents the adaptive fusion weight.

[0120] Example 6

[0121] The network model DuaISTA trained by the method proposed in this invention achieved high MPJPE and P-MPJPE values ​​on the Human3.6M dataset using the detected 2D results as input. Figure 4 Visual comparison results of 3D human pose recognition on wild videos. Figure 5 Table 1 shows the visual comparison of the present invention with other mainstream methods on the Human3.6M dataset.

[0122] Table 1. Comparison of the results of the method of the present invention and the mainstream method on Human3.6M

[0123]

[0124] T: number of input frames. P1: MPJPE error (mm). P2: P-MPJPE error (mm). P1*: P1 error of 2D ground truth. The best and second best metrics are bold and underlined, respectively.

[0125] This method, based on dual-branch spatiotemporal perception, achieves a balance between computational efficiency and feature representation capabilities through the collaborative design of a spatiotemporal reception weighted key-value branch and a global-local spatiotemporal graph convolution branch. The dual-branch features are dynamically fused with adaptive weights, preserving the spatiotemporal dependencies of joints while enhancing robustness to occlusion and rapid motion. Ultimately, the accuracy of 3D joint coordinate prediction in complex scenarios is significantly improved while ensuring real-time performance.

Claims

1. A three-dimensional human pose estimation method based on dual-branch spatiotemporal perception, characterized by: Please follow the steps below to implement: Step 1: Data preprocessing; Step 2: Construct a feature enhancement branch based on spatiotemporal reception weighted key values; Step 3: Construct a feature enhancement branch based on global-local spatiotemporal graph convolution; Step 4: Fusion features regress 3D joint coordinates.

2. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: Step 1.1: Collect 2D skeleton sequences from monocular videos or images, extract joint keypoint coordinates through a 2D pose detector, and construct a training set with 3D annotations. Step 1.2: Perform random spatiotemporal enhancement on the input sequence to generate diverse training samples. Step 1.3: Input the 2D joint coordinate sequence into the linear projection layer, map it to the high-dimensional feature space, and generate the initial feature tensor. Among them, B, T, J, d in are batch size, sequence length, number of joints, and number of channels, respectively, and R represents the real number domain; Step 1.4, add learnable spatial position encoding By adding and fusing the spatiotemporal features element by element, the enhanced input features are obtained.

3. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 2, characterized in that: The random spatiotemporal enhancement operation in step 1.2 includes inter-frame displacement, rotation perturbation and temporal interpolation.

4. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 1, characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1, x in First reshaped into Constructing a spatial bidirectional attention map Computed using a custom CUDA function, we perform linear complexity aggregation of global joint interactions, and the spatial bidirectional attention map is represented as: wkv=RUN CUDA (B,J,C,γ s ,δ s ,K s ,V s ) (1) Among them, K s Indicates a key, V s Represents the value, γ s represents the normalized spatial attenuation along the joint dimension, δ s represents the initialization factor; In step 2.2, the Sigmoid activation function is used to modulate the receiver of the spatial bidirectional attention map. The mathematical relationship is defined as follows: The s =(σ(R s )⊙wkv)W o (2) Among them, R s Represents the receiving vector, W o represents the linear projection matrix, ⊙ represents element-by-element multiplication; In step 2.3, the layers are then normalized and fed into a multilayer perceptron, followed by a residual connection. The mathematical relationship is defined as follows: S R =MLP(LN(S M ))+O s +X in (3) Step 2.4, construct a temporal bidirectional attention map The linear complexity of global time interactions between consecutive steps is aggregated, and the mathematical relationship is defined as follows: wkv=RUN CUDA (B,J,C,γ t ,δ t ,K t ,V t ) (4) Among them, K t Indicates a key, V t Represents the value, γ t represents the normalized spatial attenuation along the time dimension, δ t represents the initialization factor; In step 2.5, the Sigmoid activation function is used to modulate the receiver of the temporal bidirectional attention map. The mathematical relationship is defined as follows: The t =(σ(R t )⊙wkv)W o (5) Among them, R t Represents the receiving vector, W o represents the linear projection matrix, ⊙ represents element-by-element multiplication; Step 2.6, then input to the normalization layer and multi-layer perceptron, and then perform residual connection. The mathematical relationship is defined as follows: F R =MLP(LN(T R ))+O t +S R (6)。 5. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 1, characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1, build a global-local spatial graph convolution module; Step 3.2, construct the global-local temporal graph convolution module.

6. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 5, characterized in that: The step 3.1 is specifically as follows: reshape the features output from step 1.4 into Then it is fed into the global-local spatial graph convolution module to construct the skeletal adjacency matrix and the global learnable matrix to obtain the spatial structure matrix. The mathematical relationship is defined as follows: Among them, A J represents the bone adjacency matrix, represents the global learnable matrix, * represents the matrix transpose; Where E is the identity matrix, To normalize the spatial matrix, in order to retain the original feature information while introducing nonlinearity, the input Y in We add , and then apply the ReLU activation, which is defined as: Y o =ReLU(Y in +BN(Y SG +H (0) )) (9) Where BN(·) is the batch normalization operation, then the residual features are combined with Y o Fusion, then through layer normalization and multi-layer perceptron, adding additional residual connections after these operations, the mathematical relationship is defined as follows: AND sg =MLP(LN(Y o +x in ))+Y o +x in (10)。 7. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 5, characterized in that: The step 3.2 is specifically as follows: transpose the output of step 3.1 and reshape it into Then it is fed into the global-local time graph convolution module to construct the time adjacency matrix and the time dynamic matrix to obtain the time matrix. The mathematical relationship is defined as follows: Among them, C T represents the temporal adjacency matrix, represents the time dynamic matrix; in To normalize the time matrix, in order to retain the original feature information while introducing nonlinearity, the input Y s We add , and then apply the ReLU activation, which is defined as: AND p =ReLU(Y s +BN(Y TG +K (0) )) (13) Then, the residual features are compared with Y o Fusion is then performed through layer normalization and multi-layer perceptrons. Additional residual connections are added after these operations. The mathematical relationship is defined as follows: AND t =MLP(LN(Y p +Y sg ))+Y p +Y sg (14)。 8. The method for 3D human posture estimation based on dual-branch spatiotemporal perception according to claim 1, characterized in that: The step 4 is specifically implemented according to the following steps: The output features of the feature enhancement branch of the spatiotemporal receiver weighted key and the feature enhancement branch of the global-local spatiotemporal graph convolution are dynamically fused through adaptive weights. The mathematical relationship is defined as follows: Among them, F i represents the feature embedding of the i-th layer, represents element-wise multiplication, and They correspond to the outputs of the spatiotemporal feature enhancement module and the spatiotemporal graph convolution module of the previous layer, and Represents the adaptive fusion weight.

Images