Human body posture estimation method based on conditional double-branch diffusion model

By adopting a conditional double-branch diffusion model in monocular three-dimensional human posture estimation, combining the GCN network and attention mechanism, joint-level semantic information is introduced, the problems of depth ambiguity and self-occlusion are solved, and the accuracy of three-dimensional human posture estimation is significantly improved. .

CN118968552APending Publication Date: 2024-11-15BEIJING UNIV OF TECH

Patent Information

Application Number
CN202411014083.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing monocular three-dimensional human posture estimation method has challenges in dealing with depth ambiguity and self-occlusion problems, especially the lack of global attention, which leads to insufficient accuracy of estimation of joint position in three-dimensional human body.

Method used

Using a conditional dual-branch diffusion model method, combined with GCN network and attention mechanism, the global and local correlation between joints is modeled through the dual-branch interactive module, and joint-level semantic information is introduced to alleviate depth ambiguity and self-occlusion. question.

Benefits of technology

The accuracy of three-dimensional human pose estimation in single-frame images is significantly improved, and the high-accurate 3D pose estimation is achieved through multiple iterations and feature fusion modules, overcoming the impact of depth blur and occlusion on performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968552A_ABST
    Figure CN118968552A_ABST
Patent Text Reader

Abstract

The invention discloses a three-dimensional human body posture estimation method based on a conditional double-branch diffusion model, and the method comprises the steps: firstly obtaining a heat map of a 2D posture from an existing 2D posture detector, predicting the depth distribution of joint points from a training set, and obtaining an initialized uncertain 3D posture; next, taking the uncertain 3D attitude as input of a double-branch diffusion model lifting network, taking joint-level semantic information as a condition, performing feature fusion through a feature fusion module, and performing local and global modeling on attitude joint features respectively through a GCN network of a double-branch network and an attention mechanism; and then interaction between global information and local information is carried out through a double-branch interaction module, and a high-accuracy 3D attitude is obtained through multiple iterations. According to the invention, the precision of three-dimensional human body posture estimation based on the single-frame image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and posture estimation, and is used in industries such as human body reconstruction and human-computer interaction. Specifically, the present invention is a three-dimensional human body posture estimation method based on a conditional double-branch diffusion model. Background Art

[0002] Monocular 3D human pose estimation aims to locate the 3D coordinates of human joints from a single image. It has a wide range of applications, such as human-computer interaction, action recognition, and human reconstruction. The current mainstream methods can be divided into two categories: 1) one-stage methods estimate the 3D human joint positions directly from single-view images; and 2) two-stage methods first estimate the positions of 2D keypoints from the image and then lift the estimated 2D pose to 3D joint coordinates. With the maturity of 2D keypoint detectors, more methods focus on lifting 2D poses to 3D poses; however, due to inherent depth ambiguity and potential self-occlusion problems, estimating 3D human joint positions from 2D poses remains challenging. The human skeleton is connected by joints, and the 2D skeleton is approximately a graph structure. Therefore, most existing methods use graph convolutional networks (GCNs) for pose estimation. GCN networks usually only focus on the local relationship between joints and adjacent joints, while there are also correlations between non-adjacent joints. Therefore, modeling the global correlation between joint points is equally important. The diffusion model is a probabilistic generative model that provides a gradual transition from one distribution to another, and can generate samples that match the deterministic data distribution from the initial uncertain noise distribution. Therefore, the present invention performs 3D human posture estimation based on the diffusion model, regards the 3D human posture estimation as a reverse diffusion process, and considers the text information of the added joints as a condition. At the same time, a posture and semantic feature fusion module is proposed to better fuse different information, thereby alleviating the depth ambiguity and self-occlusion problems. In view of the problem that the existing methods lack global attention, the present invention aims to study a dual-branch diffusion model, in which one branch uses the GCN network as the backbone to model the local correlation between adjacent joints; and the other branch uses the attention mechanism to model the global correlation between the entire joints. In order to better improve the performance of the two parallelized sub-networks, the present invention also introduces a dual-branch interaction method to more effectively capture the global and local correlations of body joints. Summary of the invention

[0003] The present invention proposes a three-dimensional human posture estimation method based on a conditional dual-branch diffusion model, which can effectively alleviate the inherent depth ambiguity and occlusion problems of posture estimation tasks. A dual-branch diffusion posture estimation module based on a combination of a GCN network and an attention mechanism, a diffusion model conditional generation module based on joint-level semantics, and a feature fusion module are proposed. The present invention first obtains a heat map of a 2D posture from an existing 2D posture detector, and predicts the depth distribution of joint points from a training set to obtain an initialized uncertain 3D posture. Next, the uncertain 3D posture is used as the input of our dual-branch diffusion model lifting network, and the joint-level semantic information is used as a condition to perform feature fusion through a feature fusion module. The GCN network and attention mechanism of the dual-branch network are used to model the posture joint features locally and globally, respectively, and then the dual-branch interaction module is used to interact between global and local information. Through multiple iterations, a high-accuracy 3D posture is obtained. This method improves the accuracy of three-dimensional human posture estimation based on a single-frame image.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] First, the existing 2D posture estimation network is used to obtain the heat map of the human joints, and the depth distribution is obtained from the training set. Then, the uncertain 3D posture is obtained from the 2D joint heat map and the depth distribution; the semantic information based on the human joint points is obtained through the diffusion model conditional generation module. The uncertain 3D posture features and joint semantic information are fused through the feature fusion module. The fused features are iterated multiple times through the GCN network and the attention dual-branch diffusion model to finally obtain a highly certain 3D posture. During this period, the dual-branch feature interaction module promotes the GCN network and the attention module to better model the global and local context between joints. The present invention uses a real annotated 3D posture constraint network to calculate the loss for training.

[0006] The uncertain 3D pose comprises the following steps:

[0007] Given an input RGB image, the existing 2D pose extraction network is used to obtain a heat map of human joints. Then the training set is used to predict the depth distribution of human joints. The initial uncertain 3D pose is obtained based on the 2D heat map and depth sampling obtained by the 2D pose extraction network.

[0008] The diffusion model condition generation module is used to obtain semantic information based on joint text, and the joint text is used as the condition of the diffusion model.

[0009] The initial uncertain 3D posture is transformed through a linear layer to obtain the initialized joint features; the joint semantic information obtained by the diffusion condition generation module is used as a condition, and the joint features are fused with the joint semantic information through the feature fusion module to obtain the fused features as the input of the GCN network and the attention dual-branch interactive network; the fused features are processed by the dual-branch interactive network module and then passed through a regression layer to obtain the 3D posture of the human body. The obtained 3D posture is used as the input of the dual-branch diffusion network again, and after K iterations of denoising, the final more accurate 3D posture is obtained.

[0010] The overall network framework diagram of the present invention is as follows: Figure 1 As shown. The present invention constructs a new human posture estimation network based on a conditional dual-branch diffusion model. For single-frame posture estimation, most of the current methods use a GCN network, but this lacks the global relationship between joints and is easily affected by depth blur and occlusion. Moreover, these methods only use the posture information of the human body without introducing semantic information for guidance. The present invention introduces the diffusion model into the posture estimation task, which can well alleviate the impact of depth blur and occlusion on performance. In view of the problem that the GCN network lacks global information, the present invention adopts a GCN network and an attention parallel network. The GCN network focuses on the local information between the human joints, and the attention block can better model the global information between the joints. The parallel GCN network and the attention module, and finally the designed dual-branch interaction module, can well process the global and local information between the joints. Common human posture estimation methods rely entirely on image information or posture information, lack of exploration of semantic information, and the present invention innovatively adds the semantic information of human joints as a condition on the basis of the diffusion model, introduces semantic information into the posture estimation task, and uses it as a semantic prompt to help achieve more accurate posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is the overall network framework of the present invention.

[0012] Figure 2 Schematic diagram of the structure of the posture and semantic feature fusion module.

[0013] Figure 3 Schematic diagram of the GCN network and attention block structure.

[0014] Figure 4 It is a schematic diagram of the qualitative posture estimation result of the present invention. DETAILED DESCRIPTION

[0015] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0016] The basic model for implementing this method includes an initial posture extraction module, a diffusion model condition generation module, a feature fusion module, a dual-branch feature interaction module, and a diffusion model posture estimation network. First, the uncertain 3D initial posture is extracted by the initial posture extraction module, the diffusion model condition generation module generates the diffusion model condition, the feature fusion module fuses the condition and posture features, and the fused features are sent to the diffusion model posture estimation network and the dual-branch feature interaction module performs feature interaction, and finally an accurate 3D posture is obtained.

[0017] The specific modules are introduced as follows:

[0018] Initial pose extraction module: Initialize the (x, y, z) distribution. Intuitively, the x and y uncertainty distributions contain information about the possible areas where joints are located in the image. The joint heatmap obtained by the existing 2D pose extractor is used as the initial x and y. For the initial z distribution, the frequency of occurrence of z in the training set is calculated, and then the histogram of each joint is obtained as the z distribution. Finally, the initial 3D pose with high uncertainty is obtained by sampling z.

[0019] Diffusion model condition generation module: This paper introduces the semantic information of text into the 3D human pose estimation task and uses it as the condition of the diffusion model. Specifically, the description words of 17 body joints (such as left elbow, right elbow, torso, etc.) are used as input and encoded based on the CLIP text encoder to generate text features of the joints. Since CLIP is pre-trained based on contrastive learning between images and text, the CLIP text encoder not only adds semantic information to the pose features, but also adds the original image information, which will also promote the generation of more accurate 3D poses.

[0020] Feature fusion module: After generating text semantic features, it will be combined with joint features Fusion is performed through the posture and semantic feature fusion module:

[0021] x ′ = Cross_Attention(x,f T ) (1)

[0022] Feature fusion modules, such as Figure 2 As shown, the joint feature x is used as the query and the text feature f T As the key and value, the joint features and text features are fused through the multi-head self-attention module to obtain the enhanced joint feature x ′ .

[0023] Dual-branch feature processing module: This paper uses parallel graph convolution and attention as the backbone of the denoising network (such as Figure 3). Based on the use of graph convolution to process the local information between posture joints, the parallel attention module fully considers the global information between the input posture joints. Finally, the global and local information interact to obtain a more accurate 3D posture.

[0024] Graph Convolutional Network (GCN). The GCN network receives the node features of the graph as input, characterizes the connectivity between nodes through the adjacency matrix, and uses the adjacency matrix as the convolution kernel to generate a new feature representation for each joint point, and use it to update the information of the node. The GCN network formula is as follows:

[0025]

[0026] in, represents the input and output feature matrix of the GCN network, N is the number of nodes, and c μ and c μ Indicates the number of input and output channels; represents the parameter matrix; I N represents the sum of the adjacency matrix and a diagonal matrix; represents the degree matrix.

[0027] Multi-Head Self-Attention (MSA). The attention module is built on the self-attention module of Transformer. It implicitly associates the input features in the form of attention scores. The input of this module is a query matrix Q, key matrix K and value matrix V. Where N represents the number of input patches, C η represents the dimension size of each patch. The output of H-head MSA can be expressed as follows:

[0028] MSA(Q,V,K,V)=Concat(head 1 , … ,head h )W O (3)

[0029] in is the linear projection weight, and each head i The calculation is as follows:

[0030]

[0031] in is the query, key, and value matrix of the i-th attention head, C η ′ =C η / h is the dimension of each attention head.

[0032] Interaction between graph convolution and attention. The GCN network focuses on the local information between the joints of the human body, but lacks the modeling of the global information between the joints. The attention block can better model the global information between the joints, but it also lacks the structural information between the joints. In order to give full play to the advantages of these two methods, the present invention considers an interaction mechanism, so that the global information of the attention block can be integrated while retaining the structural information of the GCN network. The specific fusion method is as follows:

[0033]

[0034] Where S attn and S GCN Represent the scale coefficients of the GCN network and the attention block, and these two coefficients are learnable. In this way, the GCN network and attention can interactively process feature information, so that global and local information can be better integrated.

[0035] Diffusion model posture estimation network: The schematic diagram of the dual-branch diffusion framework of the present invention in the reasoning process is as follows Figure 1 First, the initial uncertain posture x is obtained from the initial posture module k ∈R J×3 Next, with x k As input, the dimension is transformed through the linear layer to obtain the posture potential embedding E∈R J×96 ; Use a sine function to generate the diffusion step size The joint semantic feature f is obtained by the diffusion model conditional generation module T ∈R J×512 , where E and f T First, the dimension is changed, and then it is sent to the feature fusion module for feature fusion. After that, it passes through the GCN network and the attention parallel diffusion module to fully model the global and local information between joints. The whole process is done by diffusion step Guide the diffusion process; the final posture feature passes through an MLP layer to remap the joint features to x k ∈R J×3 , and then re-enters the diffusion network, and after k iterations, the final 3D pose is obtained.

[0036] During the training process, the training network reconstructs the 3D pose x in a supervised manner, iterates k times in the diffusion model, and after the iteration is completed, the final 3D pose x is obtained. 0 .

[0037]

[0038] Among them, g θ represents the diffusion model, x t Indicates the current posture. represents the diffusion step length, f T Represents joint semantic information.

[0039] Experimental Section

[0040] Experimental data set: The present invention is trained on the Human3.6M data set and tested on the Human3.6M and MPI-INF-3DHP data sets.

[0041] Human3.6M is the most widely used indoor dataset for 3D human pose estimation, consisting of 3.6 million images captured by four cameras, in which 11 subjects performed 15 daily activities. For frame-based 3D pose estimation, five trainees were trained on 5 subjects (S1, S5, S6, S7, S8) and tested on 2 subjects (S9 and S11). The present invention uses the P1 protocol to demonstrate the results of the model. MPI-INF-3DHP contains data for indoor and complex outdoor scenes, including 1.3 million frames of image data. The dataset records data of 8 actors performing 8 activities in 14 camera views. GS, noGS, and Outdoor in the experimental table indicate that tests were conducted on green screen, non-green screen indoor scenes, and outdoor scenes, respectively.

[0042] Evaluation Metrics:

[0043] MPJPE: It is used to evaluate the average Euclidean distance between the predicted 3D joint points and the true value.

[0044] PCK: The percentage of correct joint points in three-dimensional space under different thresholds. The calculation threshold in the present invention is 150mm.

[0045] AUC: The area under the PCK curve for different error thresholds. In the present invention, the curve area under the 150 mm error threshold is counted.

[0046] Experimental setup: The network sets the number of back-diffusion steps K to 50 steps; and DDIM is used to accelerate training. The present invention uses the detected 2D posture (using CPN) and the ground-truth 2D posture as input for training and testing on Human3.6M. The ground-truth 2D posture is used for testing on MPI-INF-3DHP. The Adam optimizer is used to train the diffusion model, where the initial learning rate is set to le-4 and the decay rate after ten iterations is 0.9. The present invention is built on the PyTorch framework and trained on a single RTX 3090 GPU with a batch size set to 1024.

[0047] Comparative experiments: The present invention is compared with some baseline methods and the most advanced 3D human pose estimation methods on the Human3.6M and MPI-INF-3DHP datasets. Table 1 shows the quantitative results on the Human3.6M dataset, and Table 2 shows the quantitative results on the MPI-INF-3DHP dataset, both of which achieve good results.

[0048] Table 1. Results on the Human3.6M dataset, in millimeters (MPJPE), showing the results with extracted 2D poses and ground-truth 2D poses as input. The best results are in bold.

[0049]

[0050]

[0051] Table 2. Results on the MPIINF-3DHP dataset. The best results are marked in bold.

[0052]

[0053] Ablation experiment: The present invention conducted a series of experiments on the Human3.6M dataset to verify the role of each module. First, only the GCN network is used, then the dual-branch diffusion network is used, and then the joint semantic information is added as a condition on this basis. GCN means that the denoising network only uses the GCN network, Dual means that the denoising network is an interactive GCN network and an attention dual-branch model, and Joint-text means adding joint text as a condition. The experimental results are shown in Table 3.

[0054] Table 3. Ablation experiments on different components of the model

[0055]

[0056]

Claims

1. A method for human posture estimation based on a conditional two-branch diffusion model, characterized in that: First, a 2D posture estimation network is used to obtain a heat map of human joints, and a depth distribution is obtained from the training set. Then, an uncertain 3D posture is obtained from the 2D joint heat map and the depth distribution. The semantic information based on human joints is obtained through the diffusion model conditional generation module. The uncertain 3D posture features and joint semantic information are fused through the feature fusion module. The fused features are iterated multiple times through the GCN network and the attention dual-branch diffusion model to obtain a highly certain 3D posture. The dual-branch feature interaction module promotes the GCN network and attention module to model the global and local context between joints; finally, the network is constrained by the real annotated 3D posture and the loss is calculated for training.

2. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 1, characterized in that: The uncertain 3D pose comprises the following steps: Step 1: Given an input RGB image, a heat map of human joints is obtained through a 2D posture extraction network; then the depth distribution of human joints is predicted using the training set; the initial uncertain 3D posture is obtained based on the 2D heat map and depth sampling obtained by the 2D posture extraction network; Step 2: Use the diffusion model condition generation module to obtain semantic information based on joint text, and use the joint text as the condition of the diffusion model; Step 3: 3D pose estimation; Step 31, transform the dimension of the obtained initial uncertain 3D posture through a linear layer to obtain the initialized joint features; take the joint semantic information obtained by the diffusion condition generation module as a condition, fuse the joint features with the joint semantic information through the feature fusion module to obtain the fused features, which are used as the input of the GCN network and the attention dual-branch interactive network; the fused features are processed by the dual-branch interactive network module and then passed through a regression layer to obtain the 3D posture of the human body; Step 32: The obtained 3D posture is used as the input of the dual-branch diffusion network again, and after K iterations of denoising, the final 3D posture is obtained.

3. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 1, characterized in that: The basic model for implementing this method includes an initial posture extraction module, a diffusion model condition generation module, a feature fusion module, a dual-branch feature interaction module and a diffusion model posture estimation network; the uncertain 3D initial posture is extracted by the initial posture extraction module, the conditions of the diffusion model are generated by the diffusion model condition generation module, the conditions and posture features are fused by the feature fusion module, the fused features are sent to the diffusion model posture estimation network and the dual-branch feature interaction module performs feature interaction, and finally an accurate 3D posture is obtained.

4. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 3, characterized in that: Initial pose extraction module: Initialize (x, y, z) distribution; the x and y uncertainty distributions contain information about the possible areas where joints are located in the image, and the joint heat map obtained by the existing 2D pose extractor is used as the initial x and y; for the initial z distribution, the frequency of occurrence of z in the training set is calculated, and then the histogram of each joint is obtained as the z distribution; finally, the initial 3D pose with high uncertainty is obtained by sampling z.

5. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 3, characterized in that: Diffusion model conditional generation module: The semantic information of the text is introduced into the 3D human pose estimation task and used as the condition of the diffusion model; specifically, the description words of 17 body joints are used as input and encoded based on the CLIP text encoder to generate the text features f of the joints. T ∈R m×Cemb ; CLIP text encoder is used to add semantic information to posture features.

6. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 3, characterized in that: Feature fusion module: After generating text semantic information, it is combined with joint posture features Fusion is performed through the feature fusion module: x′=Cross_Atttention(x,f T )(1) The joint feature x of the feature fusion module is used as the query and the text feature f T As keys and values, the joint features and text features are fused through a multi-head self-attention module to obtain the enhanced joint feature x′.

7. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 3, characterized in that: Dual-branch feature interaction module: parallel graph convolution and attention are used as the backbone of the diffusion model. On the basis of using graph convolution to process local information between posture joints, the parallel attention module fully considers the global information between input posture joints. Finally, the global and local information interact to obtain accurate 3D posture. The GCN network receives the node features of the graph as input, characterizes the connectivity between nodes through the adjacency matrix, and uses the adjacency matrix as the convolution kernel to generate a new feature representation for each joint point, and use it to update the information of the node; the GCN network formula is as follows: in, represents the input and output feature matrix of the GCN network, N is the number of nodes, and c μ and c μ Indicates the number of input and output channels; represents the parameter matrix; represents the sum of the adjacency matrix and a diagonal matrix; represents the degree matrix; Multi-head self-attention MSA; the attention module is built by the self-attention module of Transformer; implicitly associates input features in the form of attention scores; the input is a query matrix Q, key matrix K and value matrix V; Where N represents the number of input patches, C η Represents the dimension size of each patch; the output of H-head MSA is expressed as follows: MSA(Q,V,K,V)=Concat(head1,…head i ,…head h )W O (3) in is the linear projection weight, and each head i The calculation is as follows: in is the query, key, and value matrix of the i-th attention head, C′ η =C η / h is the dimension of each attention head; Interaction between graph convolution and attention; consider an interaction mechanism to integrate the global information of the attention block while retaining the GCN network structure information; the specific fusion method is as follows: Where S attn and S GCN Represent the scale coefficients of the GCN network and the attention block respectively.

8. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 3, characterized in that: Diffusion model pose estimation network: During the inference process, the dual-branch diffusion framework obtains the initial uncertain pose x from the initial pose module. k ∈R J×3 ; x k As input, the dimension is transformed through the linear layer to obtain the posture potential embedding E∈R J ×96 ; Use a sine function to generate the diffusion step size The joint semantic information f is obtained by the diffusion model conditional generation module T ∈R J×512 , where E and f T First, the dimension is changed, and then it is sent to the feature fusion module for feature fusion. After that, it passes through the GCN network and the attention parallel diffusion module to fully model the global and local information between joints. Guidance; Final The pose features are passed through an MLP layer to remap the joint features to x k ∈R J×3 , and then re-enter the diffusion network, and after k iterations, the final 3D pose is obtained; During the training process, the training network reconstructs the 3D pose x in a supervised manner, iterating k times in the diffusion model. After the iteration is completed, the final 3D pose x0 is obtained; Among them, g θ represents the diffusion model, x t Indicates the current posture, x t-1 Indicates the posture at the previous moment. represents the diffusion step length, f T represents the joint semantic information, k is the number of iterations, and K is the total number of iterations.

9. The method for estimating human posture based on a conditional dual-branch diffusion model according to claim 1, characterized in that: The GCN network sets the number of back-diffusion steps K to 50 steps and uses DDIM to accelerate training; The detected 2D pose and ground-truth 2D pose are used as input for training and testing on Human3.6M respectively; the ground-truth 2D pose is used for testing on MPI-INF-3DHP; the Adam optimizer is used to train the diffusion model, where the initial learning rate is set to le-4 and the decay rate is 0.9 after ten iterations; it is built based on the PyTorch framework and trained on a single RTX3090 GPU with a batch size of 1024.

Citation Information

Patent Citations

  • Three-dimensional hand attitude estimation method based on spatial perception and structural modeling

    CN118155239A

  • Assembly body change detection method, device and medium based on attention mechanism

    US20220358334A1

Cited By

  • Multimodal three-dimensional hand grid reconstruction method based on diffusion model

    CN119152149A

  • Controllable attitude generation method and related device

    CN119621942A

  • Monocular three-dimensional human body posture estimation method and device based on diffusion model and feature fusion

    CN119810917A

  • Global data sensing method, device and equipment based on diffusion model

    CN119988801A

  • Global Data Sensing Method, Device and Equipment Based on Diffusion Model

    CN119988801B